环境：Primary还可用的情况下

Linux  RedHat 5.6 x86_64
Oracle 11.2.0.2 x86_64
Primary database : PRI11G
Standby database ：STA11G

步骤：（一定要Standby database所在服务器上执行）





如果在Primary databae上执行如下提示：

DGMGRL> failover to sta11g
Performing failover NOW, please wait...
Error: ORA-16600: not connected to target standby database for failover

Failed.
Unable to failover


在Standby database执行failover


DGMGRL> failover to sta11g;
DGMGRL> 

DGMGRL> failover to sta11g;

Performing failover NOW, please wait...

Error: ORA-16748: Data Guard broker cannot open the primary database



Failed.

Unable to failover



原Standby database的aler日志如下：

aler日志观察dg broker步骤



Thu Aug 08 18:32:27 2013
NSV0 started with pid=37, OS id=1716
Thu Aug 08 18:32:30 2013
RSM0 started with pid=38, OS id=1726
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='STA11G';
ALTER SYSTEM SET log_archive_format='log_%t_%s_%r.arc' SCOPE=SPFILE SID='STA11G';
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
ALTER SYSTEM SET fal_server='dg_pri11g' SCOPE=BOTH;
Thu Aug 08 18:32:35 2013
Standby controlfile consistent with primary
RFS[3]: Selected log 4 for thread 1 sequence 11519 dbid 1058399494 branch 785177303
Thu Aug 08 18:32:35 2013
Archived Log entry 41 added for thread 1 sequence 11518 ID 0x3f54c93c dest 1:
Thu Aug 08 18:32:36 2013
Media Recovery Waiting for thread 1 sequence 11519 (in transit)
Recovery of Online Redo Log: Thread 1 Group 4 Seq 11519 Reading mem 0
  Mem# 0: /u01/oradata/STA11G/standby_04.log
Thu Aug 08 18:33:30 2013
Data Guard Broker: Beginning failover
Thu Aug 08 18:33:30 2013
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
Thu Aug 08 18:33:30 2013
MRP0: Background Media Recovery cancelled with status 16037
Errors in file /u01/app/oracle/diag/rdbms/sta11g/STA11G/trace/STA11G_pr00_846.trc:
ORA-16037: user requested cancel of managed recovery operation
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Some recovered datafiles maybe left media fuzzy
Media recovery may continue but open resetlogs may fail
Thu Aug 08 18:33:30 2013
MRP0: Background Media Recovery process shutdown (STA11G)
Managed Standby Recovery Canceled (STA11G)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH FORCE
Attempt to do a Terminal Recovery (STA11G)
Media Recovery Start: Managed Standby Recovery (STA11G)
started logmerger process
Thu Aug 08 18:33:31 2013
Managed Standby Recovery not using Real Time Apply
Parallel Media Recovery started with 2 slaves
Media Recovery Log /u01/arch/STA11G/log_1_11518_785177303.arc
Media Recovery Waiting for thread 1 sequence 11519 (in transit)
krsv_proc_kill: Killing 4 processes (all RFS, wait for I/O)



Begin: Standby Redo Logfile archival
End: Standby Redo Logfile archival
Terminal Recovery timestamp is '08/08/2013 18:33:34'
Terminal Recovery: applying standby redo logs.
Terminal Recovery: thread 1 seq# 11519 redo required
Terminal Recovery:
Recovery of Online Redo Log: Thread 1 Group 4 Seq 11519 Reading mem 0
  Mem# 0: /u01/oradata/STA11G/standby_04.log
Identified End-Of-Redo (failover) for thread 1 sequence 11519 at SCN 0xffff.ffffffff
Incomplete Recovery applied until change 10625988265 time 08/08/2013 18:33:53
Media Recovery Complete (STA11G)
Terminal Recovery: successful completion
Forcing ARSCN to IRSCN for TR 2:2036053673
Attempt to set limbo arscn 2:2036053673 irscn 2:2036053673
Resetting standby activation ID 1062521148 (0x3f54c93c)
Thu Aug 08 18:33:35 2013
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance STA11G - Archival Error
ORA-16014: log 4 sequence# 11519 not archived, no available destinations
ORA-00312: online log 4 thread 1: '/u01/oradata/STA11G/standby_04.log'
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH FORCE
ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWN
ALTER DATABASE SWITCHOVER TO PRIMARY (STA11G)
Maximum wait for role transition is 15 minutes.
Backup controlfile written to trace file /u01/app/oracle/diag/rdbms/sta11g/STA11G/trace/STA11G_rsm0_1726.trc
Standby terminal recovery start SCN: 10625987714
RESETLOGS after incomplete recovery UNTIL CHANGE 10625988265
Online logfile pre-clearing operation disabled by switchover
Online log /u01/oradata/STA11G/redo01.log: Thread 1 Group 1 was previously cleared
Online log /u01/oradata/STA11G/redo02.log: Thread 1 Group 2 was previously cleared
Online log /u01/oradata/STA11G/redo03.log: Thread 1 Group 3 was previously cleared


Standby became primary SCN: 10625987713
Thu Aug 08 18:33:35 2013
Setting recovery target incarnation to 6
Switchover: Complete - Database mounted as primary
Completed: ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWN
ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY
Completed: ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY
ALTER DATABASE OPEN
Data Guard Broker initializing...
Errors in file /u01/app/oracle/diag/rdbms/sta11g/STA11G/trace/STA11G_rsm0_1726.trc:
ORA-01190: control file or data file 21 is from before the last RESETLOGS
ORA-01110: data file 21: '/u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf'
ORA-1190 signalled during: ALTER DATABASE OPEN...
Failover succeeded. Primary database is now STA11G.
Failover failed with ORA-16748.
Thu Aug 08 18:34:53 2013
ARC0: Becoming the 'no SRL' ARCH
Thu Aug 08 18:34:54 2013
ARC2: Becoming the 'no SRL' ARCH
Thu Aug 08 18:35:48 2013
Archiver process freed from errors. No longer stopped
Thu Aug 08 18:35:48 2013
ALTER SYSTEM SET log_archive_dest_2='service="dg_pri11g"','LGWR SYNC AFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=1 db_unique_name="pri11g" net_timeout=30','valid_for=(all_logfiles,primary_role)' SCOPE=BOTH;
Using STANDBY_ARCHIVE_DEST parameter default value as /u01/arch/STA11G
ALTER SYSTEM SET log_archive_dest_state_2='RESET' SCOPE=BOTH;
Thu Aug 08 18:36:31 2013
Using STANDBY_ARCHIVE_DEST parameter default value as /u01/arch/STA11G
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
Thu Aug 08 18:36:41 2013
Using STANDBY_ARCHIVE_DEST parameter default value as /u01/arch/STA11G
ALTER SYSTEM SET log_archive_dest_state_2='RESET' SCOPE=BOTH;
Thu Aug 08 18:39:28 2013
alter database open
Data Guard Broker initializing...
Data Guard Broker initialization complete
Data Guard: verifying database primary role...



Failover之后，却在open的阶段报错了，报错信息如下：


ALTER DATABASE OPEN
Data Guard Broker initializing...
Errors in file /u01/app/oracle/diag/rdbms/sta11g/STA11G/trace/STA11G_rsm0_1726.trc:
ORA-01190: control file or data file 21 is from before the last RESETLOGS
ORA-01110: data file 21: '/u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf'
ORA-1190 signalled during: ALTER DATABASE OPEN...


参考MOS[266991.1]解决，文章在下面

1>、先看是为什么 file_id =21的数据文件报错

控制文件中：

sys@STA11G> select hxfil,fhrls change#,fhrlc_i,fhrlc time from x$kcvfh;

     HXFIL CHANGE#       FHRLC_I TIME
---------- ---------------- ---------- --------------------
1 10625988266     822940415 08/08/2013 18:33:35
2 10625988266     822940415 08/08/2013 18:33:35
         .............

20 10625988266     822940415 08/08/2013 18:33:35
21 10424579917     801852960 12/12/2012 16:56:00
22 10625988266     822940415 08/08/2013 18:33:35
23 10625988266     822940415 08/08/2013 18:33:35
25 10625988266     822940415 08/08/2013 18:33:35
26 10625988266     822940415 08/08/2013 18:33:35

25 rows selected.

17:07:50 sys@STA11G> set numwidth 20
17:07:56 sys@STA11G> col name for a60
 sys@STA11G> select file#,name,status,checkpoint_change# from v$datafile;

            FILE# NAME                                                            STATUS    CHECKPOINT_CHANGE#
-------------------- ------------------------------------------------------------ ------- --------------------
             1 /u01/oradata/STA11G/datafile/o1_mf_system_906mtvlp_.dbf            SYSTEM        10625988266
             2 /u01/oradata/STA11G/datafile/o1_mf_sysaux_906mz00j_.dbf            ONLINE        10625988266
            ........

            20 /u01/oradata/STA11G/datafile/o1_mf_tsbpart0_906mz8rz_.dbf          ONLINE        10625988266
            21 /u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf               ONLINE        10533466679
            22 /u01/oradata/STA11G/datafile/o1_mf_rman_906mzmw3_.dbf              ONLINE        10625988266
            23 /u01/oradata/STA11G/datafile/o1_mf_undotbs2_906mkh1f_.dbf          ONLINE        10625988266
            25 /u01/oradata/STA11G/datafile/o1_mf_test_906mzq2t_.dbf              ONLINE        10625988266
            26 /u01/oradata/STA11G/datafile/o1_mf_test_ddl_906mkhc8_.dbf          ONLINE        10625988266

25 rows selected.


数据文件中：

sys@STA11G> select file#,to_char(checkpoint_change#,'999999999999'),to_char(RESETLOGS_CHANGE#,'999999999999')from v$datafile_header;

     FILE# TO_CHAR(CHECK TO_CHAR(RESET
---------- ------------- -------------
     1   10625988266   10625988266
     2   10625988266   10625988266
     ..........

     20   10625988266   10625988266
     21   10533466679   10424579917
     22   10625988266   10625988266
     23   10625988266   10625988266
     25   10625988266   10625988266
     26   10625988266   10625988266

25 rows selected.

可以观察到，file# = 21 的文件checkpoint_change、resetlogs_change# 自己在控制文件、数据文头部中是一致的，但与其他数据文件的值却小很多，说明与其他数据文件比较要旧很多，但自身在控制文件、数据文件头是一致的，猜测可能是只读表空间的设置！

[checkpoint_change#  10533466679 < 10625988266 & RESETLOGS_CHANGE# < 10424579917 ]

sys@STA11G> select file#,name ,status,enabled from v$datafile;

     FILE# NAME                                                         STATUS     ENABLED
---------- ------------------------------------------------------------ ------- ----------
     1 /u01/oradata/STA11G/datafile/o1_mf_system_906mtvlp_.dbf          SYSTEM     DISABLED
     2 /u01/oradata/STA11G/datafile/o1_mf_sysaux_906mz00j_.dbf          ONLINE     DISABLED
     ......

     20 /u01/oradata/STA11G/datafile/o1_mf_tsbpart0_906mz8rz_.dbf       ONLINE     DISABLED
     21 /u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf            ONLINE     READ ONLY
     22 /u01/oradata/STA11G/datafile/o1_mf_rman_906mzmw3_.dbf           ONLINE     DISABLED
     23 /u01/oradata/STA11G/datafile/o1_mf_undotbs2_906mkh1f_.dbf       ONLINE     DISABLED
     25 /u01/oradata/STA11G/datafile/o1_mf_test_906mzq2t_.dbf           ONLINE     DISABLED
     26 /u01/oradata/STA11G/datafile/o1_mf_test_ddl_906mkhc8_.dbf      ONLINE      DISABLED

25 rows selected.

发现问题了，file#=21 的数据文件为read only，其他为disable（因为不是open状态，是mount状态）

其实也可以查询dba_tablespace.status 但为mount状态，还不能查询dba_tablespace

2> 根据 MOS【266991.1】解决

16:18:38 sys@STA11G> alter database datafile 21 offline drop;

Database altered.

16:18:47 sys@STA11G> select FILE# from v$recover_file;

            FILE#
--------------------
            21

16:18:52 sys@STA11G> select file#,name,status from v$datafile_header;

            FILE# NAME                                                            STATUS
-------------------- ------------------------------------------------------------ -------
             1 /u01/oradata/STA11G/datafile/o1_mf_system_906mtvlp_.dbf            ONLINE
             2 /u01/oradata/STA11G/datafile/o1_mf_sysaux_906mz00j_.dbf            ONLINE
            ......
            20 /u01/oradata/STA11G/datafile/o1_mf_tsbpart0_906mz8rz_.dbf          ONLINE
            21 /u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf               OFFLINE
            22 /u01/oradata/STA11G/datafile/o1_mf_rman_906mzmw3_.dbf              ONLINE
            23 /u01/oradata/STA11G/datafile/o1_mf_undotbs2_906mkh1f_.dbf          ONLINE
            25 /u01/oradata/STA11G/datafile/o1_mf_test_906mzq2t_.dbf              ONLINE
            26 /u01/oradata/STA11G/datafile/o1_mf_test_ddl_906mkhc8_.dbf          ONLINE

25 rows selected.

---此部与MOS【266991.1】报错不一样

16:19:51 sys@STA11G>  recover database until cancel;
ORA-00283: recovery session canceled due to errors
ORA-16433: The database must be opened in read/write mode.


16:20:11 sys@STA11G> select open_mode from v$database;

OPEN_MODE
--------------------
MOUNTED

16:20:18 sys@STA11G> select name,status,checkpoint_change# from v$datafile;

NAME                                                          STATUS    CHECKPOINT_CHANGE#
------------------------------------------------------------ ------- --------------------
/u01/oradata/STA11G/datafile/o1_mf_system_906mtvlp_.dbf      SYSTEM           10625988266
/u01/oradata/STA11G/datafile/o1_mf_sysaux_906mz00j_.dbf      ONLINE           10625988266
.......
/u01/oradata/STA11G/datafile/o1_mf_tsbpart0_906mz8rz_.dbf    ONLINE           10625988266
/u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf         RECOVER          10533466679
/u01/oradata/STA11G/datafile/o1_mf_rman_906mzmw3_.dbf        ONLINE           10625988266
/u01/oradata/STA11G/datafile/o1_mf_undotbs2_906mkh1f_.dbf    ONLINE           10625988266
/u01/oradata/STA11G/datafile/o1_mf_test_906mzq2t_.dbf        ONLINE           10625988266
/u01/oradata/STA11G/datafile/o1_mf_test_ddl_906mkhc8_.dbf    ONLINE           10625988266

25 rows selected.

--因为默认undo表空间为undotbs1，但数据库为undotbs2，所以修改默认undo表空间即可

16:22:37 sys@STA11G> alter database open;
ERROR:
ORA-03114: not connected to ORACLE


alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-30012: undo tablespace 'UNDOTBS1' does not exist or of wrong type
Process ID: 1194
Session ID: 9 Serial number: 3


16:24:21 STA11G> alter system set undo_tablespace='UNDOTBS2' scope=spfile;

System altered.

SQL> startup mount

16:24:46 sys@STA11G> alter database open;

Database altered.

16:25:13 sys@STA11G> select name,to_char(RESETLOGS_TIME,'DD-MON-YYYY HH:MI::SS') from v$database;

NAME       TO_CHAR(RESETLOGS_TIME,'
--------- ------------------------
PRI11G       08-AUG-2013 06:33::35


16:25:54 sys@STA11G> select file#,name,to_char(RESETLOGS_TIME,'DD-MON-YYYY HH:MI::SS') from v$datafile_header;

     FILE# NAME                                                         TO_CHAR(RESETLOGS_TIME,'
---------- ------------------------------------------------------------ ------------------------
     1 /u01/oradata/STA11G/datafile/o1_mf_system_906mtvlp_.dbf          08-AUG-2013 06:33::35
     2 /u01/oradata/STA11G/datafile/o1_mf_sysaux_906mz00j_.dbf          08-AUG-2013 06:33::35
     ..........
     20 /u01/oradata/STA11G/datafile/o1_mf_tsbpart0_906mz8rz_.dbf       08-AUG-2013 06:33::35
     21                                                                                        -- in datafile header entry is there
     22 /u01/oradata/STA11G/datafile/o1_mf_rman_906mzmw3_.dbf           08-AUG-2013 06:33::35
     23 /u01/oradata/STA11G/datafile/o1_mf_undotbs2_906mkh1f_.dbf       08-AUG-2013 06:33::35
     25 /u01/oradata/STA11G/datafile/o1_mf_test_906mzq2t_.dbf           08-AUG-2013 06:33::35
     26 /u01/oradata/STA11G/datafile/o1_mf_test_ddl_906mkhc8_.dbf       08-AUG-2013 06:33::35

25 rows selected.

16:26:02 sys@STA11G> select file# from v$recover_file;

     FILE#
----------
     21

16:26:11 sys@STA11G> recover datafile 8;
ORA-00283: recovery session canceled due to errors
ORA-01124: cannot recover data file 8 - file is in use or recovery
ORA-01110: data file 8: '/u01/oradata/STA11G/datafile/o1_mf_ggs_906mkh9b_.dbf'


16:26:17 sys@STA11G> alter database datafile 21 online;

Database altered.


16:26:57 sys@STA11G> select tablespace_name,status from dba_tablespaces;

TABLESPACE_NAME                 STATUS
------------------------------ ---------
SYSTEM                          ONLINE
SYSAUX                          ONLINE
TEMP                            ONLINE
......
TSBPART04                       ONLINE
TBS                             READ ONLY
RMAN                            ONLINE
FLB_1                           ONLINE
TEST                            ONLINE
TEST_DDL                        ONLINE

26 rows selected.

16:27:12 sys@STA11G> select file#,name,status from v$datafile;

     FILE# NAME                                                          STATUS
---------- ------------------------------------------------------------ -------
     1 /u01/oradata/STA11G/datafile/o1_mf_system_906mtvlp_.dbf           SYSTEM
     2 /u01/oradata/STA11G/datafile/o1_mf_sysaux_906mz00j_.dbf           ONLINE
     .............
     20 /u01/oradata/STA11G/datafile/o1_mf_tsbpart0_906mz8rz_.dbf        ONLINE
     21 /u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf             RECOVER
     22 /u01/oradata/STA11G/datafile/o1_mf_rman_906mzmw3_.dbf            ONLINE
     23 /u01/oradata/STA11G/datafile/o1_mf_undotbs2_906mkh1f_.dbf        ONLINE
     25 /u01/oradata/STA11G/datafile/o1_mf_test_906mzq2t_.dbf            ONLINE
     26 /u01/oradata/STA11G/datafile/o1_mf_test_ddl_906mkhc8_.dbf        ONLINE

25 rows selected.

16:27:17 sys@STA11G> select name,checkpoint_change# from v$datafile_header;

NAME     CHECKPOINT_CHANGE#
------------------------------------------------------------ ------------------
/u01/oradata/STA11G/datafile/o1_mf_system_906mtvlp_.dbf     1.0626E+10
/u01/oradata/STA11G/datafile/o1_mf_sysaux_906mz00j_.dbf     1.0626E+10
.................
/u01/oradata/STA11G/datafile/o1_mf_tsbpart0_906mz8rz_.dbf     1.0626E+10
/u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf     1.0533E+10
/u01/oradata/STA11G/datafile/o1_mf_rman_906mzmw3_.dbf     1.0626E+10
/u01/oradata/STA11G/datafile/o1_mf_undotbs2_906mkh1f_.dbf     1.0626E+10
/u01/oradata/STA11G/datafile/o1_mf_test_906mzq2t_.dbf     1.0626E+10
/u01/oradata/STA11G/datafile/o1_mf_test_ddl_906mkhc8_.dbf     1.0626E+10

25 rows selected.

16:27:50 sys@STA11G> select name,to_char(RESETLOGS_TIME,'DD-MON-YYYY HH:MI::SS') from v$datafile_header;

NAME                                                          TO_CHAR(RESETLOGS_TIME,'
------------------------------------------------------------ ------------------------
/u01/oradata/STA11G/datafile/o1_mf_system_906mtvlp_.dbf      08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_sysaux_906mz00j_.dbf      08-AUG-2013 06:33::35
..............
/u01/oradata/STA11G/datafile/o1_mf_tsbpart0_906mz8rz_.dbf    08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf         12-DEC-2012 04:56::00
/u01/oradata/STA11G/datafile/o1_mf_rman_906mzmw3_.dbf        08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_undotbs2_906mkh1f_.dbf    08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_test_906mzq2t_.dbf        08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_test_ddl_906mkhc8_.dbf    08-AUG-2013 06:33::35

25 rows selected.

16:28:09 sys@STA11G> alter tablespace tbs read write;

Tablespace altered.

--- RECOVER --> ONLINE

16:28:23 sys@STA11G> select file#,name,status from v$datafile;

     FILE# NAME                                                          STATUS
---------- ------------------------------------------------------------ -------
     1 /u01/oradata/STA11G/datafile/o1_mf_system_906mtvlp_.dbf          SYSTEM
     2 /u01/oradata/STA11G/datafile/o1_mf_sysaux_906mz00j_.dbf          ONLINE
     ..................
     20 /u01/oradata/STA11G/datafile/o1_mf_tsbpart0_906mz8rz_.dbf       ONLINE
     21 /u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf            ONLINE
     22 /u01/oradata/STA11G/datafile/o1_mf_rman_906mzmw3_.dbf           ONLINE
     23 /u01/oradata/STA11G/datafile/o1_mf_undotbs2_906mkh1f_.dbf       ONLINE
     25 /u01/oradata/STA11G/datafile/o1_mf_test_906mzq2t_.dbf           ONLINE
     26 /u01/oradata/STA11G/datafile/o1_mf_test_ddl_906mkhc8_.dbf       ONLINE

25 rows selected.

 sys@STA11G> select name,to_char(RESETLOGS_TIME,'DD-MON-YYYY HH:MI::SS') from v$datafile_header;

NAME                                                         TO_CHAR(RESETLOGS_TIME,'
------------------------------------------------------------ ------------------------
/u01/oradata/STA11G/datafile/o1_mf_system_906mtvlp_.dbf      08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_sysaux_906mz00j_.dbf      08-AUG-2013 06:33::35
.........
/u01/oradata/STA11G/datafile/o1_mf_tsbpart0_906mz8rz_.dbf    08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_tbs_906mzgjs_.dbf         08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_rman_906mzmw3_.dbf        08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_undotbs2_906mkh1f_.dbf    08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_test_906mzq2t_.dbf        08-AUG-2013 06:33::35
/u01/oradata/STA11G/datafile/o1_mf_test_ddl_906mkhc8_.dbf    08-AUG-2013 06:33::35

25 rows selected.


---恢复原主库，现备库

此时观察dg broker的配置

从原standby 现在Primary查看如下：

[oracle@dbatwo dbs]$ dgmgrl /

DGMGRL> show configuration

Configuration - FSFO

  Protection Mode: MaxAvailability
  Databases:
    sta11g - Primary database
      Warning: ORA-16629: database reports a different protection level from the protection mode

    pri11g - Physical standby database (disabled)
      ORA-16661: the standby database needs to be reinstated

Fast-Start Failover: DISABLED

Configuration Status:
WARNING

从原Primary 现Standby查看如下：-- 知道Physical standby在failover后，Standby 需要重建，但我们还可能通过在 新primary database上reinstated

[oracle@dbaone ~]$ dgmgrl sys/oracle

DGMGRL> show configuration;
ORA-16795: the standby database needs to be re-created

Configuration details cannot be determined by DGMGRL

在 新Primary database上reinstate -- 借助flashback + covert 特性 -- 但reinstate 失败了

DGMGRL> reinstate database pri11g
Reinstating database "pri11g", please wait...
Operation requires shutdown of instance "PRI11G" on database "pri11g"
Shutting down instance "PRI11G"...
Database closed.
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "PRI11G" on database "pri11g"
Starting instance "PRI11G"...
ORACLE instance started.
Database mounted.
Continuing to reinstate database "pri11g" ...
Error: ORA-16653: failed to reinstate database

Failed.
Reinstatement of database "pri11g" failed

观察 新Standby 的alert日志


Thu Aug 08 18:36:06 2013
NSV1 started with pid=39, OS id=22462 
Thu Aug 08 18:36:10 2013
Shutting down instance (immediate)
Stopping background process SMCO
Stopping background process FBDA
Shutting down instance: further logons disabled

.............

---关闭数据库

---启动数据库 nomount --> mount

.............

Thu Aug 08 18:36:35 2013
alter database  mount
Successful mount of redo thread 1, with mount id 1096617075
Allocated 3981120 bytes in shared pool for flashback generation buffer
Starting background process RVWR
Thu Aug 08 18:36:39 2013
RVWR started with pid=21, OS id=22615 
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: alter database  mount

--启动dg broker进程

Starting Data Guard Broker (DMON)
Thu Aug 08 18:36:43 2013
INSV started with pid=22, OS id=22621 
Thu Aug 08 18:36:49 2013
NSV1 started with pid=23, OS id=22630 
Thu Aug 08 18:36:52 2013

--启动闪回进程，并执行闪回数据库操作，这个scn值就是刚才 原standby 新primary 的aler记录 “Standby became primary SCN: 10625987713” 
--在新Primary database上查询得知

--18:10:55 sys@STA11G> set numwidth 20
--18:11:00 sys@STA11G> select STANDBY_BECAME_PRIMARY_SCN from v$database;

--STANDBY_BECAME_PRIMARY_SCN
--------------------------
            10625987713

RSM0 started with pid=24, OS id=22636 
FLASHBACK DATABASE TO SCN 10625987713
Flashback Restore Start
Flashback Restore Complete
Flashback Media Recovery Start
started logmerger process
Parallel Media Recovery started with 2 slaves
Flashback Media Recovery Log /u01/arch/PRI11G/1_11512_785177303.dbf
Flashback Media Recovery Log /u01/arch/PRI11G/1_11513_785177303.dbf
Flashback Media Recovery Log /u01/arch/PRI11G/1_11514_785177303.dbf
Flashback Media Recovery Log /u01/arch/PRI11G/1_11515_785177303.dbf
Flashback Media Recovery Log /u01/arch/PRI11G/1_11516_785177303.dbf
Flashback Media Recovery Log /u01/arch/PRI11G/1_11517_785177303.dbf
Thu Aug 08 18:36:59 2013
Recovery of Online Redo Log: Thread 1 Group 2 Seq 11518 Reading mem 0
  Mem# 0: /u01/oradata/PRI11G/redo02.log
Incomplete Recovery applied until change 10625987714 time 08/08/2013 18:32:57
Flashback Media Recovery Complete
Completed: FLASHBACK DATABASE TO SCN 10625987713

--在convert时报错了

alter database convert to physical standby
ALTER DATABASE CONVERT TO PHYSICAL STANDBY (PRI11G)
Flush standby redo logfile failed:1649
Errors in file /u01/app/oracle/diag/rdbms/pri11g/PRI11G/trace/PRI11G_rsm0_22636.trc  (incident=580657):
ORA-00600: internal error code, arguments: [krhpfh_03-1204], [fno =], [21], [fhfno =], [17], [], [], [], [], [], [], []
ORA-01110: data file 21: '/u01/oradata/PRI11G/datafile/o1_mf_tbs_8rgzjyby_.dbf'
Incident details in: /u01/app/oracle/diag/rdbms/pri11g/PRI11G/incident/incdir_580657/PRI11G_rsm0_22636_i580657.trc
Thu Aug 08 18:37:02 2013
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORA-600 signalled during: alter database convert to physical standby...
Thu Aug 08 18:37:03 2013
Dumping diagnostic data in directory=[cdmp_20130808183703], requested by (instance=1, osid=22636 (RSM0)), summary=[incident=580657].
Thu Aug 08 18:37:38 2013
Sweep [inc][580657]: completed
Sweep [inc2][580657]: completed




新standby不能reinstate 主要问题：



alter database convert to physical standby
ALTER DATABASE CONVERT TO PHYSICAL STANDBY (PRI11G)
Flush standby redo logfile failed:1649
Errors in file /u01/app/oracle/diag/rdbms/pri11g/PRI11G/trace/PRI11G_rsm0_22636.trc  (incident=580657):
ORA-00600: internal error code, arguments: [krhpfh_03-1204], [fno =], [21], [fhfno =], [17], [], [], [], [], [], [], []
ORA-01110: data file 21: '/u01/oradata/PRI11G/datafile/o1_mf_tbs_8rgzjyby_.dbf'
Incident details in: /u01/app/oracle/diag/rdbms/pri11g/PRI11G/incident/incdir_580657/PRI11G_rsm0_22636_i580657.trc




参考杨版主的博客：http://yangtingkun.itpub.net/post/468/526544


这个错误产生的原因是文件头被损坏，其中krhpfh_03-1208错误信息中包含了两部分信息，前一部分krhpfh_03是ORA-600错误的错误号，而后面的1208是导致这个ORA-600错误的错误号
也就是ORA-01208: data file is an old version - not accessing current version

导致这个ORA-600出现的其他常见错误还包括：

ORA-01202: wrong incarnation of this file - wrong creation time；

ORA-01204: file number is string rather than string - wrong file ；

ORA-01206: file is not part of this database - wrong database id ；

ORA-01209: data file is from before the last RESETLOGS以及ORA-01210: data file header is media corrupt等等。

总的来说就是Oracle发现数据文件的头与控制文件中的不符所致。一般而言，不是由于磁盘损坏了头块，就是由于误操作所致


从上面的文字可知，是 file#=21的数据文件有问题，在处理Standby failover to Primary时已经知道是什么问题了！设置为read only的缘故！

此例中是 ORA-01204

$ oerr ora 01204
01204, 00000, "file number is %s rather than %s - wrong file"
// *Cause:  The file number in the file header is not correct. This is probably
//         a restored backup of the wrong file, but from the same database.
// *Action: Restore a copy of the correct data file and do recovery as needed.

18:28:21 sys@PRI11G> select file#,name,to_char(RESETLOGS_TIME,'DD-MON-YYYY HH:MI::SS') from v$datafile_header;

     FILE# NAME                                                          TO_CHAR(RESETLOGS_TIME,'
---------- ------------------------------------------------------------ ------------------------
     1 /u01/oradata/PRI11G/system01.dbf                                  05-JUN-2012 04:48::23
     2 /u01/oradata/PRI11G/sysaux01.dbf                                  05-JUN-2012 04:48::23
     ......
     20 /u01/oradata/PRI11G/datafile/o1_mf_tsbpart0_8gntx8ny_.dbf        05-JUN-2012 04:48::23
     21 /u01/oradata/PRI11G/datafile/o1_mf_tbs_8rgzjyby_.dbf             12-DEC-2012 04:56::00
     22 /u01/oradata/PRI11G/datafile/o1_mf_rman_8rrvjlr2_.dbf            05-JUN-2012 04:48::23
     23 /u01/oradata/PRI11G/datafile/o1_mf_undotbs2_8s6n0om6_.dbf        05-JUN-2012 04:48::23
     25 /u01/oradata/PRI11G/datafile/o1_mf_test_8wc4w5jz_.dbf            05-JUN-2012 04:48::23
     26 /u01/oradata/PRI11G/datafile/o1_mf_test_ddl_8z1vvggd_.dbf        05-JUN-2012 04:48::23

25 rows selected.
18:36:08 sys@PRI11G> set numwidth 20
18:36:09 sys@PRI11G> select file#,name,checkpoint_change# ,status,enabled from v$datafile;

            FILE# NAME                                                             CHECKPOINT_CHANGE#   STATUS  ENABLED
-------------------- ------------------------------------------------------------ -------------------- ------- ----------
             1 /u01/oradata/PRI11G/system01.dbf                                    10625987714          SYSTEM  READ WRITE
             2 /u01/oradata/PRI11G/sysaux01.dbf                                    10625987714          RECOVER READ WRITE
             ......
            20 /u01/oradata/PRI11G/datafile/o1_mf_tsbpart0_8gntx8ny_.dbf           10625987714          RECOVER READ WRITE
            21 /u01/oradata/PRI11G/datafile/o1_mf_tbs_8rgzjyby_.dbf                10533466679          ONLINE  READ ONLY
            22 /u01/oradata/PRI11G/datafile/o1_mf_rman_8rrvjlr2_.dbf               10625987714          RECOVER READ WRITE
            23 /u01/oradata/PRI11G/datafile/o1_mf_undotbs2_8s6n0om6_.dbf           10625987714          RECOVER READ WRITE
            25 /u01/oradata/PRI11G/datafile/o1_mf_test_8wc4w5jz_.dbf               10625987714          RECOVER READ WRITE
            26 /u01/oradata/PRI11G/datafile/o1_mf_test_ddl_8z1vvggd_.dbf           10625987714          RECOVER READ WRITE

25 rows selected.

18:28:56 sys@PRI11G> select status from v$instance;

STATUS
------------
MOUNTED

11:31:39 sys@PRI11G> alter tablespace tbs read write;
alter tablespace tbs read write
*
ERROR at line 1:
ORA-01109: database not open


09:55:45 sys@PRI11G> alter database open;
alter database open
*
ERROR at line 1:
ORA-16649: possible failover to another database prevents this database from being opened

09:59:08 sys@PRI11G> alter system set dg_broker_start=false;

System altered.

09:59:20 sys@PRI11G> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open

如果通过resetlogs 打开数据库，这个备库肯定不可用了




附：

Recovering READONLY tablespace backups made before a RESETLOGS Open (文档 ID 266991.1)


@***Checked for relevance on 13-Aug-2012***


How to recover a READONLY tablespace backups made before a RESETLOGS in a new
incarnation:

* Backups of a  tablespace made after it was made read-only 
 (only if it was not made read/write again before the RESETLOGS)


This is  a inhouse tested scenario 

Here are the steps :

STEPS
-----

SQL> select name,status,checkpoint_change# from v$datafile;
NAME                                     STATUS          CHECKPOINT_CHANGE#
---------------------------------------- -------         ------------------
D:\ORACLE\ORADATA\AMAR\SYSTEM01.DBF      SYSTEM             766796
D:\ORACLE\ORADATA\AMAR\RBS01.DBF         ONLINE             766796
D:\ORACLE\ORADATA\AMAR\USERS01.DBF       ONLINE             766796
D:\ORACLE\ORADATA\AMAR\TEMP01.DBF        ONLINE             766796
D:\ORACLE\ORADATA\AMAR\TOOLS01.DBF       ONLINE             766796
D:\ORACLE\ORADATA\AMAR\INDX01.DBF        ONLINE             766796
D:\ORACLE\ORADATA\AMAR\DR01.DBF          ONLINE             766796
D:\ORACLE\ORADATA\AMAR\TEST01.DBF        ONLINE             423736  --> THIS IS THE READ ONLY TABLESPACE
D:\ORACLE\ORADATA\AMAR\JAGU01.DBF        ONLINE             766864

9 rows selected.

SQL> select tablespace_name,status from dba_tablespaces;
TABLESPACE_NAME                       STATUS
------------------------------        ---------
SYSTEM                                 ONLINE
RBS                                    ONLINE
USERS                                  ONLINE
TEMP                                   ONLINE
TOOLS                                  ONLINE
INDX                                   ONLINE
DRSYS                                  ONLINE
TEST                                   READ ONLY
JAGU                                   ONLINE

9 rows selected.

SQL> shutdown abort
ORACLE instance shut down.

----------ASSUME THERE WAS  A  DATABASE CRASH --------------------

A few assumptions in this case 
------------------------

++ Having Restored  all the datafile except the read only
++ NOT  RESTORED THE  readonly tablespace datafile yet 
++ Trying to do the incomplete recovery

SQL> startup mount
ORACLE instance started.
Total System Global Area  187987996 bytes
Fixed Size                    75804 bytes
Variable Size              67436544 bytes
Database Buffers          120397824 bytes
Redo Buffers                  77824 bytes
Database mounted.



SQL> recover database until cancel ;
ORA-00283: recovery session canceled due to errors
ORA-01157: cannot identify/lock data file 8 - see DBWR trace file
ORA-01110: data file 8: 'D:\ORACLE\ORADATA\AMAR\TEST01.DBF'

Reason for this error 
-----------------------
++  As this file is missing .
++  SO EITHER we HAVE to  RESTORE FROM BACKUP or OFFLINE DROP THIS FILE


++  Now have dropped the datafile 8

SQL> alter database datafile 8 offline drop;

Database altered.

SQL> select name,status from v$datafile_header;
NAME                                      STATUS
---------------------------------------- -------
D:\ORACLE\ORADATA\AMAR\SYSTEM01.DBF      ONLINE
D:\ORACLE\ORADATA\AMAR\RBS01.DBF         ONLINE
D:\ORACLE\ORADATA\AMAR\USERS01.DBF       ONLINE
D:\ORACLE\ORADATA\AMAR\TEMP01.DBF        ONLINE
D:\ORACLE\ORADATA\AMAR\TOOLS01.DBF       ONLINE
D:\ORACLE\ORADATA\AMAR\INDX01.DBF        ONLINE
D:\ORACLE\ORADATA\AMAR\DR01.DBF          ONLINE
                                         OFFLINE - in datafile header entry is there
D:\ORACLE\ORADATA\AMAR\JAGU01.DBF        ONLINE

9 rows selected.

SQL> select FILE# from v$recover_file;
 FILE#
----------
         8  FILE NEEDS ReCOVERY 

+++ DONE a INCOMPLETE Recovery

SQL> recover database until cancel
cancel

SQL> alter database open resetlogs;

Database altered.

SQL> select name,to_char(RESETLOGS_TIME,'DD-MON-YYYY HH:MI::SS') from v$database

  2  ;
NAME                                         TO_CHAR(RESETLOGS_TIM
---------------------------------------- ---------------------
AMAR                                     01-MAR-2004 01:07::18

SQL> select name,to_char(RESETLOGS_TIME,'DD-MON-YYYY HH:MI::SS')
  2  from v$datafile_header;
NAME                                         TO_CHAR(RESETLOGS_TIM
---------------------------------------- ---------------------
D:\ORACLE\ORADATA\AMAR\SYSTEM01.DBF      01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\RBS01.DBF         01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\USERS01.DBF       01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\TEMP01.DBF        01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\TOOLS01.DBF       01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\INDX01.DBF        01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\DR01.DBF          01-MAR-2004 01:07::18
                                                                 MISSING FOR DATAFILE 8
D:\ORACLE\ORADATA\AMAR\JAGU01.DBF        01-MAR-2004 01:07::18

9 rows selected.

SQL> select file# from v$recover_file;
FILE#
----------
         8  - Still showing needs recovery after RESETLOGS

++ If this file was lost in the crash we will need to restore the same from the backup taken after it was made READ ONLY

SQL> recover datafile 8;
ORA-00283: recovery session canceled due to errors
ORA-00264: no recovery required


Reasons for the errors seen 
---------------------------

++ As this is a READ ONLY TS it does not need media recovery
++ We cant take the datafile online directly

SQL> alter database datafile 8 online;
alter database datafile 8 online
*
ERROR at line 1:
ORA-01190: controlfile or data file 8 is from before the last RESETLOGS
ORA-01110: data file 8: 'D:\ORACLE\ORADATA\AMAR\TEST01.DBF'

SQL> select tablespace_name,status from dba_tablespaces;
TABLESPACE_NAME                STATUS
------------------------------ ---------
SYSTEM                         ONLINE
RBS                            ONLINE
USERS                          ONLINE
TEMP                           ONLINE
TOOLS                          ONLINE
INDX                           ONLINE
DRSYS                          ONLINE
TEST                           READ ONLY -READONLY After RESETLOGS
JAGU                           ONLINE

9 rows selected.

SQL> select name,status from v$datafile;
 clsNAME                                 STATUS
---------------------------------------- -------
D:\ORACLE\ORADATA\AMAR\SYSTEM01.DBF      SYSTEM
D:\ORACLE\ORADATA\AMAR\RBS01.DBF         ONLINE
D:\ORACLE\ORADATA\AMAR\USERS01.DBF       ONLINE
D:\ORACLE\ORADATA\AMAR\TEMP01.DBF        ONLINE
D:\ORACLE\ORADATA\AMAR\TOOLS01.DBF       ONLINE
D:\ORACLE\ORADATA\AMAR\INDX01.DBF        ONLINE
D:\ORACLE\ORADATA\AMAR\DR01.DBF          ONLINE
D:\ORACLE\ORADATA\AMAR\TEST01.DBF        OFFLINE -- as we have offline drop this file
D:\ORACLE\ORADATA\AMAR\JAGU01.DBF        ONLINE

9 rows selected.

SQL> select name,status from v$datafile_header;
 clsNAME                                  STATUS
---------------------------------------- -------
D:\ORACLE\ORADATA\AMAR\SYSTEM01.DBF      ONLINE
D:\ORACLE\ORADATA\AMAR\RBS01.DBF         ONLINE
D:\ORACLE\ORADATA\AMAR\USERS01.DBF       ONLINE
D:\ORACLE\ORADATA\AMAR\TEMP01.DBF        ONLINE
D:\ORACLE\ORADATA\AMAR\TOOLS01.DBF       ONLINE
D:\ORACLE\ORADATA\AMAR\INDX01.DBF        ONLINE
D:\ORACLE\ORADATA\AMAR\DR01.DBF          ONLINE
                                         OFFLINE -- as we have offline drop this file
D:\ORACLE\ORADATA\AMAR\JAGU01.DBF        ONLINE

9 rows selected.


+++ TRYing to take the READ ONLY tablespace ONLINE
SQL> alter tablespace test online;

Tablespace altered.

SQL> select name,status from v$datafile;
NAME                                     STATUS
---------------------------------------- -------
D:\ORACLE\ORADATA\AMAR\SYSTEM01.DBF      SYSTEM
D:\ORACLE\ORADATA\AMAR\RBS01.DBF         ONLINE
D:\ORACLE\ORADATA\AMAR\USERS01.DBF       ONLINE
D:\ORACLE\ORADATA\AMAR\TEMP01.DBF        ONLINE
D:\ORACLE\ORADATA\AMAR\TOOLS01.DBF       ONLINE
D:\ORACLE\ORADATA\AMAR\INDX01.DBF        ONLINE
D:\ORACLE\ORADATA\AMAR\DR01.DBF          ONLINE
D:\ORACLE\ORADATA\AMAR\TEST01.DBF        ONLINE -File is online
D:\ORACLE\ORADATA\AMAR\JAGU01.DBF        ONLINE

9 rows selected.

SQL> select name,status from v$datafile_header;
NAME                                     STATUS
---------------------------------------- -------
D:\ORACLE\ORADATA\AMAR\SYSTEM01.DBF      ONLINE
D:\ORACLE\ORADATA\AMAR\RBS01.DBF         ONLINE
D:\ORACLE\ORADATA\AMAR\USERS01.DBF       ONLINE
D:\ORACLE\ORADATA\AMAR\TEMP01.DBF        ONLINE
D:\ORACLE\ORADATA\AMAR\TOOLS01.DBF       ONLINE
D:\ORACLE\ORADATA\AMAR\INDX01.DBF        ONLINE
D:\ORACLE\ORADATA\AMAR\DR01.DBF          ONLINE
D:\ORACLE\ORADATA\AMAR\TEST01.DBF        ONLINE - FIle is online
D:\ORACLE\ORADATA\AMAR\JAGU01.DBF        ONLINE

9 rows selected.

SQL> select tablespace_name,status from dba_tablespaces;
TABLESPACE_NAME                STATUS
------------------------------ ---------
SYSTEM                         ONLINE
RBS                            ONLINE
USERS                          ONLINE
TEMP                           ONLINE
TOOLS                          ONLINE
INDX                           ONLINE
DRSYS                          ONLINE
TEST                           READ ONLY -- Still in Read only after resetlogs and is accessible in the database.
JAGU                           ONLINE

9 rows selected.

SQL> select name,checkpoint_change# from v$datafile_header;
NAME                                       CHECKPOINT_CHANGE#
---------------------------------------- ------------------
D:\ORACLE\ORADATA\AMAR\SYSTEM01.DBF                  766872
D:\ORACLE\ORADATA\AMAR\RBS01.DBF                     766872
D:\ORACLE\ORADATA\AMAR\USERS01.DBF                   766872
D:\ORACLE\ORADATA\AMAR\TEMP01.DBF                    766872
D:\ORACLE\ORADATA\AMAR\TOOLS01.DBF                   766872
D:\ORACLE\ORADATA\AMAR\INDX01.DBF                    766872
D:\ORACLE\ORADATA\AMAR\DR01.DBF                      766872
D:\ORACLE\ORADATA\AMAR\TEST01.DBF                    423736 -- its  the same SCN as we started with 
D:\ORACLE\ORADATA\AMAR\JAGU01.DBF                    766872

9 rows selected.

SQL> select name,to_char(RESETLOGS_TIME,'DD-MON-YYYY HH:MI::SS')
  2  from v$datafile_header;
NAME                                      TO_CHAR(RESETLOGS_TIM
---------------------------------------- ---------------------
D:\ORACLE\ORADATA\AMAR\SYSTEM01.DBF      01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\RBS01.DBF         01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\USERS01.DBF       01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\TEMP01.DBF        01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\TOOLS01.DBF       01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\INDX01.DBF        01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\DR01.DBF          01-MAR-2004 01:07::18
D:\ORACLE\ORADATA\AMAR\TEST01.DBF        10-NOV-2003 01:12::26 -- Showing the LAST RESETLOGS date
D:\ORACLE\ORADATA\AMAR\JAGU01.DBF        01-MAR-2004 01:07::18

9 rows selected.

SQL> alter tablespace test read write;

Tablespace altered.

SQL> select name,checkpoint_change# from v$datafile_header;
NAME                                      CHECKPOINT_CHANGE#
---------------------------------------- ------------------
D:\ORACLE\ORADATA\AMAR\SYSTEM01.DBF                  766872
D:\ORACLE\ORADATA\AMAR\RBS01.DBF                     766872
D:\ORACLE\ORADATA\AMAR\USERS01.DBF                   766872
D:\ORACLE\ORADATA\AMAR\TEMP01.DBF                    766872
D:\ORACLE\ORADATA\AMAR\TOOLS01.DBF                   766872
D:\ORACLE\ORADATA\AMAR\INDX01.DBF                    766872
D:\ORACLE\ORADATA\AMAR\DR01.DBF                      766872
D:\ORACLE\ORADATA\AMAR\TEST01.DBF                    766928 -Sync with the other files
D:\ORACLE\ORADATA\AMAR\JAGU01.DBF                    766872

9 rows selected.

Now the Readonly tablespace before the incarnation is a part of the database