RMAN恢复遭遇ORA-00600

renjie_41

源库和目标库版本：10.2.0.4.12    RAC
restore controlfile并mount后，restore database正常，但执行rescover database报错

MAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 10/17/2013 11:28:44
ORA-00283: recovery session canceled due to errors
RMAN-11003: failure during parse/execution of SQL statement: alter database recover logfile '/archaccdb1/acc1_161997_684251212.dbf'
ORA-00283: recovery session canceled due to errors
ORA-12801: error signaled in parallel query server P007, instance xjaccdb1:accdb1 (1)
ORA-00600: 内部错误代码, 参数: [3020], [319], [745187], [1], [161997], [645553], [356], []
ORA-10567: Redo is inconsistent with data block (file#

刚开始怀疑是由于dbid的原因导致redo的问题，但把redo file重建后问题依旧；也没发现到坏块；
急求各位老师的帮助

renjie_41

目标库是在新的主机建立的一个空库，设置和源库完全相同。
其中警告日志报错如下：
...
Thu Oct 17 13:45:48 2013
alter database recover logfile '/archaccdb1/acc1_161997_684251212.dbf'
Thu Oct 17 13:45:48 2013
Media Recovery Log /archaccdb1/acc1_161997_684251212.dbf
ORA-279 signalled during: alter database recover logfile '/archaccdb1/acc1_161997_684251212.dbf'...
Thu Oct 17 13:45:48 2013
alter database recover logfile '/archaccdb1/acc2_157788_684251212.dbf'
Thu Oct 17 13:45:48 2013
Media Recovery Log /archaccdb1/acc2_157788_684251212.dbf
Thu Oct 17 13:45:53 2013
Errors in file /oracle/admin/accdb/bdump/accdb1_p014_7733398.trc:
ORA-00600: 内部错误代码, 参数: [3020], [319], [745217], [1338728193], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 319, block# 745217)
ORA-10564: tablespace BILLCFG
ORA-01110: 数据文件 319: '/dev/racc_data330'
ORA-10560: block type '58'
Thu Oct 17 13:45:54 2013
Trace dumping is performing id=[cdmp_20131017134554]
Thu Oct 17 13:45:54 2013
Errors in file /oracle/admin/accdb/bdump/accdb1_p014_7733398.trc:
ORA-00600: 内部错误代码, 参数: [3020], [319], [745217], [1338728193], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 319, block# 745217)
ORA-10564: tablespace BILLCFG
ORA-01110: 数据文件 319: '/dev/racc_data330'
ORA-10560: block type '58'
Thu Oct 17 13:45:57 2013
Errors with log /archaccdb1/acc2_157788_684251212.dbf
Recovery interrupted!
Recovered data files to a consistent state at change 13412091463909
Thu Oct 17 13:46:03 2013
Media Recovery failed with error 12801
ORA-283 signalled during: alter database recover logfile '/archaccdb1/acc2_157788_684251212.dbf'...

renjie_41

相应trace文件

Maclean Liu(刘相兵

ORA-00600: 内部错误代码, 参数: [3020], [319], [745187], [1], [161997], [645553], [356], []
ORA-10567: Redo is inconsistent with data block (file#

This is called a 'STUCK RECOVERY'.
There is an inconsistency between the information stored in the redo
and the information stored in a database block being recovered.
ARGUMENTS:
For Oracle 9.2 and earlier:
Arg [a] Block DBA
Arg [b] Redo Thread
Arg [c] Redo RBA Seq
Arg [d] Redo RBA Block No
Arg [e] Redo RBA Offset.
For Oracle 10.1
Arg [a] Absolute file number of the datafile.
Arg [b] Block number
Arg [c] Block DBA
FUNCTIONALITY:
kernel cache recovery parallel
IMPACT:
INSTANCE FAILURE during recovery.
SUGGESTIONS:
There have been cases of receiving this error when RECOVER has
been issued, but either some datafiles were not restored to disk,
or the restoration has not finished.
Therefore, ensure that the entire backup has been restored and that
the restore has finished PRIOR to issuing a RECOVER database command.
If problems continue, consider restoring from a backup and doing a
point-in-time recovery to a time PRIOR to the one implied by
the ORA-600[3020] error.
Example:
SQL> recover database until time 'YYYY-MON-DD:HH:MI:SS';
This error can also be caused by a lost update.
During normal operations, block updates/writes are being performed to
a number of files including database datafiles, redo log files,
archived redo log files etc.
This error can be reported if any of these updates are lost for some
reason.
Therefore, thoroughly check your operating system and disk hardware.
In the case of a lost update, restore an old copy of the datafile and
attempt to recover and roll forward again.
If the Known Issues section below does not help in terms of identifying
a solution, please submit the trace files and alert.log to Oracle
Support Services for further analysis.

Maclean Liu(刘相兵

ORA-10567: Redo is inconsistent with data block (file# 319, block# 745217)
ORA-10564: tablespace BILLCFG

可能是该数据块 错误讹误 file# 319, block# 745217

试试

recover database allow 1 corruption;

renjie_41

RMAN> recover database allow 1 corruption;

Starting recover at 18-OCT-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=5435 instance=accdb2 devtype=DISK

starting media recovery

archive log thread 1 sequence 161997 is already on disk as file /archaccdb2/acc1_161997_684251212.dbf
archive log thread 1 sequence 161998 is already on disk as file /archaccdb2/acc1_161998_684251212.dbf
archive log thread 1 sequence 161999 is already on disk as file /archaccdb2/acc1_161999_684251212.dbf
archive log thread 1 sequence 162000 is already on disk as file /archaccdb2/acc1_162000_684251212.dbf
archive log thread 1 sequence 162001 is already on disk as file /archaccdb2/acc1_162001_684251212.dbf
archive log thread 1 sequence 162002 is already on disk as file /archaccdb2/acc1_162002_684251212.dbf
archive log thread 1 sequence 162003 is already on disk as file /archaccdb2/acc1_162003_684251212.dbf
archive log thread 1 sequence 162004 is already on disk as file /archaccdb2/acc1_162004_684251212.dbf
archive log thread 1 sequence 162005 is already on disk as file /archaccdb2/acc1_162005_684251212.dbf
archive log thread 1 sequence 162006 is already on disk as file /archaccdb2/acc1_162006_684251212.dbf
archive log thread 1 sequence 162007 is already on disk as file /archaccdb2/acc1_162007_684251212.dbf
archive log thread 1 sequence 162008 is already on disk as file /archaccdb2/acc1_162008_684251212.dbf
archive log thread 1 sequence 162009 is already on disk as file /archaccdb2/acc1_162009_684251212.dbf
archive log thread 1 sequence 162010 is already on disk as file /archaccdb2/acc1_162010_684251212.dbf
archive log thread 1 sequence 162011 is already on disk as file /archaccdb2/acc1_162011_684251212.dbf
archive log thread 1 sequence 162012 is already on disk as file /archaccdb2/acc1_162012_684251212.dbf
archive log thread 1 sequence 162013 is already on disk as file /archaccdb2/acc1_162013_684251212.dbf
archive log thread 1 sequence 162014 is already on disk as file /archaccdb2/acc1_162014_684251212.dbf
archive log thread 1 sequence 162015 is already on disk as file /archaccdb2/acc1_162015_684251212.dbf
archive log thread 1 sequence 162016 is already on disk as file /archaccdb2/acc1_162016_684251212.dbf
archive log thread 1 sequence 162017 is already on disk as file /archaccdb2/acc1_162017_684251212.dbf
archive log thread 1 sequence 162018 is already on disk as file /archaccdb2/acc1_162018_684251212.dbf
archive log thread 1 sequence 162019 is already on disk as file /archaccdb2/acc1_162019_684251212.dbf
archive log thread 1 sequence 162020 is already on disk as file /archaccdb2/acc1_162020_684251212.dbf
archive log thread 2 sequence 157788 is already on disk as file /archaccdb2/acc2_157788_684251212.dbf
archive log thread 2 sequence 157789 is already on disk as file /archaccdb2/acc2_157789_684251212.dbf
archive log thread 2 sequence 157790 is already on disk as file /archaccdb2/acc2_157790_684251212.dbf
archive log thread 2 sequence 157791 is already on disk as file /archaccdb2/acc2_157791_684251212.dbf
archive log thread 2 sequence 157792 is already on disk as file /archaccdb2/acc2_157792_684251212.dbf
archive log thread 2 sequence 157793 is already on disk as file /archaccdb2/acc2_157793_684251212.dbf
archive log thread 2 sequence 157794 is already on disk as file /archaccdb2/acc2_157794_684251212.dbf
archive log thread 2 sequence 157795 is already on disk as file /archaccdb2/acc2_157795_684251212.dbf
archive log thread 2 sequence 157796 is already on disk as file /archaccdb2/acc2_157796_684251212.dbf
archive log thread 2 sequence 157797 is already on disk as file /archaccdb2/acc2_157797_684251212.dbf
archive log thread 2 sequence 157798 is already on disk as file /archaccdb2/acc2_157798_684251212.dbf
archive log thread 2 sequence 157799 is already on disk as file /archaccdb2/acc2_157799_684251212.dbf
archive log thread 2 sequence 157800 is already on disk as file /archaccdb2/acc2_157800_684251212.dbf
archive log thread 2 sequence 157801 is already on disk as file /archaccdb2/acc2_157801_684251212.dbf
archive log thread 2 sequence 157802 is already on disk as file /archaccdb2/acc2_157802_684251212.dbf
archive log thread 2 sequence 157803 is already on disk as file /archaccdb2/acc2_157803_684251212.dbf
archive log thread 2 sequence 157804 is already on disk as file /archaccdb2/acc2_157804_684251212.dbf
archive log thread 2 sequence 157805 is already on disk as file /archaccdb2/acc2_157805_684251212.dbf
archive log thread 2 sequence 157806 is already on disk as file /archaccdb2/acc2_157806_684251212.dbf
archive log thread 2 sequence 157807 is already on disk as file /archaccdb2/acc2_157807_684251212.dbf
archive log thread 2 sequence 157808 is already on disk as file /archaccdb2/acc2_157808_684251212.dbf
archive log thread 2 sequence 157809 is already on disk as file /archaccdb2/acc2_157809_684251212.dbf
archive log filename=/archaccdb2/acc1_161997_684251212.dbf thread=1 sequence=161997
archive log filename=/archaccdb2/acc2_157788_684251212.dbf thread=2 sequence=157788
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 10/18/2013 14:56:49
ORA-00283: recovery session canceled due to errors
RMAN-11003: failure during parse/execution of SQL statement: alter database recover logfile '/archaccdb2/acc2_157788_684251212.dbf'
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [319], [744728], [1338727704], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 319, block# 744728)
ORA-10564: tablespace BILLCFG
ORA-01110: data file 319: '/dev/racc_data330'
ORA-10560: block type '58'

Liu Maclean(刘相兵

方案1 从别的backup中restore 出datafile 319

方案2 将file# 319 offline

方案3　RMAN-11003: failure during parse/execution of SQL statement: alter database recover logfile '/archaccdb2/acc2_157788_684251212.dbf'

仅recover到 157788之前


方案4 通过 bbed 修改数据块 file# 319, block# 744728  来绕过该问题

xifenfei

恢复到现在，搞定了吗？这个应该open库，找回来大部分数据，没有问题

renjie_41

还没有，去MOS上查了，感觉像是这个bug，Bug 7207932，在10.2.0.5以上版本得到修复，但不太确认，所以向各位老师求证下

renjie_41

SQL> alter database datafile 319 offline;
RMAN> run{
2> set until sequence 157788 thread 2;
3> recover database;
4> }

executing command: SET until clause

Starting recover at 19-OCT-13
using channel ORA_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 10/19/2013 16:27:54
RMAN-06556: datafile 1 must be restored from backup older than scn 13412090430988

这些方法都试了，依然遭遇这个问题。之前别的11gRAC数据库割接，也用的同样方法步骤，没发现类似问题，会不会是10.2.0.4的bug？在另一套crm库最这个项目，昨天也出现了RMAN-00600