10g RAC (raw) 迁移到 11g RAC(ASM) , ASM diskgroup corrupted

miloluo

Hi, 各位
在做一个Oracle 10g RAC(raw devices) 到 Oracle 11g RAC(ASM) 以rman方式迁移的时候，碰到了个问题：
在把10g备份恢复到11g之后，由于之前的redo log指向是裸设备，所以，在ASM相应的disk group建立了一些之后要用的redo log files。但是一旦建立就发生 ASM的corrupted. 导致disk group直接dismount.

OS:        Redhat Enterprise Linux 5.4 64bit
源库：    Oracle 10g(10.2.0.4) RAC  (raw devices)
目标库： Oracle 11g(11.2.0.3) RAC(ASM)

升级的步骤大概如下：
============================================================================
-1. 源库: 运行 utlu112i_5.sql, 检查是否都满足升级到是11g的条件。
-2. 源库: 备份源库
-3. 目标库: 安装oracle 11g RAC
-4. 目标库: 用源库备份restore和recover目标库

-5. 目标库: 一些调整工作(例如，把redo日志的路径从原来的裸设备弄到ASM相应的disk group)，最终是为了能执行下面的语句：

                alter database open resetlogs upgrade


-6. 目标库: 升级目标库的数据字典
-7. 目标库: 测试目标库的可用性

============================================================================







[ 本帖最后由 miloluo 于 2012-8-22 06:43 编辑 ]

miloluo

现在进行到第5步的时候，有如下报错：

这是在操作升级的节点中的问题发生时的日志：

alert_+ASM2.log
=====================

1. Tue Aug 21 21:48:31 2012
   
2. WARNING: cache read  a corrupt block: group=1(DG_DATA) dsk=1 blk=4 disk=1 (DG_DATA_0001) incarn=3915931844 au=0 blk=4 count=1
   
3. Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_20359.trc:
   
4. ORA-15196: invalid ASM block header [kfc.c:26076] [hard_kfbh] [2147483649] [4] [34 != 130]
   
5. NOTE: a corrupted block from group DG_DATA was dumped to /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_20359.trc
   
6. WARNING: cache read (retry) a corrupt block: group=1(DG_DATA) dsk=1 blk=4 disk=1 (DG_DATA_0001) incarn=3915931844 au=0 blk=4 count=1
   
7. Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_20359.trc:
   
8. ORA-15196: invalid ASM block header [kfc.c:26076] [hard_kfbh] [2147483649] [4] [34 != 130]
   
9. ORA-15196: invalid ASM block header [kfc.c:26076] [hard_kfbh] [2147483649] [4] [34 != 130]
   
10. ERROR: cache failed to read group=1(DG_DATA) dsk=1 blk=4 from disk(s): 1(DG_DATA_0001)
    
11. ORA-15196: invalid ASM block header [kfc.c:26076] [hard_kfbh] [2147483649] [4] [34 != 130]
    
12. ORA-15196: invalid ASM block header [kfc.c:26076] [hard_kfbh] [2147483649] [4] [34 != 130]
    
13. NOTE: cache initiating offline of disk 1 group DG_DATA
    
14. NOTE: process _user20359_+asm2 (20359) initiating offline of disk 1.3915931844 (DG_DATA_0001) with mask 0x7e in group 1
    
15. WARNING: Disk 1 (DG_DATA_0001) in group 1 in mode 0x7f is now being taken offline on ASM inst 2
    
16. NOTE: initiating PST update: grp = 1, dsk = 1/0xe96860c4, mask = 0x6a, op = clear
    
17. Tue Aug 21 21:48:31 2012
    
18. GMON updating disk modes for group 1 at 21 for pid 35, osid 20359
    
19. ERROR: Disk 1 cannot be offlined, since diskgroup has external redundancy.
    
20. ERROR: too many offline disks in PST (grp 1)
    
21. WARNING: Offline of disk 1 (DG_DATA_0001) in group 1 and mode 0x7f failed on ASM inst 2
    
22. Tue Aug 21 21:48:31 2012
    
23. NOTE: halting all I/Os to diskgroup 1 (DG_DATA)
    
24. Tue Aug 21 21:48:31 2012
    
25. NOTE: cache dismounting (not clean) group 1/0x3B08906D (DG_DATA)
    
26. NOTE: messaging CKPT to quiesce pins Unix process pid: 21477, image: oracle@rac11 (B000)
    
27. System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_20359.trc
    
28. Tue Aug 21 21:48:32 2012
    
29. ERROR: ORA-15130 in COD recovery for diskgroup 1/0x3b08906d (DG_DATA)
    
30. ERROR: ORA-15130 thrown in RBAL for group number 1
    
31. Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_4839.trc:
    
32. ORA-15130: diskgroup "DG_DATA" is being dismounted
    
33. Tue Aug 21 21:48:34 2012
    
34. NOTE: LGWR doing non-clean dismount of group 1 (DG_DATA)
    
35. NOTE: LGWR sync ABA=2.578 last written ABA 2.578
    
36. Tue Aug 21 21:48:34 2012
    
37. kjbdomdet send to inst 1
    
38. detach from dom 1, sending detach message to inst 1
    
39. Tue Aug 21 21:48:34 2012
    
40. List of instances:
    
41. 1 2
    
42. Dirty detach reconfiguration started (new ddet inc 1, cluster inc 8)
    
43. Global Resource Directory partially frozen for dirty detach
    
44. * dirty detach - domain 1 invalid = TRUE
    
45. 299 GCS resources traversed, 0 cancelled
    
46. Dirty Detach Reconfiguration complete
    
47. Tue Aug 21 21:48:34 2012
    
48. WARNING: dirty detached from domain 1
    
49. NOTE: cache dismounted group 1/0x3B08906D (DG_DATA)
    
50. SQL> alter diskgroup DG_DATA dismount force /* ASM SERVER */
    
51. Received dirty detach msg from inst 1 for dom 1
    
52. List of instances:
    
53. 1 2
    
54. Dirty detach reconfiguration started (new ddet inc 2, cluster inc 8)
    
55. Global Resource Directory partially frozen for dirty detach
    
56. * dirty detach - domain 1 invalid = TRUE
    
57. 50 GCS resources traversed, 0 cancelled
    
58. freeing rdom 1
    
59. Dirty Detach Reconfiguration complete
    
60. ERROR: ORA-15130 in COD recovery for diskgroup 1/0x3b08906d (DG_DATA)
    
61. ERROR: ORA-15130 thrown in RBAL for group number 1
    
62. Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_4839.trc:
    
63. ORA-15130: diskgroup "DG_DATA" is being dismounted
    
64. ERROR: ORA-15130 in COD recovery for diskgroup 1/0x3b08906d (DG_DATA)
    
65. ERROR: ORA-15130 thrown in RBAL for group number 1
    
66. Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_4839.trc:
    
67. ORA-15130: diskgroup "DG_DATA" is being dismounted
    
68. NOTE: AMDU dump of disk group DG_DATA created at /u01/app/oracle/diag/asm/+asm/+ASM2/trace
    
69. Tue Aug 21 21:48:40 2012
    
70. NOTE: cache deleting context for group DG_DATA 1/0x3b08906d
    
71. GMON dismounting group 1 at 22 for pid 27, osid 21477
    
72. NOTE: Disk  in mode 0x8 marked for de-assignment
    
73. NOTE: Disk  in mode 0x8 marked for de-assignment
    
74. NOTE: Disk  in mode 0x8 marked for de-assignment
    
75. NOTE: Disk  in mode 0x8 marked for de-assignment
    
76. NOTE: Disk  in mode 0x8 marked for de-assignment
    
77. NOTE: Disk  in mode 0x8 marked for de-assignment
    
78. NOTE: Disk  in mode 0x8 marked for de-assignment
    
79. NOTE: Disk  in mode 0x8 marked for de-assignment
    
80. NOTE: Disk  in mode 0x8 marked for de-assignment
    
81. NOTE: Disk  in mode 0x8 marked for de-assignment
    
82. SUCCESS: diskgroup DG_DATA was dismounted
    
83. SUCCESS: alter diskgroup DG_DATA dismount force /* ASM SERVER */
    
84. Tue Aug 21 21:48:41 2012
    
85. NOTE: diskgroup resource ora.DG_DATA.dg is offline
    
86. ERROR: PST-initiated MANDATORY DISMOUNT of group DG_DATA
    
87. NOTE: ASM client rac2:rac disconnected unexpectedly.
    
88. NOTE: check client alert log.
    
89. NOTE: Trace records dumped in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_20761.trc
    
90. Tue Aug 21 21:48:47 2012
    
91. Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_21515.trc:
    
92. ORA-17503: ksfdopn:2 Failed to open file +DG_DATA/rac/spfilerac.ora
    
93. ORA-15001: diskgroup "DG_DATA" does not exist or is not mounted

复制代码

alert_rac2.log
============================

1. Tue Aug 21 21:48:12 2012
   
2. alter database
   
3. add logfile thread 1
   
4. GROUP 9  ('+dg_data/rac/onlinelog/t1_g9_redo.log') SIZE 90M ,
   
5. GROUP 10 ('+dg_data/rac/onlinelog/t1_g10_redo.log') SIZE 90M ,
   
6. GROUP 11 ('+dg_data/rac/onlinelog/t1_g11_redo.log') SIZE 90M ,
   
7. GROUP 12 ('+dg_data/rac/onlinelog/t1_g12_redo.log') SIZE 90M ,
   
8. GROUP 13 ('+dg_data/rac/onlinelog/t1_g13_redo.log') SIZE 90M ,
   
9. GROUP 14 ('+dg_data/rac/onlinelog/t1_g14_redo.log') SIZE 90M
   
10. Tue Aug 21 21:48:40 2012
    
11. SUCCESS: diskgroup DG_DATA was dismounted
    
12. SUCCESS: diskgroup DG_DATA was dismounted
    
13. Tue Aug 21 21:48:41 2012
    
14. ORA-301 signalled during: alter database
    
15. add logfile thread 1
    
16. GROUP 9  ('+dg_data/rac/onlinelog/t1_g9_redo.log') SIZE 90M ,
    
17. GROUP 10 ('+dg_data/rac/onlinelog/t1_g10_redo.log') SIZE 90M ,
    
18. GROUP 11 ('+dg_data/rac/onlinelog/t1_g11_redo.log') SIZE 90M ,
    
19. GROUP 12 ('+dg_data/rac/onlinelog/t1_g12_redo.log') SIZE 90M ,
    
20. GROUP 13 ('+dg_data/rac/onlinelog/t1_g13_redo.log') SIZE 90M ,
    
21. GROUP 14 ('+dg_data/rac/onlinelog/t1_g14_redo.log') SIZE 90M ...
    
22. Tue Aug 21 21:48:41 2012
    
23. Errors in file /u01/app/oracle/diag/rdbms/rac/rac2/trace/rac2_lmon_20198.trc:
    
24. ORA-00202: control file: '+DG_DATA/rac/control01.ctl'
    
25. ORA-15078: ASM diskgroup was forcibly dismounted
    
26. Tue Aug 21 21:48:41 2012
    
27. Errors in file /u01/app/oracle/diag/rdbms/rac/rac2/trace/rac2_ckpt_20216.trc:
    
28. ORA-00206: error in writing (block 4, # blocks 1) of control file
    
29. ORA-00202: control file: '+DG_DATA/rac/control02.ctl'
    
30. ORA-15078: ASM diskgroup was forcibly dismounted
    
31. ORA-15078: ASM diskgroup was forcibly dismounted
    
32. ORA-00206: error in writing (block 4, # blocks 1) of control file
    
33. ORA-00202: control file: '+DG_DATA/rac/control01.ctl'
    
34. ORA-15078: ASM diskgroup was forcibly dismounted
    
35. ORA-15078: ASM diskgroup was forcibly dismounted
    
36. Errors in file /u01/app/oracle/diag/rdbms/rac/rac2/trace/rac2_ckpt_20216.trc:
    
37. ORA-00221: error on write to control file
    
38. ORA-00206: error in writing (block 4, # blocks 1) of control file
    
39. ORA-00202: control file: '+DG_DATA/rac/control02.ctl'
    
40. ORA-15078: ASM diskgroup was forcibly dismounted
    
41. ORA-15078: ASM diskgroup was forcibly dismounted
    
42. ORA-00206: error in writing (block 4, # blocks 1) of control file
    
43. ORA-00202: control file: '+DG_DATA/rac/control01.ctl'
    
44. ORA-15078: ASM diskgroup was forcibly dismounted
    
45. ORA-15078: ASM diskgroup was forcibly dismounted
    
46. Tue Aug 21 21:48:41 2012
    
47. System state dump requested by (instance=2, osid=20216 (CKPT)), summary=[abnormal instance termination].
    
48. System State dumped to trace file /u01/app/oracle/diag/rdbms/rac/rac2/trace/rac2_diag_20188.trc
    
49. Dumping diagnostic data in directory=[cdmp_20120821214841], requested by (instance=2, osid=20216 (CKPT)), summary=[abnormal instance termination].
    
50. CKPT (ospid: 20216): terminating the instance due to error 221
    
51. Tue Aug 21 21:48:45 2012
    
52. ORA-1092 : opitsk aborting process
    
53. Tue Aug 21 21:48:45 2012
    
54. License high water mark = 6
    
55. Instance terminated by CKPT, pid = 20216
    
56. USER (ospid: 21507): terminating the instance
    
57. Instance terminated by USER, pid = 21507

复制代码

上传相关的日志和trace文件。
在MOS中发现一个问题和这个问题类似：
<[size=130%]Alert: Querying V$ASM_FILE Gives ORA-15196 After ASM Was Upgraded From 10gR2 To 11gR2 with an AU size > 1M [ID 1145365.1]>

但是，我之前的10g是RAC是raw devices，并不存在AU size一说。
请各位帮忙看看。

谢谢。

[ 本帖最后由 miloluo 于 2012-8-22 06:41 编辑 ]

Maclean Liu(刘相兵

今后请去 OTN中文论坛发帖提问 https://forums.oracle.com/forums/forum.jspa?forumID=314
已经代你发帖 并回复了https://forums.oracle.com/forums ... 429994&tstart=0