RAC一个节点启动不了

cameldba

10g rac,重启一个节点（OS.CRS,DB都重启了），发现一直停在mount状态，一直在OPEN，搞了一个小时还没有正常启动，没有报错，结果重新启动RAC的两个节点实例，才搞定。alert.log,crs,集群日志没有任何报错信息。在一个节点的alert有一条可能相关的信息：
Thread 1 advanced to log sequence 189311
  Current log# 2 seq# 189311 mem# 0: /iomdata1/iomdb/redo13.log
  Current log# 2 seq# 189311 mem# 1: /iomdata1/iomdb/redo14.log
Sun Mar  4 06:59:59 2012
GES: Potential blocker (pid=6558972) on resource TT-00000000-00000000;
enqueue info in file /oracle/app/oracle/admin/iomdb/bdump/iomdb1_lmd0_2119230.trc and DIAG trace file
Sun Mar  4 07:01:24 2012

为什么会无法open呢？

Maclean Liu(刘相兵

请上传 /oracle/app/oracle/admin/iomdb/bdump/iomdb1_lmd0_2119230.trc  和 当时 diag的trace

可能的一个bug:

1. Hdr: 5192411 10.2.0.1 RDBMS 10.2.0.1 RAC PRODID-5 PORTID-197
   
2. Abstract: DATABASE OPEN HANGS ON TT ENQUEUE
   
3. 
   
4. *** 04/27/06 08:28 am ***
   
5. 
   
6. 
   
7. =========================   
   
8. PROBLEM:
   
9. 
   
10. 1. Clear description of the problem encountered:
    
11. 
    
12.    Instance open can hang waiting on TT enqueue.
    
13. 
    
14. 2. Pertinent configuration information (MTS/OPS/distributed/etc)
    
15. 
    
16.    3-node RAC on HP/UX Itanium.
    
17. 
    
18. 3. Indication of the frequency and predictability of the problem  
    
19. 
    
20.     Seems intermittent, but ct. has given a sequence of events below to
    
21. repeat this:
    
22. 
    
23. 4. Sequence of events leading to the problem  
    
24. 
    
25. step 1. Instance 1,2,3 : open
    
26. step 2. Instance 1 : shutdown abort
    
27. step 3. Instance 2 : startup --> this will hang attempting to get TT enqueue.
    
28. 
    
29. 
    
30. 5. Technical impact on the customer. Include persistent after effects.
    
31. 
    
32.     Unable to start instance.
    
33. 
    
34. =========================   
    
35. DIAGNOSTIC ANALYSIS:
    
36. 
    
37. 
    
38. - From alert_node1.log :
    
39. 
    
40. Tue Apr 25 13:24:39 2006
    
41. GES: Potential blocker (pid=29817) on resource TT-0x0-0x0;
    
42. enqueue info in file /db/trace/bdump/ocoredb1_lmd0_21047.trc and DIAG trace
    
43. file
    
44. 
    
45. - The corresponding ocoredb1_diag_20660.trc :
    
46. 
    
47. [---cut---]
    
48. Dumping process info for ospid 29817:
    
49. ...
    
50. #4  0x4000000002e2deb0:0 in ksliwat () at ksl.c:7238
    
51. #5  0x4000000002d76c20:0 in kslwaitns () at ksl.c:7435
    
52. #6  0x4000000002d76d80:0 in kskthbwt () at ksk.c:1907
    
53. #7  0x4000000002d75a00:0 in kslwait () at ksl.c:7419
    
54. #8  0x4000000005653e40:0 in kjsdrmwt () at
    
55. /project/qa/reza/5036736/kjs.c:4129
    
56. #9  0x40000000056519a0:0 in kjsmbesmi () at
    
57. /project/qa/reza/5036736/kjs.c:4495
    
58. #10 0x40000000056f35e0:0 in kjbenterdlm ()
    
59. #11 0x4000000005477260:0 in kclusnaff () at
    
60. /project/qa/reza/5034539/kcl.c:9683
    
61. #12 0x40000000054b1b90:0 in kcblus () at kcb.c:16342
    
62. #13 0x40000000029c19e0:0 in $cold_ktuswr+0x1d90 () at ksl.c:3964
    
63. #14 0x40000000021d37a0:0 in ktusmous_online_undoseg () at ktusm.c:649
    
64. #15 0x4000000001e34710:0 in ktusmout_online_ut () at ktusm.c:1339
    
65. #16 0x4000000001da8100:0 in ktusmiut_init_ut () at ktusm.c:1010
    
66. #17 0x4000000001dae380:0 in ktuini () at ktu.c:2944
    
67. #18 0x400000000209da80:0 in adbdrv () at dbsdrv.c:4436
    
68. #19 0x4000000002e71090:0 in opiexe () at opiexe.c:2634
    
69. #20 0x4000000002e97460:0 in opiosq0 () at opiosq.c:598
    
70. #21 0x400000000238eb60:0 in kpooprx () at kpoal8.c:1607
    
71. #22 0x4000000002485c20:0 in kpoal8 () at kpoal8.c:657
    
72. #23 0x4000000002e3cd80:0 in opiodr () at opiodr.c:615
    
73. #24 0x4000000002ecbe90:0 in ttcpip () at ttcpip.c:946
    
74. #25 0x4000000002eca660:0 in opitsk () at opitsk.c:835
    
75. #26 0x40000000021c0890:0 in opiino () at opiino.c:1108
    
76. #27 0x4000000002e3cd80:0 in opiodr () at opiodr.c:615
    
77. #28 0x400000000214d1d0:0 in opidrv () at opidrv.c:793
    
78. #29 0x400000000214c6f0:0 in sou2o () at sou2o.c:128
    
79. #30 0x400000000213c340:0 in opimai_real () at opimai.c:257
    
80. #31 0x4000000002043a20:0 in main () at opimai.c:173
    
81. ...
    
82. Current wait and wait history for this process is  'gcs drm freeze in enter
    
83. server mode'
    
84. Current SQL statement 'ALTER DATABASE OPEN'
    
85. ...
    
86. Then we have a dump of the blocking process, followed by a systemstate for
    
87. this instance. In the systemstate MMON also holds a TT lock but on
    
88. TT-7FFFFFFF-00000001.
    
89. 
    
90. -The systemstate from node 3 shows a process waiting on TT-0
    
91. -The systemstate from node 2 shows a process holding TT-0 , but this
    
92. systemstate is dumped a little later than the process dump and is for another
    
93. 'alter database open' attempt.
    
94. 
    
95. - Based on the stack reference to kjsdrmwt asked ct. to disable DRM i.e. set:
    
96. 
    
97.    _gc_affinity_time=0
    
98.    _gc_undo_affinity=false
    
99. 
    
100. - The problem reproduces with _gc_affinity_time=0.
     
101. - Still waiting for ct. to test with  _gc_undo_affinity=false.
     
102. - Attempted to get systemstates level 266 , but the stacks didn't get
     
103. printed.
     
104. 
     
105. =========================   
     
106. WORKAROUND:
     
107. 
     
108. None.
     
109. 
     
110. =========================   
     
111. RELATED BUGS:
     
112. 
     
113. A couple of similar bugs, but fixed in 10.2.0.1:
     
114. 
     
115. bug:3523133
     
116. bug:3838679
     
117. 
     
118. 
     
119. =========================   
     
120. REPRODUCIBILITY:
     
121. 1. State if the problem is reproducible; indicate where and predictability
     
122. 
     
123. - Seems to be reproducable.
     
124. 
     
125. 2. List the versions in which the problem has reproduced
     
126. 
     
127. 3. List any versions in which the problem has not reproduced
     
128. 
     
129. There is no 10.2.0.2 available yet for this platform.

复制代码

GES: Potential blocker (pid=29817) on resource TT-0x0-0x0;
enqueue info in file /db/trace/bdump/ocoredb1_lmd0_21047.trc and DIAG trace
file

需要 pid=29817 stack call 来确认该问题