linux crs 节点2执行root.sh不断重启问题

makaveli

现在在linux 5.8上装一套11.1.0.6的crs软件，在最后执行root.sh的时候节点一正常，但是节点2一执行就老自动重启，麻烦帮忙诊断下原因，谢谢了。

在第一次执行root.sh的时候发现节点2的时间和节点1不一致，相差了几个小时，后来手动更改过节点2的系统时间。再次在节点2执行root.sh的时候报如下错误：
Checking to see if Oracle CRS stack is already configured
Oracle CRS stack is already configured and will be running under init(1M)

根据晚上找到的解决办法删除了两个节点/etc/oracle/scls_scr/rac1/oracle/cssfatal 文件，然后在节点2上每次执行root.sh节点2就自动重启了。

Liu Maclean(刘相兵

2节点的日志不是很完整

1. [    CSSD]2012-12-12 09:49:55.833 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61801504, LAT 61786794
   
2. [    CSSD]2012-12-12 09:49:55.834 [1337203008] >TRACE:   clssgmReconfigThread:  completed for reconfig(250766476), with status(0)
   
3. [    CSSD]2012-12-12 09:49:56.835 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61802514, LAT 61786794
   
4. [    CSSD]2012-12-12 09:49:57.837 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61803514, LAT 61786794
   
5. [    CSSD]2012-12-12 09:49:58.839 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61804514, LAT 61786794
   
6. [    CSSD]2012-12-12 09:49:59.841 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61805514, LAT 61786794
   
7. [    CSSD]2012-12-12 09:50:00.843 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61806514, LAT 61786794
   
8. [    CSSD]2012-12-12 09:50:01.845 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61807514, LAT 61786794
   
9. [    CSSD]2012-12-12 09:50:02.847 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61808524, LAT 61786794
   
10. [    CSSD]2012-12-12 09:50:03.849 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61809524, LAT 61786794
    
11. [    CSSD]2012-12-12 09:50:04.851 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61810524, LAT 61786794
    
12. [    CSSD]2012-12-12 09:50:05.853 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61811524, LAT 61786794
    
13. [    CSSD]2012-12-12 09:50:06.856 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61812524, LAT 61786794
    
14. [    CSSD]2012-12-12 09:50:07.858 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61813534, LAT 61786794
    
15. [    CSSD]2012-12-12 09:50:08.861 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61814534, LAT 61786794
    
16. [    CSSD]2012-12-12 09:50:09.863 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61815534, LAT 61786794
    
17. [    CSSD]2012-12-12 09:50:10.865 [1305733440] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 0, misstime 30000, cur 61816534, LAT 61786794
    
18. [    CSSD]2012-12-12 09:50:11.127 [1305733440] >TRACE:   clssnmSetupAckWait: Ack message type (15)
    
19. [    CSSD]2012-12-12 09:50:11.127 [1305733440] >TRACE:   clssnmSetupAckWait: node(1) is ACTIVE
    
20. [    CSSD]2012-12-12 09:50:11.127 [1305733440] >TRACE:   clssnmSendUpdate: syncSeqNo(250766477)
    
21. [    CSSD]2012-12-12 09:50:11.127 [1305733440] >TRACE:   clssnmWaitForAcks: Ack message type(15), ackCount(1)
    
22. [    CSSD]2012-12-12 09:50:11.127 [1211324736] >TRACE:   clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
    
23. [    CSSD]2012-12-12 09:50:11.127 [1211324736] >TRACE:   clssnmUpdateNodeState: node 1, state (3/3) unique (1355275074/1355275074) prevConuni(0) birth (250766475/250766475) (old/new)
    
24. [    CSSD]2012-12-12 09:50:11.127 [1211324736] >TRACE:   clssnmUpdateNodeState: node 2, state (5/0) unique (1355304198/1355304198) prevConuni(1355304198) birth (250766476/250766476) (old/new)
    
25. [    CSSD]2012-12-12 09:50:11.127 [1211324736] >TRACE:   clssnmDeactivateNode: node 2 (nsbxdb2) left cluster
    
26. 
    
27. [    CSSD]2012-12-12 09:50:11.127 [1211324736] >USER:    clssnmHandleUpdate: SYNC(250766477) from node(1) completed
    
28. [    CSSD]2012-12-12 09:50:11.127 [1211324736] >USER:    clssnmHandleUpdate: NODE 1 (nsbxdb1) IS ACTIVE MEMBER OF CLUSTER
    
29. [    CSSD]2012-12-12 09:50:11.127 [1211324736] >TRACE:   clssnmHandleUpdate: diskTimeout set to (200000)ms
    
30. [    CSSD]2012-12-12 09:50:11.127 [1305733440] >TRACE:   clssnmWaitForAcks: done, msg type(15)
    
31. [    CSSD]2012-12-12 09:50:11.127 [1305733440] >TRACE:   clssnmDoSyncUpdate: Sync 250766477 complete!
    
32. [    CSSD]2012-12-12 09:50:11.128 [1337203008] >TRACE:   clssgmReconfigThread:  started for reconfig (250766477)
    
33. [    CSSD]2012-12-12 09:50:11.128 [1337203008] >USER:    NMEVENT_RECONFIG [00][00][00][02]
    
34. [    CSSD]2012-12-12 09:50:11.128 [1337203008] >TRACE:   clssgmCleanupGrocks: cleaning up grock crs_version type 2
    
35. [    CSSD]2012-12-12 09:50:11.128 [1337203008] >TRACE:   clssgmCleanupGrocks: cleaning up grock CRSDMAIN type 2
    
36. [    CSSD]2012-12-12 09:50:11.128 [1337203008] >TRACE:   clssgmCleanupGrocks: cleaning up grock EVMDMAIN type 2
    
37. [    CSSD]2012-12-12 09:50:11.128 [1337203008] >TRACE:   clssgmCleanupGrocks: cleaning up grock ocr_nsbxdb_cluster type 2
    
38. [    CSSD]2012-12-12 09:50:11.128 [1337203008] >TRACE:   clssgmCleanupGrocks: cleaning up grock _ORA_CRS_MEMBER_nsbxdb1 type 3
    
39. [    CSSD]2012-12-12 09:50:11.128 [1337203008] >TRACE:   clssgmCleanupGrocks: cleaning up grock #CSS_CLSSOMON type 2
    
40. [    CSSD]2012-12-12 09:50:11.129 [1337203008] >TRACE:   clssgmEstablishConnections: 1 nodes in cluster incarn 250766477
    
41. [    CSSD]2012-12-12 09:50:11.129 [1274263872] >TRACE:   clssgmPeerListener: connects done (1/1)
    
42. [    CSSD]2012-12-12 09:50:11.129 [1337203008] >TRACE:   clssgmEstablishMasterNode: MASTER for 250766477 is node(1) birth(250766475)
    
43. [    CSSD]2012-12-12 09:50:11.129 [1337203008] >TRACE:   clssgmMasterCMSync: Synchronizing group/lock status
    
44. [    CSSD]2012-12-12 09:50:11.130 [1337203008] >TRACE:   clssgmMasterSendDBDone: group/lock status synchronization complete
    
45. [    CSSD]CLSS-3000: reconfiguration successful, incarnation 250766477 with 1 nodes
    
46. 
    
47. [    CSSD]CLSS-3001: local node number 1, master node number 1
    
48. 
    
49. [    CSSD]2012-12-12 09:50:11.130 [1337203008] >TRACE:   clssgmReconfigThread:  completed for reconfig(250766477), with status(1)
    
50. [    CSSD]2012-12-12 10:01:01.947 [1211324736] >TRACE:   clssnmConnComplete: MSGSRC 2, type 5, node 2, flags 0x0001, con 0x2aaaac085c10, probe (nil), nodekillsz 0
    
51. [    CSSD]2012-12-12 10:01:01.947 [1211324736] >TRACE:   clssnmConnComplete: node 2, nsbxdb2, con(0x2aaaac085c10), probcon((nil)), ninfcon((nil)), node unique 1355277704, prev unique 0, msg unique 1355277704 node state 0
    
52. [    CSSD]2012-12-12 10:01:01.948 [1211324736] >TRACE:   clssnmsendConnAck: node 2, node state 0
    
53. [    CSSD]2012-12-12 10:01:01.948 [1211324736] >TRACE:   clssnmSendConnAck: connected to node 2 , ninfcon (0x2aaaac085c10), state (0)
    
54. [    CSSD]2012-12-12 10:01:01.948 [1211324736] >TRACE:   clssnmConnComplete: connecting to node 2 (con 0x2aaaac085c10), ninfcon (0x2aaaac085c10), state (0)
    
55. [    CSSD]2012-12-12 10:01:01.948 [1211324736] >TRACE:   clssnmConnComplete: ninf->killinfosz 0
    
56. [    CSSD]2012-12-12 10:01:02.225 [1305733440] >TRACE:   clssnmDoSyncUpdate: Initiating sync 250766478
    
57. [    CSSD]2012-12-12 10:01:02.225 [1305733440] >TRACE:   clssnmDoSyncUpdate: diskTimeout set to (27000)ms
    
58. [    CSSD]2012-12-12 10:01:02.225 [1305733440] >TRACE:   clssnmSetupAckWait: Ack message type (11)
    
59. [    CSSD]2012-12-12 10:01:02.225 [1305733440] >TRACE:   clssnmSetupAckWait: node(1) is ALIVE
    
60. [    CSSD]2012-12-12 10:01:02.225 [1305733440] >TRACE:   clssnmSetupAckWait: node(2) is ALIVE
    
61. [    CSSD]2012-12-12 10:01:02.225 [1305733440] >TRACE:   clssnmSendSync: syncSeqNo(250766478)
    
62. [    CSSD]2012-12-12 10:01:02.225 [1211324736] >TRACE:   clssnmHandleSync: diskTimeout set to (27000)ms
    
63. [    CSSD]2012-12-12 10:01:02.225 [1305733440] >TRACE:   clssnmWaitForAcks: Ack message type(11), ackCount(2)
    
64. [    CSSD]2012-12-12 10:01:02.226 [1211324736] >TRACE:   clssnmHandleSync: Acknowledging sync: src[1] srcName[nsbxdb1] seq[14] sync[250766478]
    
65. [    CSSD]2012-12-12 10:01:02.226 [4091973808] >USER:    NMEVENT_SUSPEND [00][00][00][02]
    
66. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmWaitForAcks: done, msg type(11)
    
67. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmNeedConfReq: No configuration to change
    
68. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmDoSyncUpdate: node(2) is transitioning from joining state to active state
    
69. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmSetupAckWait: Ack message type (13)
    
70. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmSetupAckWait: node(1) is ACTIVE
    
71. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmSetupAckWait: node(2) is ACTIVE
    
72. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmSendVote: syncSeqNo(250766478)
    
73. [    CSSD]2012-12-12 10:01:02.226 [1211324736] >TRACE:   clssnmSendVoteInfo: node(1) syncSeqNo(250766478)
    
74. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmWaitForAcks: Ack message type(13), ackCount(2)
    
75. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmWaitForAcks: done, msg type(13)
    
76. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmCheckDskInfo: Checking disk info...
    
77. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmEvict: Start
    
78. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmWaitOnEvictions: Start
    
79. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmSetupAckWait: Ack message type (15)
    
80. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmSetupAckWait: node(1) is ACTIVE
    
81. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmSetupAckWait: node(2) is ACTIVE
    
82. [    CSSD]2012-12-12 10:01:02.226 [1305733440] >TRACE:   clssnmSendUpdate: syncSeqNo(250766478)
    
83. [    CSSD]2012-12-12 10:01:02.227 [1211324736] >TRACE:   clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
    
84. [    CSSD]2012-12-12 10:01:02.227 [1211324736] >TRACE:   clssnmUpdateNodeState: node 1, state (3/3) unique (1355275074/1355275074) prevConuni(0) birth (250766475/250766475) (old/new)
    
85. [    CSSD]2012-12-12 10:01:02.227 [1211324736] >TRACE:   clssnmUpdateNodeState: node 2, state (2/3) unique (1355277704/1355277704) prevConuni(0) birth (250766478/250766478) (old/new)
    
86. [    CSSD]2012-12-12 10:01:02.227 [1211324736] >USER:    clssnmHandleUpdate: SYNC(250766478) from node(1) completed
    
87. [    CSSD]2012-12-12 10:01:02.227 [1211324736] >USER:    clssnmHandleUpdate: NODE 1 (nsbxdb1) IS ACTIVE MEMBER OF CLUSTER
    
88. [    CSSD]2012-12-12 10:01:02.227 [1211324736] >USER:    clssnmHandleUpdate: NODE 2 (nsbxdb2) IS ACTIVE MEMBER OF CLUSTER
    
89. [    CSSD]2012-12-12 10:01:02.227 [1211324736] >TRACE:   clssnmHandleUpdate: diskTimeout set to (200000)ms
    
90. [    CSSD]2012-12-12 10:01:02.227 [1305733440] >TRACE:   clssnmWaitForAcks: Ack message type(15), ackCount(1)
    
91. [    CSSD]2012-12-12 10:01:02.227 [1337203008] >TRACE:   clssgmReconfigThread:  started for reconfig (250766478)
    
92. [    CSSD]2012-12-12 10:01:02.227 [1337203008] >USER:    NMEVENT_RECONFIG [00][00][00][06]
    
93. [    CSSD]2012-12-12 10:01:02.227 [1305733440] >TRACE:   clssnmWaitForAcks: done, msg type(15)
    
94. [    CSSD]2012-12-12 10:01:02.227 [1305733440] >TRACE:   clssnmDoSyncUpdate: Sync 250766478 complete!
    
95. [    CSSD]2012-12-12 10:01:02.228 [1337203008] >TRACE:   clssgmEstablishConnections: 2 nodes in cluster incarn 250766478
    
96. [    CSSD]2012-12-12 10:01:02.228 [1274263872] >TRACE:   clsc_send_msg: (0x2aaaac08b290) NS err (12571, 12560), transport (513, 113, 0)
    
97. 
    
98. [    CSSD]2012-12-12 10:01:02.228 [1274263872] >ERROR:   clssgmPeerEventHndlr: send to node 2 failed in connect completion, con (0x2aaaac08b9f0), rc 3
    
99. [    CSSD]2012-12-12 10:04:07.255 [1284753728] >WARNING: clssnmPollingThread: node nsbxdb2 (2) at 50% heartbeat fatal, eviction in 14.140 seconds
    
100. [    CSSD]2012-12-12 10:04:07.255 [1284753728] >TRACE:   clssnmPollingThread: node nsbxdb2 (2) is impending reconfig, flag 1037, misstime 15860
     
101. [    CSSD]2012-12-12 10:04:07.255 [1284753728] >TRACE:   clssnmPollingThread: diskTimeout set to (27000)ms impending reconfig status(1)
     
102. [    CSSD]2012-12-12 10:04:14.249 [1284753728] >WARNING: clssnmPollingThread: node nsbxdb2 (2) at 75% heartbeat fatal, eviction in 7.150 seconds
     
103. [    CSSD]2012-12-12 10:04:15.252 [1284753728] >WARNING: clssnmPollingThread: node nsbxdb2 (2) at 75% heartbeat fatal, eviction in 6.150 seconds
     
104. [    CSSD]2012-12-12 10:04:19.251 [1284753728] >WARNING: clssnmPollingThread: node nsbxdb2 (2) at 90% heartbeat fatal, eviction in 2.150 seconds
     
105. [    CSSD]2012-12-12 10:04:20.253 [1284753728] >WARNING: clssnmPollingThread: node nsbxdb2 (2) at 90% heartbeat fatal, eviction in 1.150 seconds
     
106. [    CSSD]2012-12-12 10:04:21.255 [1284753728] >WARNING: clssnmPollingThread: node nsbxdb2 (2) at 90% heartbeat fatal, eviction in 0.150 seconds
     
107. [    CSSD]2012-12-12 10:04:21.406 [1284753728] >TRACE:   clssnmPollingThread: Eviction started for node nsbxdb2 (2), flags 0x040d, state 3, wt4c 0
     
108. [    CSSD]2012-12-12 10:04:21.406 [1305733440] >TRACE:   clssnmDoSyncUpdate: Initiating sync 250766479
     
109. [    CSSD]2012-12-12 10:04:21.406 [1305733440] >TRACE:   clssnmDoSyncUpdate: diskTimeout set to (27000)ms
     
110. [    CSSD]2012-12-12 10:04:21.406 [1305733440] >TRACE:   clssnmSetupAckWait: Ack message type (11)
     
111. [    CSSD]2012-12-12 10:04:21.406 [1305733440] >TRACE:   clssnmSetupAckWait: node(1) is ALIVE

复制代码

就日志看 2节点的heartbeat一直有问题

建议你检查 私有心跳网络配置， 并彻底重装一次； 或者直接安装 clusteware 11.2.0.3 而不要装11gR1

makaveli

Liu Maclean(刘相兵 发表于 2012-12-12 14:49
2节点的日志不是很完整就日志看 2节点的heartbeat一直有问题

建议你检查 私有心跳网络配置， 并彻底重装一 ...

谢谢ml，心跳确实没问题，都能正常ping通，延迟也不高，最后我重装了，一切ok了就。