11g R2 RAC其中一个节点自动重启

hahahehe775

前两天在公司装了一套ORACLE 11G R2 11.2.0.3的RAC ，装完之后就没有再看过，数据现在还没导进去。

RAC2中 有tomcat 公司某几个部门在使用，下午17:40-17:45之间有人反映RAC2 这台机器挂掉了。
看了/var/log/memsage 机器确实重启了。

看了OS对系统日志 和 GRID的日志也没分析出来为什么，请大师指点一下 什么情况。


CENTOS Linux 5.7
附件是日志传不上来了，用的快盘的

rac2.zip
http://www.kuaipan.cn/file/id_12116643907900495.htm


rac1.zip
http://www.kuaipan.cn/file/id_12116643907900496.htm

harryzhang

在节点1的ocssd.log


2013-05-20 17:39:42.190: [    CSSD][3844565312]clssnmPollingThread: node cloudb01 (2) at 50% heartbeat fatal, removal in 14.250 seconds
2013-05-20 17:39:42.190: [    CSSD][3844565312]clssnmPollingThread: node cloudb01 (2) is impending reconfig, flag 2294796, misstime 15750
2013-05-20 17:39:42.190: [    CSSD][3844565312]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)
2013-05-20 17:39:42.191: [    CSSD][3838232896]clssnmvDHBValidateNCopy: node 2, cloudb01, has a disk HB, but no network HB, DHB has rcfg 264106997, wrtcnt, 411217, LATS 427359444, lastSeqNo 73191, uniqueness 1368682883, timestamp 1369042767/801477684
2013-05-20 17:39:42.388: [    CSSD][3846142272]clssnmSendingThread: sending status msg to all nodes
2013-05-20 17:39:42.388: [    CSSD][3846142272]clssnmSendingThread: sent 5 status msgs to all nodes
2013-05-20 17:39:47.398: [    CSSD][3846142272]clssnmSendingThread: sending status msg to all nodes
2013-05-20 17:39:47.398: [    CSSD][3846142272]clssnmSendingThread: sent 5 status msgs to all nodes
2013-05-20 17:39:49.205: [    CSSD][3844565312]clssnmPollingThread: node cloudb01 (2) at 75% heartbeat fatal, removal in 7.240 seconds
2013-05-20 17:39:51.406: [    CSSD][3846142272]clssnmSendingThread: sending status msg to all nodes
2013-05-20 17:39:51.407: [    CSSD][3846142272]clssnmSendingThread: sent 4 status msgs to all nodes
2013-05-20 17:39:54.215: [    CSSD][3844565312]clssnmPollingThread: node cloudb01 (2) at 90% heartbeat fatal, removal in 2.230 seconds, seedhbimpd 1
2013-05-20 17:39:55.415: [    CSSD][3846142272]clssnmSendingThread: sending status msg to all nodes
2013-05-20 17:39:55.415: [    CSSD][3846142272]clssnmSendingThread: sent 4 status msgs to all nodes
2013-05-20 17:39:56.441: [    CSSD][3844565312]clssnmPollingThread: Removal started for node cloudb01 (2), flags 0x23040c, state 3, wt4c 0
2013-05-20 17:39:56.441: [    CSSD][3844565312]clssnmMarkNodeForRemoval: node 2, cloudb01 marked for removal

在节点2的ocssd.log

2013-05-20 17:39:18.437: [    CSSD][2888833344]clssnmSendingThread: sending status msg to all nodes
2013-05-20 17:39:18.437: [    CSSD][2888833344]clssnmSendingThread: sent 4 status msgs to all nodes





2013-05-20 17:44:06.179: [    CSSD][3578307376]clsu_load_ENV_levels: Module = CSSD, LogLevel = 2, TraceLevel = 0
2013-05-20 17:44:06.180: [    CSSD][3578307376]clsu_load_ENV_levels: Module = GIPCNM, LogLevel = 2, TraceLevel = 0

看看心跳线是否有过异常。

hahahehe775

harryzhang 发表于 2013-5-21 11:45
在节点1的ocssd.log

感谢提醒，问题已经找到。节点2 在17:39-17:45 之间服务器机房做调整，导致网线松动了。节点1的心跳通往节点2时候 失败。导致节点2被提出集群。这就是所谓的脑裂？
奇怪的是 开始一直在分析重启的节点，日志中真是毫无价值信息，翻到节点1时候才看出点问题来。

harryzhang

没搞多网卡绑定么。