40 积分	0 好友	14 主题

发消息

[备份恢复] linux 5.4 +10.2.0.4 rac 其中一个节点莫名down掉

1^#

发表于 2013-10-21 22:21:39 | 查看: 4569| 回复: 5

本帖最后由 taoy_2008 于 2013-10-25 19:17 编辑

环境为 redhat  linux 5.4 +10.2.0.4 rac+asm

  ocr  vote 放在  /dev/raw/raw1-5 . 数据放在asm上.

在10-21 16点的时候提示其中一节点莫名down掉

  ocssd.log

[ CSSD]2013-10-18 13:56:31.434 [1285069120] >TRACE: clssgmReconfigThread:  completed for reconfig(2), with status(1)
[ CSSD]2013-10-18 13:56:31.520 [1243109696] >TRACE: clssgmCommonAddMember: clsomon joined (1/0x1000000/#CSS_CLSSOMON)
[ CSSD]2013-10-21 16:39:15.998 [1253599552] >WARNING: clssnmPollingThread: node bdhis2 (2) at 50 1.920796e-315artbeat fatal, eviction in 29.590 seconds
[ CSSD]2013-10-21 16:39:15.999 [1253599552] >TRACE: clssnmPollingThread: node bdhis2 (2) is impending reconfig, flag 1037, misstime 30410
[ CSSD]2013-10-21 16:39:15.999 [1253599552] >TRACE: clssnmPollingThread: diskTimeout set to (57000)ms impending reconfig status(1)
[ CSSD]2013-10-21 16:39:17.000 [1253599552] >WARNING: clssnmPollingThread: node bdhis2 (2) at 50 1.920796e-315artbeat fatal, eviction in 28.580 seconds
[ CSSD]2013-10-21 16:39:30.996 [1253599552] >WARNING: clssnmPollingThread: node bdhis2 (2) at 75 1.920796e-315artbeat fatal, eviction in 14.590 seconds
[ CSSD]2013-10-21 16:39:31.998 [1253599552] >WARNING: clssnmPollingThread: node bdhis2 (2) at 75 1.920796e-315artbeat fatal, eviction in 13.590 seconds
[ CSSD]2013-10-21 16:39:39.993 [1253599552] >WARNING: clssnmPollingThread: node bdhis2 (2) at 90 1.920796e-315artbeat fatal, eviction in 5.590 seconds
[ CSSD]2013-10-21 16:39:40.995 [1253599552] >WARNING: clssnmPollingThread: node bdhis2 (2) at 90 1.482197e-323artbeat fatal, eviction in 4.590 seconds
[ CSSD]2013-10-21 16:39:41.997 [1253599552] >WARNING: clssnmPollingThread: node bdhis2 (2) at 90 1.482197e-323artbeat fatal, eviction in 3.590 seconds
[ CSSD]2013-10-21 16:39:42.999 [1253599552] >WARNING: clssnmPollingThread: node bdhis2 (2) at 90 1.482197e-323artbeat fatal, eviction in 2.590 seconds
[ CSSD]2013-10-21 16:39:44.000 [1253599552] >WARNING: clssnmPollingThread: node bdhis2 (2) at 90 1.482197e-323artbeat fatal, eviction in 1.580 seconds
[ CSSD]2013-10-21 16:39:44.992 [1253599552] >WARNING: clssnmPollingThread: node bdhis2 (2) at 90 1.482197e-323artbeat fatal, eviction in 0.590 seconds
[ CSSD]2013-10-21 16:39:45.584 [1253599552] >TRACE: clssnmPollingThread: Eviction started for node bdhis2 (2), flags 0x040d, state 3, wt4c 0
[ CSSD]2013-10-21 16:39:45.584 [1274579264] >TRACE: clssnmDoSyncUpdate: Initiating sync 3
[ CSSD]2013-10-21 16:39:45.584 [1274579264] >TRACE: clssnmDoSyncUpdate: diskTimeout set to (57000)ms
[ CSSD]2013-10-21 16:39:45.584 [1274579264] >TRACE: clssnmSetupAckWait: Ack message type (11)
[ CSSD]2013-10-21 16:39:45.584 [1274579264] >TRACE: clssnmSetupAckWait: node(1) is ALIVE
[ CSSD]2013-10-21 16:39:45.584 [1274579264] >TRACE: clssnmSendSync: syncSeqNo(3)
[ CSSD]2013-10-21 16:39:45.584 [1274579264] >TRACE: clssnmWaitForAcks: Ack message type(11), ackCount(1)
[ CSSD]2013-10-21 16:39:45.584 [1201150272] >TRACE: clssnmHandleSync: diskTimeout set to (57000)ms
[ CSSD]2013-10-21 16:39:45.584 [1201150272] >TRACE: clssnmHandleSync: Acknowledging sync: src[1] srcName[bdhis1] seq[1] sync[3]
[ CSSD]2013-10-21 16:39:45.584 [1274579264] >TRACE: clssnmWaitForAcks: done, msg type(11)
[ CSSD]2013-10-21 16:39:45.584 [40895728] >USER: NMEVENT_SUSPEND [00][00][00][06]
[ CSSD]2013-10-21 16:39:45.584 [1274579264] >TRACE: clssnmDoSyncUpdate: Terminating node 2, bdhis2, misstime(60000) state(5)
[ CSSD]2013-10-21 16:39:45.584 [1274579264] >TRACE: clssnmSetupAckWait: Ack message type (13)
[ CSSD]2013-10-21 16:39:45.584 [1274579264] >TRACE: clssnmSetupAckWait: node(1) is ACTIVE
[ CSSD]2013-10-21 16:39:45.585 [1274579264] >TRACE: clssnmWaitForAcks: Ack message type(13), ackCount(1)
[ CSSD]2013-10-21 16:39:45.585 [1201150272] >TRACE: clssnmSendVoteInfo: node(1) syncSeqNo(3)
[ CSSD]2013-10-21 16:39:45.585 [1274579264] >TRACE: clssnmWaitForAcks: done, msg type(13)
[ CSSD]2013-10-21 16:39:45.585 [1274579264] >TRACE: clssnmCheckDskInfo: Checking disk info...
[ CSSD]2013-10-21 16:39:45.585 [1274579264] >TRACE: clssnmCheckDskInfo: node 2, bdhis2, state 5 with leader 2 has smaller cluster size 1; my cluster size 1 with leader 1
[ CSSD]2013-10-21 16:39:45.585 [1274579264] >TRACE: clssnmEvict: Start
[ CSSD]2013-10-21 16:39:45.585 [1274579264] >TRACE: clssnmEvict: Evicting node 2, bdhis2, birth 1, death 3, impendingrcfg 1, stateflags 0x40d
[ CSSD]2013-10-21 16:39:45.585 [1274579264] >TRACE: clssnmSendShutdown: req to node 2, kill time 268320254
[ CSSD]2013-10-21 16:39:45.585 [1274579264] >TRACE: clssnmDiscHelper: bdhis2, node(2) connection failed, con (0x174564a0), probe((nil))
[ CSSD]2013-10-21 16:39:45.585 [1274579264] >TRACE: clssnmWaitOnEvictions: Start
[ CSSD]2013-10-21 16:39:45.585 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:45.593 [1138211136] >TRACE: clssnmReadDskHeartbeat: node(2) is down. rcfg(3) wrtcnt(268506) LATS(268320264) Disk lastSeqNo(268506)
[ CSSD]2013-10-21 16:39:45.593 [1148700992] >TRACE: clssnmReadDskHeartbeat: node(2) is down. rcfg(3) wrtcnt(268506) LATS(268320264) Disk lastSeqNo(268506)
[ CSSD]2013-10-21 16:39:45.593 [1127721280] >TRACE: clssnmReadDskHeartbeat: node(2) is down. rcfg(3) wrtcnt(268506) LATS(268320264) Disk lastSeqNo(268506)
[ CSSD]2013-10-21 16:39:46.587 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:47.589 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:48.591 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:49.592 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:50.594 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:51.596 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:52.598 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:53.600 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:54.602 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:55.604 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:56.605 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:57.607 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:58.609 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:39:59.611 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:00.613 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:01.615 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:02.616 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:03.618 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:04.620 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:05.622 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:06.624 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:07.626 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:08.628 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:09.629 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:10.631 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:11.633 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:12.635 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:13.637 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:14.639 [1274579264] >TRACE: clssnmWaitOnEvictions: node 2, bdhis2, undead 1
[ CSSD]2013-10-21 16:40:15.640 [1274579264] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 1, misstime 60000, cur 268350314, LAT 268320264
[ CSSD]2013-10-21 16:40:15.641 [1274579264] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 1, misstime 60000, cur 268350314, LAT 268320264
[ CSSD]2013-10-21 16:40:16.642 [1274579264] >WARNING: clssnmWaitOnEvictions: no killecho, node 2, undead 1,
[

分享0