RAC 10g LMON进程关闭一个节点 后有自动重启

黑豆

alert的日志如下：

Tue Apr 16 15:28:17 BEIST 2013
LMS2 (ospid: 28049554) is not heartbeating for 214 seconds.
LMON detects unhealthy receivers.
Please check LMON and DIAG trace files for detail.
Tue Apr 16 15:28:24 BEIST 2013
LMON (ospid: 6946994) is terminating the instance.
LMON: terminating instance due to error 481
Instance terminated by LMON, pid = 6946994
Tue Apr 16 15:28:39 BEIST 2013
Starting ORACLE instance (normal)
sskgpgetexecname failed to get name
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 en1 10.1.2.0 configured from OCR for use as a cluster interconnect
Interface type 1 en0 172.16.5.0 configured from OCR for use as  a public interface
Picked latch-free SCN scheme 3
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.5.0.
System parameters with non-default values:
  processes                = 1000


LMON的log如下：
*** 2013-04-16 15:28:17.700
kjfmrcvrchk: receiver LMS[1] has no heartbeat for 214 sec (1366097083.1366097297.0).
kjfmrcvrchk: receiver LMS[1] not in running mode
kjfmrcvrchk: Dumping callstack of lms1
Submitting asynchronized dump request [20]
kjfmrcvrchk: receiver LMS[2] has no heartbeat for 214 sec (1366097083.1366097297.0).
kjfmrcvrchk: Dumping callstack of lms2
Submitting asynchronized dump request [20]
kjfmrcvrchk: receiver LMS[3] has no heartbeat for 223 sec (1366097074.1366097297.0).
kjfmrcvrchk: receiver LMS[3] not in running mode
kjfmrcvrchk: Dumping callstack of lms3
Submitting asynchronized dump request [20]
kjfmrcvrchk: receivers are not healthy. kill instance.
ksuitm: waiting up to [5] seconds before killing DIAG(29819032)


lms2的log如下：
*** 2013-04-16 15:00:51.611
DRM(669) win(1) lms 2 finished replaying gcs resources
lms 2 finished fixing gcs write protocol
DRM(669) win(2) lms 2 finished replaying gcs resources
lms 2 finished fixing gcs write protocol
DRM(669) win(3) lms 2 finished replaying gcs resources
lms 2 finished fixing gcs write protocol
DRM(669) win(4) lms 2 finished replaying gcs resources
lms 2 finished fixing gcs write protocol
DRM(669) win(5) lms 2 finished replaying gcs resources
lms 2 finished fixing gcs write protocol
DRM(669) win(6) lms 2 finished replaying gcs resources
lms 2 finished fixing gcs write protocol
DRM(669) win(7) lms 2 finished replaying gcs resources
lms 2 finished fixing gcs write protocol
DRM(669) win(8) lms 2 finished replaying gcs resources
lms 2 finished fixing gcs write protocol
*** 2013-04-16 15:27:16.182
KJM_HISTORY: RCVR STALL OP(20) context 34 elapsed 67517223 us
KJM HIST LMS2:
  20:34:67517223 20:34:1095926 20:34:4314184 20:34:2373912 20:34:10399973 20:34:45501047 20:34:55987 20:34:2285 20:34:12 20:34:124337
  20:34:157169 20:34:35572 20:34:1261345 20:34:1036929 20:34:4257077 20:34:195782 20:34:4882022 20:34:362351 20:34:1928676 20:33:30
  20:44:458022 1:1454462 14:32:1040714 1:1 14:36:51 1:507757 12:10:349863 7:38 6:0 10:1
  17:4 16:1224065 15:1 14:32:16653 1:23 13:65518:17945116 20:34:742776 20:34:149480 20:34:3518201 20:34:2368635
  20:34:1031444 20:34:5193234 20:34:4941342 1:46 12:2:1064198 7:1 6:0 10:0 17:1 16:1
  15:0 12:29315 7:0 6:0 10:1 17:1 16:0 15:1 12:29315 7:1
  6:0 10:0 17:2 16:1
----------------------------------------
SO: 700000108568648, type: 4, owner: 70000010a3bbc68, flag: INIT/-/-/0x00
  (session) sid: 1098 trans: 0, creator: 70000010a3bbc68, flag: (51) USR/- BSY/-/-/-/-/-
            DID: 0000-0000-00000000, short-term DID: 0000-0000-00000000
            txn branch: 0
            oct: 0, prv: 0, sql: 0, psql: 0, user: 0/SYS
  last wait for 'latch: gcs resource hash' wait_time=0.090439 sec, seconds since wait started=160
          address=70000010cbd2ac0, number=56, tries=2
          blocking sess=0x0 seq=9343
  Dumping Session Wait History
   for 'latch: gcs resource hash' count=1 wait_time=0.090439 sec
          address=70000010cbd2ac0, number=56, tries=2
   for 'latch: gcs resource hash' count=1 wait_time=0.292981 sec
          address=70000010cbd2ac0, number=56, tries=1
   for 'latch: gcs resource hash' count=1 wait_time=0.292984 sec
          address=70000010cbd2ac0, number=56, tries=0
   for 'gcs remote message' count=1 wait_time=1.064180 sec
          waittime=18, poll=0, event=0
   for 'gcs remote message' count=1 wait_time=0.029312 sec
          waittime=18, poll=0, event=0
   for 'gcs remote message' count=1 wait_time=0.029313 sec
          waittime=18, poll=0, event=0
   for 'gcs remote message' count=1 wait_time=0.029314 sec
          waittime=18, poll=0, event=0
   for 'gcs remote message' count=1 wait_time=0.029313 sec
          waittime=18, poll=0, event=0
   for 'gcs remote message' count=1 wait_time=0.029317 sec
          waittime=18, poll=0, event=0
   for 'gcs remote message' count=1 wait_time=0.029326 sec
          waittime=18, poll=0, event=0


                                                                      15:25:23]
                                                           15:25:16 - 15:25:22]
  temporary object counter: 0
KTU Session Commit Cache Dump for IDLs:
KTU Session Commit Cache Dump for Non-IDLs:
----------------------------------------
UOL used : 0 locks(used=0, free=0)
KGX Atomic Operation Log 70000010cbf0898
Mutex 0(0, 0) idn 0 oper NONE
Cursor Pin uid 1098 efd 5 whr 10 slp 0
KGX Atomic Operation Log 70000010cbf08e0
Mutex 0(0, 0) idn 0 oper NONE
Library Cache uid 1098 efd 0 whr 0 slp 0
KGX Atomic Operation Log 70000010cbf0928
Mutex 0(0, 0) idn 0 oper NONE
Library Cache uid 1098 efd 0 whr 0 slp 0
*** 2013-04-16 15:27:49.142
KJM_HISTORY: RCVR STALL OP(13) context 65518 elapsed 178146896 us
KJM HIST LMS2:
  13:65518:178146896 20:34:759550 20:34:1903142 20:34:6229745 20:34:21981 20:34:206482 20:34:1019122 20:34:1854645 20:34:1168788 20:34:631279
  20:34:10420027 20:34:7972270 20:34:67517223 20:34:1095926 20:34:4314184 20:34:2373912 20:34:10399973 20:34:45501047 20:34:55987 20:34:2285
  20:34:12 20:34:124337 20:34:157169 20:34:35572 20:34:1261345 20:34:1036929 20:34:4257077 20:34:195782 20:34:4882022 20:34:362351
  20:34:1928676 20:33:30 20:44:458022 1:1454462 14:32:1040714 1:1 14:36:51 1:507757 12:10:349863 7:38
  6:0 10:1 17:4 16:1224065 15:1 14:32:16653 1:23 13:65518:17945116 20:34:742776 20:34:149480
  20:34:3518201 20:34:2368635 20:34:1031444 20:34:5193234 20:34:4941342 1:46 12:2:1064198 7:1 6:0 10:0
  17:1 16:1 15:0 12:29315
----------------------------------------
SO: 700000108568648, type: 4, owner: 70000010a3bbc68, flag: INIT/-/-/0x00
  (session) sid: 1098 trans: 0, creator: 70000010a3bbc68, flag: (51) USR/- BSY/-/-/-/-/-
            DID: 0000-0000-00000000, short-term DID: 0000-0000-00000000
            txn branch: 0
            oct: 0, prv: 0, sql: 0, psql: 0, user: 0/SYS
  last wait for 'latch: gcs resource hash' wait_time=0.018692 sec, seconds since wait started=9
          address=70000010cbd4140, number=56, tries=3
          blocking sess=0x0 seq=9347
  Dumping Session Wait History
   for 'latch: gcs resource hash' count=1 wait_time=0.018692 sec
          address=70000010cbd4140, number=56, tries=3
   for 'latch: gcs resource hash' count=1 wait_time=0.292979 sec
          address=70000010cbd4140, number=56, tries=2
   for 'latch: gcs resource hash' count=1 wait_time=0.292981 sec
          address=70000010cbd4140, number=56, tries=1
   for 'latch: gcs resource hash' count=1 wait_time=0.292982 sec
          address=70000010cbd4140, number=56, tries=0
   for 'latch: gcs resource hash' count=1 wait_time=0.023082 sec
          address=70000010cbd4140, number=56, tries=0
   for 'latch: gcs resource hash' count=1 wait_time=0.090439 sec
          address=70000010cbd2ac0, number=56, tries=2
   for 'latch: gcs resource hash' count=1 wait_time=0.292981 sec
          address=70000010cbd2ac0, number=56, tries=1
   for 'latch: gcs resource hash' count=1 wait_time=0.292984 sec
          address=70000010cbd2ac0, number=56, tries=0
   for 'gcs remote message' count=1 wait_time=1.064180 sec
          waittime=18, poll=0, event=0
   for 'gcs remote message' count=1 wait_time=0.029312 sec
          waittime=18, poll=0, event=0


                                                           15:27:37 - 15:27:40]
                                                           15:27:34 - 15:27:36]
                                                                      15:27:33]
                                                           15:25:49 - 15:27:32]
  temporary object counter: 0
KTU Session Commit Cache Dump for IDLs:
KTU Session Commit Cache Dump for Non-IDLs:
----------------------------------------
UOL used : 0 locks(used=0, free=0)
KGX Atomic Operation Log 70000010cbf0898
Mutex 0(0, 0) idn 0 oper NONE
Cursor Pin uid 1098 efd 5 whr 10 slp 0
KGX Atomic Operation Log 70000010cbf08e0
Mutex 0(0, 0) idn 0 oper NONE
Library Cache uid 1098 efd 0 whr 0 slp 0
KGX Atomic Operation Log 70000010cbf0928
Mutex 0(0, 0) idn 0 oper NONE
Library Cache uid 1098 efd 0 whr 0 slp 0



请指点

Maclean Liu(刘相兵

版本也不写， 你提问过很多次了， 每次提问都是 这样。 我觉得你不适合做IT

黑豆

这次提问的时候我看了，alert里有这个信息Starting up ORACLE RDBMS Version: 10.2.0.5.0.
所以没有写。我敬重技术很牛的人，但是您这么说，想表达什么？

Maclean Liu(刘相兵

关于上述评价 我是看了帖子后，一时之气 可能评价不恰， 相关的话 我收回。

可能有时候我是一个 完美主义者，所以有些东西 我看不对眼的 就要说。

自始至终 我都希望 IT人之间 有一定的默契 可以 流畅地交流， 无需过多的 题外话来告诉你 缺这少哪 ， 但在中国 这似乎始终行不通。

黑豆

相关知识：
RAC的集群状态维护是由LMON进程提供，进程提供了CGS和NM两个服务。

IMR是由CGS提供的重构机制，用于确认实例之间连通性、快速的排除故障节点以减少对数据的损害，这个过程中，每个实例都需要做出投票。

重构触发类型有：
network hearbead异常；
节点加入或离开集群；
controlfile heatbeat异常。


集成商给予的解决办法：
此版本为10.2.0.5.0，需要更新新的PSU(Patch Set Update Patches)，未说明bug号，继续咨询中

Maclean Liu(刘相兵

虽然我没有深入看这个问题， 但我建议你把 DRM 关掉

clevernby

黑豆 发表于 2013-4-16 17:01
这次提问的时候我看了，alert里有这个信息Starting up ORACLE RDBMS Version: 10.2.0.5.0.
所以没有写。我 ...

其实这里的版本信息是远远不够的，没有32位还是64位、没有安装过哪些CPU/PSU、没有安装过哪些Patch、没有clusterware的信息、没有操作系统版本的信息。
如果您能够在第一次提问时就把这些内容一一清晰的罗列出来，那可以省去后续反复沟通的过程，而这些反复地沟通只是为了确认环境，这多亏啊~沟通是需要成本的，把它花在刀刃上吧