0 积分	1 好友	2 主题

发消息

[RAC性能调优] RAC相继重启，疑惑

1^#

发表于 2013-11-30 23:17:26 | 查看: 4977| 回复: 4

我遇到一个RAC的问题，通过朋友推荐，我向您请教一下，希望您能帮忙解惑，先谢谢了。
      我维护一个4节点RAC，数据库版本11.2.0.3.3 操作系统 AIX，出现4个节点相继重启的问题。

      事情的过程是这样的：
      1、node1的私网断了一段时间（20分钟左右），node1被踢出集群。
      2、node1重启（这是node1的私网还没好），重启过程中把其他节点通过IMR 给kill掉了（红色部分是node1 kill 其他节点的信息）：
见附件

node1_alert.txt (8.94 KB, 下载次数: 859)

我不明白的是，node1重启，为什么会kill其他正常节点呢？
您有遇到过类似的问题吗？这是什么原因呢？
是不是因为node1，启动过程中自己的私网不通，但他认为是其他节点有问题，所以把其他节点kill了？

分享0

收藏0 回复只看该作者道具举报

Liu Maclean(刘相兵

2^#

发表于 2013-12-1 10:22:21

Starting background process MARK
Thu Nov 28 10:44:07 2013
MARK started with pid=34, OS id=49873248
NOTE: MARK has subscribed
lmon registered with NM - instance number 1 (internal mem no 0)
Thu Nov 28 10:44:44 2013
Dumping diagnostic data in directory=[cdmp_20131128104444], requested by (instance=2, osid=55378232 (LGWR)), summary=[incident=312193].
Thu Nov 28 10:44:53 2013
LMON (ospid: 63439316) detects hung instances during IMR reconfiguration
LMON (ospid: 63439316) tries to kill the instance 4 in 20 seconds.
Please check instance 4's alert log and LMON trace file for more details.
Thu Nov 28 10:45:13 2013
Remote instance kill is issued with system inc 0
Remote instance kill map (size 1) : 4
LMON received an instance eviction notification from instance 1
The instance eviction reason is 0x20000000
The instance eviction map is 4
Thu Nov 28 10:45:15 2013
Dumping diagnostic data in directory=[cdmp_20131128104515], requested by (instance=4, osid=12124406 (PMON)), summary=[abnormal instance termination].
Thu Nov 28 10:48:18 2013
Remote instance kill is issued with system inc 66
Remote instance kill map (size 1) : 4
LMON received an instance eviction notification from instance 1
The instance eviction reason is 0x40000000
The instance eviction map is 4
Thu Nov 28 10:48:22 2013
Dumping diagnostic data in directory=[cdmp_20131128104821], requested by (instance=4, osid=53281130 (PMON)), summary=[abnormal instance termination].
Thu Nov 28 10:49:04 2013
LMON (ospid: 63439316) detects hung instances during IMR reconfiguration
LMON (ospid: 63439316) tries to kill the instance 3 in 20 seconds.
Please check instance 3's alert log and LMON trace file for more details.
Thu Nov 28 10:49:24 2013
Remote instance kill is issued with system inc 66
Remote instance kill map (size 1) : 3
LMON received an instance eviction notification from instance 1
The instance eviction reason is 0x20000000
The instance eviction map is 3
Thu Nov 28 10:50:12 2013
No connectivity to other instances in the cluster during startup. Hence, LMON is terminating the instance. Please check the LMON trace file for details. Also, please check the network logs of this instance along with clusterwide network health for problems and then re-start this instance.
LMON (ospid: 63439316): terminating the instance
Thu Nov 28 10:50:13 2013
System state dump requested by (instance=1, osid=63439316 (LMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/bjscnfzc/bjscnfzc1/trace/bjscnfzc1_diag_65536264.trc
Thu Nov 28 10:50:13 2013
ORA-1092 : opitsk aborting process
Thu Nov 28 10:50:13 2013
License high water mark = 2
Dumping diagnostic data in directory=[cdmp_20131128105012], requested by (instance=1, osid=63439316 (LMON)), summary=[abnormal instance termination].
Instance terminated by LMON, pid = 63439316
USER (ospid: 62128628): terminating the instance
Instance terminated by USER, pid = 62128628
Thu Nov 28 11:00:21 2013

复制代码

压缩打包后上传

/u01/app/oracle/diag/rdbms/bjscnfzc/bjscnfzc1/trace/bjscnfzc1_diag_65536264.trc

以及当时LMON的TRACE

回复只看该作者道具举报

Liu Maclean(刘相兵

3^#

发表于 2013-12-1 10:30:11

BUG 14135323 - RAC INSTANCE CANNOT JOIN RUNNING RAC
BUG 14550939 - LMON TERMINATING INSTANCE

没有找到确认为real bug的案例，一种思路是使用 cluster_interconnects 来避免使用HAIP，因为 HAIP有一些问题

回复只看该作者道具举报