0 积分	1 好友	1 主题

发消息

[Exadata硬件] 脑裂处理方案（总算写完了~~）

1^#

发表于 2014-4-14 14:00:09 | 查看: 6671| 回复: 6

本帖最后由 373797651@qq.co 于 2014-4-14 17:23 编辑

1. 客户问题描述
1.1. 故障描述：
2014年04月03日13点30分，有业务部门说xxx无法连接，所以立即登录系统查看，发现双节点的RAC环境中节点2上的instance宕掉了，所以尝试启动node2，但是恢复了近20分也没起来~并且还影响到了node1，所以当机立断强制杀掉node2的实例，然后重启node1,万幸node1起来了，尝试重启node2也起来了；

1.2. 环境描述：
OS: AIX 5.3.6(RAC)
ORACLE: 9.0.2.8

2. 排查
2.1. 分析日志：
Node1:
Thu Apr  3 13:30:38 2014
Waiting for clusterware split-brain resolution
Evicting instance 2 from cluster

Node2:
Thu Apr  3 13:14:46 2014
IPC Send timeout detected. Sender ospid 4792512
Thu Apr  3 13:14:48 2014
Communications reconfiguration: instance 0
Thu Apr  3 13:16:33 2014
Waiting for IPC send timeout resolution
Thu Apr  3 13:16:33 2014
Waiting for clusterware split-brain resolution

Thu Apr  3 13:26:33 2014
Errors in file /oracle/admin/fwsk/bdump/node2_lmon_561382.trc:
ORA-29740: evicted by member 1, group incarnation 3
LMON: terminating instance due to error 29740
Thu Apr  3 13:26:33 2014
Errors in file /oracle/admin/fwsk/bdump/node2_lck0_606258.trc:
ORA-29740: evicted by member , group incarnation
Thu Apr  3 13:26:33 2014
Errors in file /oracle/admin/fwsk/bdump/node2_lmd0_380928.trc:
ORA-29740: evicted by member , group incarnation
Thu Apr  3 13:26:33 2014
Errors in file /oracle/admin/fwsk/bdump/node2_lms1_372798.trc:
ORA-29740: evicted by member , group incarnation

Node2 13:14:46检测到IPC send timeout
Node2 13:14:48 Communications reconfiguration:instance 0,instance 0表示实例关闭
Node2 13:16:33集群发生脑裂，clusterware split-brain
Node1 13:30:38集群发生脑裂，clusterware split-brain
Node1 13:30:38 node2 被驱逐 Evicting instance 2 from cluster

2.2. 分析trc
*** 2014-04-03 13:14:48.344
kjxgrcomerr: Communications reconfig: instance 0 (2,2)
Submitting asynchronized dump request [2]
kjxgrrcfgchk: Initiating reconfig, reason 3
*** 2014-04-03 13:14:53.463
kjxgmrcfg: Reconfiguration started, reason 3

在node2的系统时间为：
13:14:48  Initiating reconfig,reason 3
            Reconfiguration started,reason 3，这里的reason 3是重要信息

*** 2014-04-03 13:16:33.780
kjxgrrecp2: Waiting for split-brain resolution, upd 1, seq 3
*** 2014-04-03 13:26:33.810
Voting results, upd 1, seq 3, bitmap: 0
*** 2014-04-03 13:26:33.810
kjxgrdtrt: Evicted by 1, seq (3, 3)

在node2的系统时间为：
13:16:33  Waiting for split-brain resolution，此时cluster正等待解决脑裂
13:16:33  Evicted by 1，此时节点2上的instance被驱逐

2.3. 查询matelink
依据metalink ID 1003644.1
For troubleshooting ORA-29740 errors, the 'reason' will be very important.In the above example,the first section indicates the reason for theinitiated reconfiguration.  The reasons are as follows:

Reason 0 = No reconfiguration
Reason 1 = The Node Monitor generated the reconfiguration.
Reason 2 = An instance death was detected.
Reason 3 = Communications Failure
Reason 4 = Reconfiguration after suspend

引起 reason 3的几种可能性：
a) The LMON processes loose communication between one another.
b) One instance loses communications with the LMD process of another instance.
c) An LMON process is blocked, spinning, or stuck and is notresponding to the other instance(s) LMON process.
d) An LMD process is blocked or spinning.

引起ORA-29740的几种情况
a) Network Problems.
b) Resource Starvation (CPU, I/O, etc..)
c) Severe Contention in Database.
d) An Oracle bug.

3. 解决方案
3.1.    解决通讯问题（reason 3）
修改两个节点的系统时间（现在两个系统时间相差15分）

3.2 解决ora-29740 （参数调优）
查看网络参数：
查看所有的：
#/usr/sbin/no –a | more
分项查看：
#no -a | fgrep ipqmaxlen
#no -a | fgrep rfc1323
#no -a | fgrep sb_max
#no -a | fgrep tcp_recvspace
#no -a | fgrep tcp_sendspace
#no -a | fgrep udp_recvspace
#no -a | fgrep udp_sendspace
查看下面这些网络参数：
Network Preparation
=======================================
PARAMETER RECOMMENDED    VALUE
ipqmaxlen                512
rfc1323                  1
sb_max                  1500000
tcp_recvspace             65536
tcp_sendspace             65536
udp_recvspace            1351680
udp_sendspace            135168

注意：
udp_recvspace：应该是udp_sendspace的10倍，但是必须小于sb_max
udp_sendspace：这个值至少应该是4K＋（db_block_size*db_multiblock_read_count）的大小。

若有值不满足，进行修改:
no -r -o ipqmaxlen=512
no -p -o rfc1323=1
no -p -o sb_max=1620000
no -p -o tcp_recvspace=65536
no -p -o tcp_sendspace=65536
no -p -o udp_recvspace=1351680
no -p -o udp_sendspace=135168

"set sq_max_size=20" added to /etc/system
ndd -set /dev/udp udp_xmit_hiwat 65536
ndd -set /dev/udp udp_recv_hiwat 65536
ndd -set /dev/udp udp_max_buf 1048576

分享0