0 积分	1 好友	2 主题

发消息

[SQL调优] 10.2.0.4+ASM单实列驱除问题

1^#

发表于 2013-5-24 18:55:18 | 查看: 4213| 回复: 3

HI：刘大各位好！
  今天遇到一个问题ASM单实例被CSS驱除了，导致数据库宕机！
操作系统SESE10SP2，确定是单机CRSCTL CHECK CRS只有CSSD进程
PS -EF|GREP D.BIN 只有CSSD进程，数据库无RAC进程。OCRCHECK 是LOCAL，/ETC/ORACLE/OCR.LOC也是LOCAL=YES

1.数据库关键日志
Fri May 24 09:02:01 2013
Errors in file /home/oracle/product/admin/cqzjoa/bdump/cqzjoa_rbal_9310.trc:
ORA-15064: communication failure with ASM instance
Fri May 24 09:02:01 2013
Errors in file /home/oracle/product/admin/cqzjoa/bdump/cqzjoa_lgwr_9287.trc:
ORA-15064: communication failure with ASM instance

2.ASM关键日志
Fri May 24 09:02:00 2013
Errors in file /home/oracle/product/admin/+ASM/bdump/+asm_gmon_8999.trc:
ORA-29702: error occurred in Cluster Group Service operation
ORA-29702: error occurred in Cluster Group Service operation
GMON: terminating instance due to error 29702
Instance terminated by GMON, pid = 8999
Fri May 24 09:34:34 2013
Starting ORACLE instance (normal)

3.cssd关键日志

[ CSSD]2012-11-19 09:04:23.559 [158511696] >TRACE: clssgmReconfigThread:  completed for reconfig(1), with status(1)
[ CSSD]2013-05-24 08:56:39.790 [133345872] >WARNING: clssnmPollingThread: node cqzj-nw-db-01 (0) at 50 2.000000e+00artbeat fatal, eviction in 14.184 seconds
[ CSSD]2013-05-24 08:56:39.790 [133345872] >TRACE: clssnmPollingThread: node cqzj-nw-db-01 (0) is impending reconfig, flag 5, misstime 15816
[ CSSD]2013-05-24 08:56:39.790 [133345872] >TRACE: clssnmPollingThread: diskTimeout set to (27000)ms impending reconfig status(1)
[ CSSD]2013-05-24 08:56:46.717 [133345872] >WARNING: clssnmPollingThread: node cqzj-nw-db-01 (0) at 75 2.000000e+00artbeat fatal, eviction in 7.256 seconds
[ CSSD]2013-05-24 08:56:47.771 [133345872] >WARNING: clssnmPollingThread: node cqzj-nw-db-01 (0) at 75 2.000000e+00artbeat fatal, eviction in 6.204 seconds
[ CSSD]2013-05-24 08:56:51.125 [133345872] >WARNING: clssnmPollingThread: node cqzj-nw-db-01 (0) at 90 2.000000e+00artbeat fatal, eviction in 2.848 seconds
[ CSSD]2013-05-24 08:56:52.133 [133345872] >WARNING: clssnmPollingThread: node cqzj-nw-db-01 (0) at 90 2.000000e+00artbeat fatal, eviction in 1.840 seconds
[ CSSD]2013-05-24 08:56:53.140 [133345872] >WARNING: clssnmPollingThread: node cqzj-nw-db-01 (0) at 90 2.000000e+00artbeat fatal, eviction in 0.832 seconds
[ CSSD]2013-05-24 08:56:54.008 [133345872] >TRACE: clssnmPollingThread: Eviction started for node cqzj-nw-db-01 (0), flags 0x0005, state 3, wt4c 0
[ CSSD]2013-05-24 08:56:54.008 [133345872] >WARNING: clssnmPollingThread: state(5) clusterState(1) exit
[ CSSD]2013-05-24 08:56:54.008 [133345872] >ERROR: ###################################
[ CSSD]2013-05-24 08:56:54.008 [133345872] >ERROR: clssscExit: CSSD aborting
[ CSSD]2013-05-24 08:56:54.008 [133345872] >ERROR: ###################################

4.OS关键日志
ay 24 08:55:34 CQZJ-NW-DB-01 kernel: ping timeout of 5 secs expired, last rx 8312435384, last ping 8312436634, now 8312437884
May 24 08:55:34 CQZJ-NW-DB-01 kernel:  connection2:0: iscsi: detected conn error (1011)
May 24 08:56:16 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 08:56:16 CQZJ-NW-DB-01 multipathd: checker failed path 8:192 in map 3600140571ffa2ded4915d4063dbd55d1
May 24 08:56:18 CQZJ-NW-DB-01 kernel: device-mapper: multipath: Failing path 8:192.
May 24 09:00:07 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:00:07 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:00:07 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:00:07 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:01:38 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:01:38 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:01:38 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:01:51 CQZJ-NW-DB-01 iscsid: semop up failed 22
May 24 09:01:38 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:01:55 CQZJ-NW-DB-01 kernel: oracle(43763): floating-point assist fault at ip 4000000007923d62, isr 0000020000007001
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 kernel: oracle(43763): floating-point assist fault at ip 4000000007923d62, isr 0000020000003001
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 kernel: oracle(43763): floating-point assist fault at ip 4000000007923d62, isr 0000020000007001
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 kernel: oracle(43763): floating-point assist fault at ip 4000000007923d62, isr 0000020000003001
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 kernel: oracle(43763): floating-point assist fault at ip 4000000007923d62, isr 0000020000007001
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 kernel: sd 7:0:0:0: timing out command, waited 360s
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 kernel: sd 7:0:0:0: SCSI error: return code = 0x00020000
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 kernel: end_request: I/O error, dev sdm, sector 0
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:00 CQZJ-NW-DB-01 multipathd: sdm: directio checker reports path is down
May 24 09:02:01 CQZJ-NW-DB-01 su: (to oracle) root on none
May 24 09:02:03 CQZJ-NW-DB-01 iscsid: semop up failed 22

CSSD集群件在08:57发出了驱除命令，将节点驱除，ASM在9点左右由于cssd的驱除指令终止了自己的ASM服务，而所有的数据文件都是放到ASM磁盘组中，所有9：02数据库抛出了不能访问ASM实例的错误，而意外终止，整个流程如下：

8:57CSSD发出了驱除命令-----9:01ASM实例终止------9:02数据库服务终止

OS日志虽然有报错，但是经过查看SDM是没有使用到的设备.ASM的DISK没有用，文件系统也没有挂载，是用户最近才弄的，虽然有报错但是应该和数据库关系不大。但是时间很吻合。
请大家看看。现在很疑惑，单机的网络心跳是什么意思？？？

分享0