btxgua

0 积分	1 好友	3 主题

发消息

[Exadata常规管理] 11.2.0.4.2 RAC主机频繁重启

1^#

发表于 2014-8-8 14:54:39 | 查看: 7346| 回复: 8

本帖最后由 btxgua 于 2014-8-8 15:02 编辑

操作系统:AIX 6107
#oslevel -r
6100-07
数据库：11.2.0.4.2 ，2节点RAC
Veritas文件系统为6.1版本
#vxlicrep | grep Version
Version                            = 6.1
Version                            = 6.1
Version                            = 6.1
Version                            = 6.1
Version                            = 6.1

现象是2节点突然重启了几次了。日志中包含多次重启，请帮忙分析一下2014-08-04 07:49:36时间的重启，可能相对情况比较详细一点。在这个时间点上的业务量为0，没有任何人操作数据库。8月8日的重启是模拟的，请忽略。

ocsndb1.rar

2.75 MB, 下载次数: 859

ocsndb2.rar

2.13 MB, 下载次数: 826

分享0

收藏0 回复只看该作者道具举报

Maclean Liu(刘相兵

2^#

发表于 2014-8-8 15:09:42

2014-08-04 07:49:36.966: [ CSSD][2072]clssnm_skgxncheck: checking up on reconfig
2014-08-04 07:49:36.966: [ CSSD][2072]clssnm_skgxncheck: node(2) is off in supermap
2014-08-04 07:49:36.966: [ CSSD][2072]clssnm_skgxncheck: CSS daemon failed on node 2
2014-08-04 07:49:36.966: [ CSSD][2072]clssnmMarkNodeForRemoval: node 2, ocsndb2 marked for removal
2014-08-04 07:49:36.966: [ CSSD][2072]clssnmDiscHelper: ocsndb2, node(2) connection failed, endp (0), probe(0), ninf->endp 23fa790
2014-08-04 07:49:36.966: [ CSSD][2072]clssnmDiscHelper: node 2 clean up, endp (23fa790), init state 5, cur state 5
2014-08-04 07:49:36.966: [ CSSD][2072]clssnmDiscHelper: endp_nmnode (23fa790); endp(NULL).
2014-08-04 07:49:36.966: [ CSSD][2072]clssnm_skgxncheck: finished skgxn check. 1 fatality
2014-08-04 07:49:36.967: [ CSSD][5927]clssnmDoSyncUpdate: Initiating sync 300821178
2014-08-04 07:49:36.967: [ CSSD][5927]clssscCompareSwapEventValue: changed NMReconfigInProgress val 1, from -1, changes 19
2014-08-04 07:49:36.967: [ CSSD][5927]clssnmDoSyncUpdate: local disk timeout set to 597000 ms, remote disk timeout set to 597000
2014-08-04 07:49:36.967: [ CSSD][5927]clssnmDoSyncUpdate: new values for local disk timeout and remote disk timeout will take effect when the sync is completed.
2014-08-04 07:49:36.967: [ CSSD][5927]clssnmDoSyncUpdate: Starting cluster reconfig with incarnation 300821178
2014-08-04 07:49:36.967: [ CSSD][5927]clssnmSetupAckWait: Ack message type (11)
2014-08-04 07:49:36.967: [ CSSD][5927]clssnmSetupAckWait: node(1) is ALIVE
2014-08-04 07:49:36.967: [ CSSD][5927]clssnmSendSync: syncSeqNo(300821178), indicating EXADATA fence initialization complete
2014-08-04 07:49:36.967: [ CSSD][5927]List of nodes that have ACKed my sync: NULL
2014-08-04 07:49:36.967: [ CSSD][5927]clssnmSendSync: syncSeqNo(300821178)
2014-08-04 07:49:36.967: [ CSSD][5927]clssnmWaitForAcks: Ack message type(11), ackCount(1)
2014-08-04 07:49:36.967: [ CSSD][6184]clssnmHandleSync: Node ocsndb1, number 1, is EXADATA fence capable
2014-08-04 07:49:36.967: [ CSSD][6184]clssscUpdateEventValue: NMReconfigInProgress val 1, changes 20
2014-08-04 07:49:36.967: [ CSSD][6184]clssnmHandleSync: local disk timeout set to 597000 ms, remote disk timeout set to 597000
2014-08-04 07:49:36.967: [ CSSD][6184]clssnmHandleSync: initleader 1 newleader 1

回复只看该作者道具举报

Maclean Liu(刘相兵

3^#

发表于 2014-8-8 15:12:54

2014-08-04 08:05:19.734: [ CSSD][1]clsu_load_ENV_levels: Module = CSSD, LogLevel = 2, TraceLevel = 0
2014-08-04 08:05:19.734: [ CSSD][1]clsu_load_ENV_levels: Module = GIPCNM, LogLevel = 2, TraceLevel = 0
2014-08-04 08:05:19.734: [ CSSD][1]clsu_load_ENV_levels: Module = GIPCGM, LogLevel = 2, TraceLevel = 0
2014-08-04 08:05:19.734: [ CSSD][1]clsu_load_ENV_levels: Module = GIPCCM, LogLevel = 2, TraceLevel = 0
2014-08-04 08:05:19.734: [ CSSD][1]clsu_load_ENV_levels: Module = CLSF, LogLevel = 0, TraceLevel = 0
2014-08-04 08:05:19.734: [ CSSD][1]clsu_load_ENV_levels: Module = SKGFD, LogLevel = 0, TraceLevel = 0
2014-08-04 08:05:19.734: [ CSSD][1]clsu_load_ENV_levels: Module = GPNP, LogLevel = 1, TraceLevel = 0
2014-08-04 08:05:19.734: [ CSSD][1]clsu_load_ENV_levels: Module = OLR, LogLevel = 0, TraceLevel = 0
[ CSSD][1]clsugetconf : Configuration type [4].
2014-08-04 08:05:19.734: [ CSSD][1]clssscmain: Starting CSS daemon, version 11.2.0.4.0, in (clustered) mode with uniqueness value 1407110719
2014-08-04 08:05:19.735: [ CSSD][1]clssscmain: Environment is production
2014-08-04 08:05:19.735: [ CSSD][1]clssscmain: Core file size limit extended
2014-08-04 08:05:19.774: [ CSSD][1]clssscmain: GIPCHA down 0
2014-08-04 08:05:19.783: [ CSSD][1]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
2014-08-04 08:05:19.783: [ CSSD][1]clssscExtendLimits: The current soft limit for file descriptors is 4294967295, hard limit is 4294967295
2014-08-04 08:05:19.783: [ CSSD][1]clssscExtendLimits: The current soft limit for locked memory is 4294967295, hard limit is 4294967295
2014-08-04 08:05:19.784: [ CSSD][1]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
2014-08-04 08:05:19.784: [ CSSD][1]clssscSetPrivEnv: Setting priority to 4
2014-08-04 08:05:19.877: [ CSSD][1]clssscmain: Running as user grid
2014-08-04 08:05:19.880: [ CSSD][1]clssscmain: RT queue setting: ON
2014-08-04 08:05:19.895: [ CSSD][1]clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
2014-08-04 08:05:19.896: [ CSSD][1]clssscGetParameterOLR: OLR fetch for parameter diagwait (14) failed with rc 21
2014-08-04 08:05:19.902: [ CSSD][1]clssnmInitNMInfoMin: Initializing first-reconfig to (0)
2014-08-04 08:05:19.903: [ CSSD][1]clssscmain: initgminfo done

2节点在ocssd.bin 没有生成任何日志的情况下直接重启了

回复只看该作者道具举报

btxgua

4^#

发表于 2014-8-8 15:20:15

什么原因呢？

回复只看该作者道具举报

Maclean Liu(刘相兵

5^#

发表于 2014-8-8 15:23:10

虽然存在

Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
和
[ctssd(4063866)]CRS-2409:The clock on host ocsndb2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

但不是重启主要原因

2014-08-04 07:49:36.966: [ CSSD][2072]clssnm_skgxncheck: CSS daemon failed on node 2

说明但是node 2是可以ping通的，但是css daemon不可用

回复只看该作者道具举报