Honcho

16 积分	0 好友	2 主题

发消息

RAC ons服务offline ons 进程CPU 100%

1^#

发表于 2011-10-24 16:38:14 | 查看: 9153| 回复: 7

今天一客户反映RAC 的ons 服务offline

操作系统Linux 下版本不记得，Oracle ：10.2.0.1_64

ons.log
2011-10-22 13:47:13.779: [ RACG][2540399872] [14877][2540399872][ora.abcca2210026.ons]: timeout: killed the spawned process
2011-10-22 13:47:13.783: [ RACG][2540399872] [14877][2540399872][ora.abcca2210026.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/home/app/oracle/product/10.2.0/crs_2
2011-10-22 13:47:13.783: [ RACG][2540399872] [14877][2540399872][ora.abcca2210026.ons]: clsrcexecut: cmd = /home/app/oracle/product/10.2.0/crs_2/bin/racgeut -e _USR_ORA_DEBUG=0 540 /home/app/oracle/product/10.2.0/crs_2/bin/onsctl ping
2011-10-22 13:47:13.783: [ RACG][2540399872] [14877][2540399872][ora.abcca2210026.ons]: clsrcexecut: rc = 99, time = 540.090s
2011-10-22 13:47:13.783: [ RACG][2540399872] [14877][2540399872][ora.abcca2210026.ons]: end for resource = ora.abcca2210026.ons, action = check, status = 1, time = 540.130s
2011-10-22 13:56:14.893: [ RACG][2540399872] [28346][2540399872][ora.abcca2210026.ons]: onsctl: shutting down ons daemon ...
timeout: killed the spawned process
2011-10-22 13:56:14.894: [ RACG][2540399872] [28346][2540399872][ora.abcca2210026.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/home/app/oracle/product/10.2.0/crs_2
2011-10-22 13:56:14.894: [ RACG][2540399872] [28346][2540399872][ora.abcca2210026.ons]: clsrcexecut: cmd = /home/app/oracle/product/10.2.0/crs_2/bin/racgeut -e _USR_ORA_DEBUG=0 540 /home/app/oracle/product/10.2.0/crs_2/bin/onsctl stop
2011-10-22 13:56:14.894: [ RACG][2540399872] [28346][2540399872][ora.abcca2210026.ons]: clsrcexecut: rc = 99, time = 541.000s
2011-10-22 14:05:15.117: [ RACG][2540399872] [9294][2540399872][ora.abcca2210026.ons]: timeout: killed the spawned process
2011-10-22 14:05:15.117: [ RACG][2540399872] [9294][2540399872][ora.abcca2210026.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/home/app/oracle/product/10.2.0/crs_2
2011-10-22 14:05:15.117: [ RACG][2540399872] [9294][2540399872][ora.abcca2210026.ons]: clsrcexecut: cmd = /home/app/oracle/product/10.2.0/crs_2/bin/racgeut -e _USR_ORA_DEBUG=0 540 /home/app/oracle/product/10.2.0/crs_2/bin/onsctl start
2011-10-22 14:05:15.117: [ RACG][2540399872] [9294][2540399872][ora.abcca2210026.ons]: clsrcexecut: rc = 99, time = 540.090s
2011-10-22 14:14:16.116: [ RACG][2540399872] [9294][2540399872][ora.abcca2210026.ons]: timeout: killed the spawned process
2011-10-22 14:14:16.116: [ RACG][2540399872] [9294][2540399872][ora.abcca2210026.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/home/app/oracle/product/10.2.0/crs_2

分享0

收藏0 回复只看该作者道具举报

Honcho

2^#

发表于 2011-10-24 16:38:50

crsd.log

2011-10-22 13:47:13.788: [  CRSAPP][1535293792]0CheckResource error for ora.abcca2210026.ons error code = 1
2011-10-22 13:47:13.791: [  CRSRES][1535293792]0In stateChanged, ora.abcca2210026.ons target is ONLINE
2011-10-22 13:47:13.791: [  CRSRES][1535293792]0ora.abcca2210026.ons on abcca2210026 went OFFLINE unexpectedly
2011-10-22 13:47:13.791: [  CRSRES][1535293792]0StopResource: setting CLI values
2011-10-22 13:47:13.795: [  CRSRES][1535293792]0Attempting to stop `ora.abcca2210026.ons` on member `abcca2210026`
2011-10-22 13:47:13.865: [  OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1734331696] retval lht [-27] Signal CV.
2011-10-22 13:48:43.443: [  OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1734322288] retval lht [-27] Signal CV.
2011-10-22 13:56:14.930: [  CRSRES][1535293792]0Stop of `ora.abcca2210026.ons` on member `abcca2210026` succeeded.
2011-10-22 13:56:14.930: [  CRSRES][1535293792]0ora.abcca2210026.ons RESTART_COUNT=0 RESTART_ATTEMPTS=3
2011-10-22 13:56:14.931: [  CRSRES][1535293792]0Restarting ora.abcca2210026.ons on abcca2210026
2011-10-22 13:56:14.933: [  CRSRES][1535293792]0startRunnable: setting CLI values
2011-10-22 13:56:14.934: [  CRSRES][1535293792]0Attempting to start `ora.abcca2210026.ons` on member `abcca2210026`
2011-10-22 13:58:10.558: [  OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1734321920] retval lht [-27] Signal CV.
2011-10-22 13:58:43.769: [  OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1734321920] retval lht [-27] Signal CV.
2011-10-22 13:58:43.782: [  OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1734282800] retval lht [-27] Signal CV.
2011-10-22 13:59:10.874: [  OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1734321920] retval lht [-27] Signal CV.
2011-10-22 14:02:11.791: [  OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1734321920] retval lht [-27] Signal CV.
2011-10-22 14:04:12.402: [  OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1734321920] retval lht [-27] Signal CV.
2011-10-22 14:06:44.971: [  CRSEVT][1535293792]0CAAMonitorHandler :: 0:Action Script /home/app/oracle/product/10.2.0/crs_2/bin/racgwrap(start) timed out for ora.abcca2210026.ons! (timeout=600)
2011-10-22 14:06:44.971: [  CRSAPP][1535293792]0StartResource error for ora.abcca2210026.ons error code = -2

回复只看该作者道具举报

Honcho

3^#

发表于 2011-10-24 16:39:19

top

Tasks: 368 total, 9 running, 359 sleeping, 0 stopped, 0 zombie
Cpu(s): 24.1% us, 26.4% sy,  0.0% ni, 49.5% id,  0.0% wa,  0.0% hi,  0.0% si
Mem: 8159996k total,  8056492k used, 103504k free, 94664k buffers
Swap:  8393920k total, 137464k used,  8256456k free,  5085388k cached
  PID USER    PR  NI  VIRT  RES  SHR S %CPU %MEM TIME+  COMMAND
15451 oracle 25 0 22152 4772 3496 R  100  0.1  27:52.89 ons
24892 oracle 25 0 22152 4772 3496 R  100  0.1 2793:11 ons
32330 oracle 25 0 22152 4772 3496 R  100  0.1  38:22.34 ons
9324 oracle 25 0 22152 4772 3496 R  100  0.1 2803:41 ons
13202 oracle 25 0 22152 4772 3496 R  100  0.1  29:21.80 ons
14882 oracle 25 0 22152 4772 3496 R  100  0.1 2821:47 ons
28353 oracle 25 0 22152 4772 3496 R  100  0.1 2812:48 ons
22705 oracle 25 0 22152 4772 3496 R  100  0.1 2794:57 ons
22214 oracle 16 0  6408 1248  772 R 1  0.0 0:00.67 top
7 root    34  19    0 0 0 S 0  0.0 0:07.40 ksoftirqd/2
  774 root    15 0    0 0 0 S 0  0.0  32:20.92 kjournald
1 root    16 0  4756  552  460 S 0  0.0 0:11.90 init
2 root    RT 0    0 0 0 S 0  0.0 6:32.32 migration/0
3 root    34  19    0 0 0 S 0  0.0 0:37.34 ksoftirqd/0
4 root    RT 0    0 0 0 S 0  0.0 2:55.90 migration/1
5 root    34  19    0 0 0 S 0  0.0 4:18.06 ksoftirqd/1
6 root    RT 0    0 0 0 S 0  0.0 2:28.91 migration/2
8 root    RT 0    0 0 0 S 0  0.0 7:18.59 migration/3
9 root    34  19    0 0 0 S 0  0.0 0:20.76 ksoftirqd/3
10 root    RT 0    0 0 0 S 0  0.0 0:58.82 migration/4
11 root    34  19    0 0 0 S 0  0.0  12:34.67 ksoftirqd/4
12 root    RT 0    0 0 0 S 0  0.0 3:42.98 migration/5
13 root    34  19    0 0 0 S 0  0.0 2:07.98 ksoftirqd/5
14 root    RT 0    0 0 0 S 0  0.0 0:21.66 migration/6
15 root    34  19    0 0 0 S 0  0.0 0:17.77 ksoftirqd/6
16 root    RT 0    0 0 0 S 0  0.0 1:14.68 migration/7
17 root    34  19    0 0 0 S 0  0.0  14:51.31 ksoftirqd/7
18 root    RT 0    0 0 0 S 0  0.0 0:05.60 migration/8
19 root    34  19    0 0 0 S 0  0.0 4:24.95 ksoftirqd/8
20 root    RT 0    0 0 0 S 0  0.0 0:18.80 migration/9
21 root    34  19    0 0 0 S 0  0.0 0:11.94 ksoftirqd/9
22 root    RT 0    0 0 0 S 0  0.0 0:02.63 migration/10
23 root    34  19    0 0 0 S 0  0.0 0:09.38 ksoftirqd/10
24 root    RT 0    0 0 0 S 0  0.0 0:02.96 migration/11

回复只看该作者道具举报