Oracle数据库数据恢复、性能优化

找回密码
注册
搜索
热搜: 活动 交友 discuz
Oracle数据库数据恢复、性能优化»论坛 › Oracle › Oracle数据库管理 › ORA-00481 导致10204 RAC 节点重启
发新帖
内蒙-飞

11

积分

0

好友

3

主题

[SQL调优] ORA-00481 导致10204 RAC 节点重启

1#
发表于 2012-12-28 11:31:04 | 查看: 5751| 回复: 2
刘大您好！
     数据库环境：10.2.0.4 rac +raw
         操作系统 ：    aix5300-08
         双击rac软件：  Symantec Veritas

alert错误日志：alert_ossdb2.log
Fri Dec 28 10:10:57 2012
Errors in file /oracle/app/admin/ossdb/bdump/ossdb2_lmon_3719636.trc:
ORA-00481: LMON 进程因错误而终止
Fri Dec 28 10:10:57 2012
LMON: terminating instance due to error 481
Fri Dec 28 10:10:58 2012
Shutting down instance (abort)
License high water mark = 481
Fri Dec 28 10:10:59 2012
System state dump is made for local instance
System State dumped to trace file /oracle/app/admin/ossdb/bdump/ossdb2_diag_2355372.trc
Fri Dec 28 10:11:05 2012
Termination issued to instance processes. Waiting for the processes to exit
Fri Dec 28 10:11:11 2012
Instance termination failed to kill one or more processes
Instance terminated by LMON, pid = 3719636
Fri Dec 28 10:11:17 2012
Termination issued to instance processes. Waiting for the processes to exit
Fri Dec 28 10:11:23 2012
Instance termination failed to kill one or more processes
Instance terminated by USER, pid = 2863596
Fri Dec 28 10:11:37 2012
Fri Dec 28 10:11:37 2012
Starting ORACLE instance (normal)
sskgpgetexecname failed to get name
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned off.

alert_ossdb1.log中大量以下信息：
Fri Dec 28 10:10:45 2012
LMS 0: 5404 GCS shadows traversed, 4001 replayed
Fri Dec 28 10:10:45 2012
LMS 1: 5214 GCS shadows traversed, 4001 replayed
Fri Dec 28 10:10:45 2012
LMS 2: 5241 GCS shadows traversed, 4001 replayed
LMS 2: 5251 GCS shadows traversed, 4001 replayed
LMS 2: 5258 GCS shadows traversed, 4001 replayed
LMS 2: 5257 GCS shadows traversed, 4001 replayed
Fri Dec 28 10:10:45 2012
LMS 3: 5240 GCS shadows traversed, 4001 replayed
Fri Dec 28 10:10:45 2012
LMS 0: 5270 GCS shadows traversed, 4001 replayed

trc文件在附件中。
谢谢刘大！

ossdb2_lmon_3719636 and ossdb2_diag_2355372.zip

1.45 MB, 下载次数: 1861

trc文件
2#
发表于 2012-12-28 14:44:59
10.2.0.4.0 + AIX 5.3 RAC

  2. *** 2012-12-28 09:54:06.623
  3. Begin DRM(23409)
  4. sent syncr inc 40 lvl 2121 to 0 (40,0/31/0)
  5. synca inc 40 lvl 2121 rcvd (40.0)
  6. sent syncr inc 40 lvl 2122 to 0 (40,0/34/0)
  7. synca inc 40 lvl 2122 rcvd (40.0)
  8. sent syncr inc 40 lvl 2123 to 0 (40,0/36/0)
  9. synca inc 40 lvl 2123 rcvd (40.0)
  10. sent syncr inc 40 lvl 2124 to 0 (40,0/38/0)
  11. synca inc 40 lvl 2124 rcvd (40.0)
  12. *** 2012-12-28 09:54:35.233
  13. sent syncr inc 40 lvl 2125 to 0 (40,0/31/0)
  14. synca inc 40 lvl 2125 rcvd (40.0)
  15. sent syncr inc 40 lvl 2126 to 0 (40,0/34/0)
  16. synca inc 40 lvl 2126 rcvd (40.0)
  17. sent syncr inc 40 lvl 2127 to 0 (40,0/36/0)
  18. synca inc 40 lvl 2127 rcvd (40.0)
  19. sent syncr inc 40 lvl 2128 to 0 (40,0/38/0)
  20. synca inc 40 lvl 2128 rcvd (40.0)
  21. sent syncr inc 40 lvl 2129 to 0 (40,0/31/0)
  22. synca inc 40 lvl 2129 rcvd (40.0)
  23. sent syncr inc 40 lvl 2130 to 0 (40,0/34/0)
  24. synca inc 40 lvl 2130 rcvd (40.0)
  25. sent syncr inc 40 lvl 2131 to 0 (40,0/36/0)
  26. synca inc 40 lvl 2131 rcvd (40.0)
  27. sent syncr inc 40 lvl 2132 to 0 (40,0/38/0)
  28. synca inc 40 lvl 2132 rcvd (40.0)

  30. *** 2012-12-28 10:10:40.106
  31. kjfcdrmrfg: SYNC TIMEOUT (48859, 47898, 960), step 31
  32. Submitting asynchronized dump request [28]


  35. *** 2012-12-28 10:10:51.192
  36. Dumping diagnostic information for ospid 3539304:
  37. OS pid = 3539304
  38. loadavg : 1.89 1.78 5.02
  39. swap info: free_mem = 6521.94M rsv = 64.00M
  40.            alloc = 121.16M avail = 16384.00M swap_free = 16262.84M
  41.        F S      UID     PID    PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY  TIME CMD
  42.   240103 A   oracle 3539304       1   3  39 -- 4e602510 107808          20:36:23      -  9:03 ora_lms5_ossdb2
  43. open: The file access permissions do not allow the specified action.
  44. Warning: executed in non-root mode
  45. procstack cannot verify that /unix matches the running kernel.
  46. Kernel symbols might not be validated.
  47. 3539304: ora_lms5_ossdb2
  48. 0x090000000012cf48  poll(??, ??, ??) + 0x98
  49. 0x0000000101162700  sskgxp_select(??, ??, ??, ??, ??) + 0x100
  50. 0x000000010115e034  skgxpiwait(??, ??, ??, ??, ??, ??, ??) + 0x614
  51. 0x000000010115efe8  skgxpwait(??, ??, ??, ??, ??, ??, ??) + 0x2c8
  52. 0x000000010123ebf8  ksxpwait(??, ??, ??) + 0x538
  53. 0x0000000100110b14  ksliwat(??, ??, ??, ??, ??, ??, ??, ??) + 0x654
  54. 0x000000010011bb24  kslwaitns_timed(??, ??, ??, ??, ??, ??, ??, ??) + 0x24
  55. 0x0000000100177d0c  kskthbwt(0x300000003, 0x3400000034, 0x0, 0x18, 0x0, 0x0, 0xa00000000000a, 0xdacbf7bfdacbf7bf) + 0x22c
  56. 0x000000010011bcd4  kslwait(??, ??, ??, ??, ??, ??) + 0xf4
  57. 0x0000000101238a80  ksxprcv(??, ??, ??, ??, ??, ??, ??, ??) + 0x280
  58. 0x0000000101832154  kjctr_rksxp(??, ??, ??, ??, ??, ??, ??, ??) + 0x214
  59. 0x0000000101839364  kjctrcv(??, ??, ??, ??, ??, ??, ??) + 0x164
  60. 0x0000000101846eb4  kjcsrmg(??, ??, ??, ??, ??) + 0x54
  61. 0x000000010185277c  kjmsm(??) + 0x115c
  62. 0x0000000100166d94  ksbrdp() + 0x4b4
  63. 0x000000010430c8fc  opirip(??, ??, ??) + 0x3fc
  64. 0x0000000102d9adf8  opidrv(??, ??, ??) + 0x458
  65. 0x000000010370b910  sou2o(??, ??, ??, ??) + 0x90
  66. 0x0000000100000870  opimai_real(??, ??) + 0x150
  67. 0x00000001000006d8  main(??, ??) + 0x98
  68. 0x0000000100000368  __start() + 0x98
  69. *** 2012-12-28 10:10:53.105
  70. ----------------------------------------
  71. SO: 7000004e747a8c0, type: 2, owner: 0, flag: INIT/-/-/0x00
  72.   (process) Oracle pid=26, calls cur/top: 7000004a94a32f0/7000004e770e118, flag: (6) SYSTEM
  73.             int error: 0, call error: 0, sess error: 0, txn error 0
  74.   (post info) last post received: 0 0 90
  75.               last post received-location: KJCS Post snd proxy to flush msg
  76.               last process to post me: 7000004e6442610 1 6
  77.               last post sent: 0 0 131
  78.               last post sent-location: kclcget
  79.               last process posted by me: 7000004e645df10 3 0
  80.   (latch info) wait_event=0 bits=0
  81.     Process Group: DEFAULT, pseudo proc: 7000004e64e73c0
  82.     O/S info: user: oracle, term: UNKNOWN, ospid: 3539304
  83.     OSD pid info: Unix process pid: 3539304, image: oracle@ossdb2 (LMS5)
  84. Dump of memory from 0x07000004E9413778 to 0x07000004E9413980
  85. 7000004E9413770                   00000007 00000000          [........]
  86. 7000004E9413780 07000004 E892F920 00000010 000313A7  [....... ........]
  87. 7000004E9413790 07000004 E770E118 00000003 000313A7  [.....p..........]
  88. 7000004E94137A0 07000004 E77B94A8 00000013 0003129B  [.....{..........]
  89. 7000004E94137B0 07000004 E8882D60 0000000B 000313A7  [......-`........]
  90. 7000004E94137C0 07000004 EA6E20E0 00000004 0003129B  [.....n .........]
  91. 7000004E94137D0 07000004 E6B6B748 00000014 000313A7  [.......H........]
  92. 7000004E94137E0 07000004 EAE2AC70 00000015 000313A7  [.......p........]
  93. 7000004E94137F0 00000000 00000000 00000000 00000000  [................]
  94.   Repeat 24 times
  95. ----------------------------------------
  96. SO: 7000004ea6e20e0, type: 4, owner: 7000004e747a8c0, flag: INIT/-/-/0x00
  97.   (session) sid: 2195 trans: 0, creator: 7000004e747a8c0, flag: (51) USR/- BSY/-/-/-/-/-
  98.             DID: 0000-0000-00000000, short-term DID: 0000-0000-00000000
  99.             txn branch: 0
  100.             oct: 0, prv: 0, sql: 0, psql: 0, user: 0/SYS
  101.   waiting for 'gcs remote message' blocking sess=0x0 seq=15267 wait_time=0 seconds since wait started=3
  102.           waittime=18, poll=0, event=0
  103.   Dumping Session Wait History
  104.    for 'gcs remote message' count=1 wait_time=29327
  105.           waittime=18, poll=0, event=0
  106.    for 'gcs remote message' count=1 wait_time=960
  107.           waittime=18, poll=0, event=0
  108.    for 'gcs remote message' count=1 wait_time=29507
  109.           waittime=18, poll=0, event=0
  110.    for 'gcs remote message' count=1 wait_time=29324
  111.           waittime=18, poll=0, event=0
  112.    for 'gcs remote message' count=1 wait_time=13
  113.           waittime=18, poll=0, event=0
  114.    for 'gcs remote message' count=1 wait_time=1670769
  115.           waittime=18, poll=0, event=0
  116.    for 'gcs remote message' count=1 wait_time=914
  117.           waittime=18, poll=0, event=0
  118.    for 'gcs remote message' count=1 wait_time=746
  119.           waittime=18, poll=0, event=0
  120.    for 'gcs remote message' count=1 wait_time=29585
  121.           waittime=18, poll=0, event=0
  122.    for 'gcs remote message' count=1 wait_time=29321
  123.           waittime=18, poll=0, event=0
  124.   temporary object counter: 0
  125. ----------------------------------------
  126. UOL used : 0 locks(used=0, free=0)
  127. KGX Atomic Operation Log 7000004edff40c8
  128. Mutex 0(0, 0) idn 0 oper NONE
  129. Library Cache uid 2195 efd 0 whr 0 slp 0
  130. KGX Atomic Operation Log 7000004edff4110
  131. Mutex 0(0, 0) idn 0 oper NONE
  132. Library Cache uid 2195 efd 0 whr 0 slp 0
  133. KGX Atomic Operation Log 7000004edff4158
  134. Mutex 0(0, 0) idn 0 oper NONE
  135. Library Cache uid 2195 efd 0 whr 0 slp 0
  136. Session 2195: no row
  137. ----------------------------------------
  138. SO: 7000004e77b94a8, type: 19, owner: 7000004e747a8c0, flag: INIT/-/-/0x00
  139.    GES MSG BUFFERS: st=emp chunk=0x0 hdr=0x0 lnk=0x0 flags=0x0 inc=40
  140.     outq=0 sndq=12 opid=26 prmb=0x0
  141.     mbg[i]=(117682 4410693) mbg[b]=(362 1023792) mbg[r]=(0 0)
  142.     fmq[i]=(30 361) fmq[b]=(20 0) fmq[r]=(0 0)
  143.     mop[s]=1622386 mop[q]=4706211 pendq=0 zmbq=0
  144.     nonksxp_recvs=0
  145.   ------------process 0x7000004e77b94a8--------------------
  146.   proc version      : 0
  147.   Local node        : 1
  148.   pid               : 3539304
  149.   lkp_node          : 1
  150.   svr_mode          : 0
  151.   proc state        : KJP_NORMAL
  152.   Last drm hb acked : 12287
  153.   Total accesses    : 46
  154.   Imm.  accesses    : 45
  155.   Locks on ASTQ     : 0
  156.   Locks Pending AST : 0
  157.   Granted locks     : 0
  158.   AST_Q:
  159.   PENDING_Q:
  160.   GRANTED_Q:
  161.   KJM HIST LMS5:
  162.     7:14 6:0 10:31:1 17:2 16:20 15:2 12:29333 7:10 6:1 10:31:0
  163.     17:3 16:1 12:966 7:18 6:1 10:31:1 17:2 16:66 15:1 12:29514
  164.     7:15 6:0 10:31:1 17:2 16:25 15:1 12:29329 7:152 6:1 10:31:0
  165.     17:2 16:477 12:17 7:419 6:0 10:31:1 17:17 16:1319 15:2 13:65521:99
  166.     20:35:6 20:35:5 20:35:6 20:35:6 20:35:5 20:35:6 20:35:5 20:35:7 20:35:6 20:35:6
  167.     20:35:7 20:35:6 20:35:10 20:35:17 1:0 14:44:6 1:1 14:44:10 1:0 14:44:12
  168.     1:0 14:44:18 1:0 14:73:1
  169.    DEFER MSG QUEUE ON LMS5 IS EMPTY
  170.    SEQUENCES:
  171.     0:3821860.0  1:0.0
  172. DUMP state for lms6 (ospid 3596636)
  173. DUMP IPC context for lms6 (ospid 3596636)
  174. Dumping process 28.3596636 info:
  175. *** 2012-12-28 10:10:53.110
  176. Dumping diagnostic information for ospid 3596636:
  177. OS pid = 3596636
  178. loadavg : 1.89 1.78 5.02
  179. swap info: free_mem = 6674.93M rsv = 64.00M
  180.            alloc = 120.95M avail = 16384.00M swap_free = 16263.05M
  181.        F S      UID     PID    PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY  TIME CMD
  182.   240103 A   oracle 3596636       1   5  39 -- 49cf2d510 107808          20:36:23      -  9:14 ora_lms6_ossdb2
  183. open: The file access permissions do not allow the specified action.
  184. Warning: executed in non-root mode
  185. procstack cannot verify that /unix matches the running kernel.
  186. Kernel symbols might not be validated.
  187. 3596636: ora_lms6_ossdb2
  188. 0x090000000012cf48  poll(??, ??, ??) + 0x98
  189. 0x0000000101162700  sskgxp_select(??, ??, ??, ??, ??) + 0x100
  190. 0x000000010115e034  skgxpiwait(??, ??, ??, ??, ??, ??, ??) + 0x614
  191. 0x000000010115efe8  skgxpwait(??, ??, ??, ??, ??, ??, ??) + 0x2c8
  192. 0x000000010123ebf8  ksxpwait(??, ??, ??) + 0x538
  193. 0x0000000100110b14  ksliwat(??, ??, ??, ??, ??, ??, ??, ??) + 0x654
  194. 0x000000010011bb24  kslwaitns_timed(??, ??, ??, ??, ??, ??, ??, ??) + 0x24
  195. 0x0000000100177d0c  kskthbwt(0x300000003, 0x3400000034, 0x0, 0x18, 0x0, 0x0, 0xa00000000000a, 0xdae8746ddae8746d) + 0x22c
  196. 0x000000010011bcd4  kslwait(??, ??, ??, ??, ??, ??) + 0xf4
  197. 0x0000000101238a80  ksxprcv(??, ??, ??, ??, ??, ??, ??, ??) + 0x280
  198. 0x0000000101832154  kjctr_rksxp(??, ??, ??, ??, ??, ??, ??, ??) + 0x214
  199. 0x0000000101839364  kjctrcv(??, ??, ??, ??, ??, ??, ??) + 0x164
  200. 0x0000000101846eb4  kjcsrmg(??, ??, ??, ??, ??) + 0x54
  201. 0x000000010185277c  kjmsm(??) + 0x115c
  202. 0x0000000100166d94  ksbrdp() + 0x4b4
  203. 0x000000010430c8fc  opirip(??, ??, ??) + 0x3fc
  204. 0x0000000102d9adf8  opidrv(??, ??, ??) + 0x458
  205. 0x000000010370b910  sou2o(??, ??, ??, ??) + 0x90
  206. 0x0000000100000870  opimai_real(??, ??) + 0x150
  207. 0x00000001000006d8  main(??, ??) + 0x98
  208. 0x0000000100000368  __start() + 0x98
  209. *** 2012-12-28 10:10:54.999
  210. ----------------------------------------
  211. SO: 7000004f143d0f0, type: 2, owner: 0, flag: INIT/-/-/0x00
  212.   (process) Oracle pid=28, calls cur/top: 7000004a5104328/7000004e770e698, flag: (6) SYSTEM
  213.             int error: 0, call error: 0, sess error: 0, txn error 0
  214.   (post info) last post received: 0 0 90
  215.               last post received-location: KJCS Post snd proxy to flush msg
  216.               last process to post me: 7000004e6441e30 1 6
  217.               last post sent: 0 0 131
  218.               last post sent-location: kclcget
  219.               last process posted by me: 7000004e645df10 3 0
  220.   (latch info) wait_event=0 bits=0
  221.     Process Group: DEFAULT, pseudo proc: 7000004e64e73c0
  222.     O/S info: user: oracle, term: UNKNOWN, ospid: 3596636
  223.     OSD pid info: Unix process pid: 3596636, image: oracle@ossdb2 (LMS6)
  224. Dump of memory from 0x07000004E744ECF8 to 0x07000004E744EF00
  225. 7000004E744ECF0                   00000007 00000000          [........]
  226. 7000004E744ED00 07000004 E892F788 00000010 000313A7  [................]
  227. 7000004E744ED10 07000004 E770E698 00000003 000313A7  [.....p..........]
  228. 7000004E744ED20 07000004 E77B9E98 00000013 0003129B  [.....{..........]
  229. 7000004E744ED30 07000004 E8882C50 0000000B 000313A7  [......,P........]
  230. 7000004E744ED40 07000004 F16C78A0 00000004 0003129B  [.....lx.........]
  231. 7000004E744ED50 07000004 E6B6B7D8 00000014 000313A7  [................]
  232. 7000004E744ED60 07000004 EAE2ACB8 00000015 000313A7  [................]
  233. 7000004E744ED70 00000000 00000000 00000000 00000000  [................]
  234.   Repeat 24 times
  235. ----------------------------------------
  236. SO: 7000004f16c78a0, type: 4, owner: 7000004f143d0f0, flag: INIT/-/-/0x00
  237.   (session) sid: 2194 trans: 0, creator: 7000004f143d0f0, flag: (51) USR/- BSY/-/-/-/-/-
  238.             DID: 0000-0000-00000000, short-term DID: 0000-0000-00000000
  239.             txn branch: 0
  240.             oct: 0, prv: 0, sql: 0, psql: 0, user: 0/SYS
  241.   waiting for 'gcs remote message' blocking sess=0x0 seq=32385 wait_time=0 seconds since wait started=4
  242.           waittime=18, poll=0, event=0
  243.   Dumping Session Wait History
  244.    for 'gcs remote message' count=1 wait_time=29514
  245.           waittime=18, poll=0, event=0
  246.    for 'gcs remote message' count=1 wait_time=17495
  247.           waittime=18, poll=0, event=0
  248.    for 'gcs remote message' count=1 wait_time=15943
  249.           waittime=18, poll=0, event=0
  250.    for 'gcs remote message' count=1 wait_time=13462
  251.           waittime=18, poll=0, event=0
  252.    for 'gcs remote message' count=1 wait_time=6
  253.           waittime=18, poll=0, event=0
  254.    for 'gcs remote message' count=1 wait_time=23
  255.           waittime=18, poll=0, event=0
  256.    for 'gcs remote message' count=1 wait_time=1655554
  257.           waittime=18, poll=0, event=0
  258.    for 'gcs remote message' count=1 wait_time=29519
  259.           waittime=18, poll=0, event=0
  260.    for 'gcs remote message' count=1 wait_time=29596
  261.           waittime=18, poll=0, event=0
  262.    for 'gcs remote message' count=1 wait_time=29531
  263.           waittime=18, poll=0, event=0
  264.   temporary object counter: 0
  265. ----------------------------------------
  266. UOL used : 0 locks(used=0, free=0)
  267. KGX Atomic Operation Log 7000004ecff57d8
  268. Mutex 0(0, 0) idn 0 oper NONE
  269. Library Cache uid 2194 efd 0 whr 0 slp 0
  270. KGX Atomic Operation Log 7000004ecff5820
  271. Mutex 0(0, 0) idn 0 oper NONE
  272. Library Cache uid 2194 efd 0 whr 0 slp 0
  273. KGX Atomic Operation Log 7000004ecff5868
  274. Mutex 0(0, 0) idn 0 oper NONE
  275. Library Cache uid 2194 efd 0 whr 0 slp 0
  276. Session 2194: no row
  277. ----------------------------------------
  278. SO: 7000004e77b9e98, type: 19, owner: 7000004f143d0f0, flag: INIT/-/-/0x00
  279.    GES MSG BUFFERS: st=emp chunk=0x0 hdr=0x0 lnk=0x0 flags=0x0 inc=40
  280.     outq=0 sndq=12 opid=28 prmb=0x0
  281.     mbg[i]=(116213 4404987) mbg[b]=(299 1036763) mbg[r]=(0 0)
  282.     fmq[i]=(30 11) fmq[b]=(20 0) fmq[r]=(0 0)
  283.     mop[s]=1631481 mop[q]=4709126 pendq=0 zmbq=0
  284.     nonksxp_recvs=0
  285.   ------------process 0x7000004e77b9e98--------------------
  286.   proc version      : 0
  287.   Local node        : 1
  288.   pid               : 3596636
  289.   lkp_node          : 1
  290.   svr_mode          : 0
  291.   proc state        : KJP_NORMAL
  292.   Last drm hb acked : 12287
  293.   Total accesses    : 52
  294.   Imm.  accesses    : 51
  295.   Locks on ASTQ     : 0
  296.   Locks Pending AST : 0
  297.   Granted locks     : 0
  298.   AST_Q:
  299.   PENDING_Q:
  300.   GRANTED_Q:
  301.   KJM HIST LMS6:
  302.     7:1 6:0 10:31:1 17:2 16:1 15:1 12:29519 7:13 6:0 10:31:1
  303.     17:3 16:1 15:2 12:17502 7:7 6:0 10:31:0 17:2 16:1 15:1
  304.     12:15947 7:11 6:1 10:31:0 17:3 16:10 15:1 12:13466 7:1 6:0
  305.     10:31:1 2:23 19:0 18:4 17:2 16:0 15:1 12:8 7:128 6:0
  306.     10:31:1 17:2 16:532 12:27 7:461 6:1 10:31:1 17:19 16:1200 15:1
  307.     14:44:9 1:0 14:44:11 1:0 14:44:7 1:0 13:65521:76 20:35:4 20:35:4 20:35:5
  308.     20:35:4 20:35:4 20:35:4 20:35:4
  309.    DEFER MSG QUEUE ON LMS6 IS EMPTY
  310.    SEQUENCES:
  311.     0:3903347.0  1:0.0
  312. DUMP state for lms7 (ospid 3572208)
  313. DUMP IPC context for lms7 (ospid 3572208)
  314. Dumping process 31.3572208 info:
  315. *** 2012-12-28 10:10:55.004
  316. Dumping diagnostic information for ospid 3572208:
  317. OS pid = 3572208
  318. loadavg : 1.89 1.78 5.02
  319. swap info: free_mem = 6685.82M rsv = 64.00M
  320.            alloc = 120.89M avail = 16384.00M swap_free = 16263.11M
  321.        F S      UID     PID    PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY  TIME CMD
  322.   240103 A   oracle 3572208       1   3  39 -- 626f98510 107752          20:36:23      -  8:21 ora_lms7_ossdb2
  323. open: The file access permissions do not allow the specified action.
  324. Warning: executed in non-root mode
  325. procstack cannot verify that /unix matches the running kernel.
  326. Kernel symbols might not be validated.
  327. 3572208: ora_lms7_ossdb2
  328. 0x090000000012cf48  poll(??, ??, ??) + 0x98
  329. 0x0000000101162700  sskgxp_select(??, ??, ??, ??, ??) + 0x100
  330. 0x000000010115e034  skgxpiwait(??, ??, ??, ??, ??, ??, ??) + 0x614
  331. 0x000000010115efe8  skgxpwait(??, ??, ??, ??, ??, ??, ??) + 0x2c8
  332. 0x000000010123ebf8  ksxpwait(??, ??, ??) + 0x538
  333. 0x0000000100110b14  ksliwat(??, ??, ??, ??, ??, ??, ??, ??) + 0x654
  334. 0x000000010011bb24  kslwaitns_timed(??, ??, ??, ??, ??, ??, ??, ??) + 0x24
  335. 0x0000000100177d0c  kskthbwt(0x300000003, 0x3400000034, 0x0, 0x18, 0x0, 0x0, 0xa00000000000a, 0xdb047e8bdb047e8b) + 0x22c
  336. 0x000000010011bcd4  kslwait(??, ??, ??, ??, ??, ??) + 0xf4
  337. 0x0000000101238a80  ksxprcv(??, ??, ??, ??, ??, ??, ??, ??) + 0x280
  338. 0x0000000101832154  kjctr_rksxp(??, ??, ??, ??, ??, ??, ??, ??) + 0x214
  339. 0x0000000101839364  kjctrcv(??, ??, ??, ??, ??, ??, ??) + 0x164
  340. 0x0000000101846eb4  kjcsrmg(??, ??, ??, ??, ??) + 0x54
  341. 0x000000010185277c  kjmsm(??) + 0x115c
  342. 0x0000000100166d94  ksbrdp() + 0x4b4
  343. 0x000000010430c8fc  opirip(??, ??, ??) + 0x3fc
  344. 0x0000000102d9adf8  opidrv(??, ??, ??) + 0x458
  345. 0x000000010370b910  sou2o(??, ??, ??, ??) + 0x90
  346. 0x0000000100000870  opimai_real(??, ??) + 0x150
  347. 0x00000001000006d8  main(??, ??) + 0x98
  348. 0x0000000100000368  __start() + 0x98
  349. *** 2012-12-28 10:10:56.921
  350. ----------------------------------------
  351. SO: 7000004e846a608, type: 2, owner: 0, flag: INIT/-/-/0x00
  352.   (process) Oracle pid=31, calls cur/top: 7000004a51045e8/7000004e770eef0, flag: (6) SYSTEM
  353.             int error: 0, call error: 0, sess error: 0, txn error 0
  354.   (post info) last post received: 0 0 90
  355.               last post received-location: KJCS Post snd proxy to flush msg
  356.               last process to post me: 7000004e6442610 1 6
  357.               last post sent: 0 0 24
  358.               last post sent-location: ksasnd
  359.               last process posted by me: 7000004ea458388 1 6
  360.   (latch info) wait_event=0 bits=0
  361.     Process Group: DEFAULT, pseudo proc: 7000004e64e73c0
  362.     O/S info: user: oracle, term: UNKNOWN, ospid: 3572208
  363.     OSD pid info: Unix process pid: 3572208, image: oracle@ossdb2 (LMS7)
  364. Dump of memory from 0x07000004EA42BFE0 to 0x07000004EA42C1E8
  365. 7000004EA42BFE0 00000007 00000000 07000004 E892F810  [................]
  366. 7000004EA42BFF0 00000010 000313A7 07000004 E770EEF0  [.............p..]
  367. 7000004EA42C000 00000003 000313A7 07000004 E77BAD80  [.............{..]
  368. 7000004EA42C010 00000013 0003129B 07000004 E8882E70  [...............p]
  369. 7000004EA42C020 0000000B 000313A7 07000004 E66D7C88  [.............m|.]
  370. 7000004EA42C030 00000004 0003129B 07000004 E6B6B868  [...............h]
  371. 7000004EA42C040 00000014 000313A7 07000004 EAE2AD00  [................]
  372. 7000004EA42C050 00000015 000313A7 00000000 00000000  [................]
  373. 7000004EA42C060 00000000 00000000 00000000 00000000  [................]
  374.         Repeat 23 times
  375. 7000004EA42C1E0 00000000 00000000                    [........]        
  376. ----------------------------------------
  377. SO: 7000004e66d7c88, type: 4, owner: 7000004e846a608, flag: INIT/-/-/0x00
  378.   (session) sid: 2193 trans: 0, creator: 7000004e846a608, flag: (51) USR/- BSY/-/-/-/-/-
  379.             DID: 0000-0000-00000000, short-term DID: 0000-0000-00000000
  380.             txn branch: 0
  381.             oct: 0, prv: 0, sql: 0, psql: 0, user: 0/SYS
  382.   waiting for 'gcs remote message' blocking sess=0x0 seq=56500 wait_time=0 seconds since wait started=0
  383.           waittime=18, poll=0, event=0
  384.   Dumping Session Wait History
  385.    for 'gcs remote message' count=1 wait_time=3981
  386.           waittime=18, poll=0, event=0
  387.    for 'gcs remote message' count=1 wait_time=29332
  388.           waittime=18, poll=0, event=0
  389.    for 'gcs remote message' count=1 wait_time=29507
  390.           waittime=18, poll=0, event=0
  391.    for 'gcs remote message' count=1 wait_time=14532
  392.           waittime=18, poll=0, event=0
  393.    for 'gcs remote message' count=1 wait_time=6477
  394.           waittime=18, poll=0, event=0
  395.    for 'gcs remote message' count=1 wait_time=8495
  396.           waittime=18, poll=0, event=0
  397.    for 'gcs remote message' count=1 wait_time=5241
  398.           waittime=18, poll=0, event=0
  399.    for 'gcs remote message' count=1 wait_time=13
  400.           waittime=0, poll=0, event=0
  401.    for 'gcs remote message' count=1 wait_time=1687575
  402.           waittime=18, poll=0, event=0
  403.    for 'gcs remote message' count=1 wait_time=216
  404.           waittime=18, poll=0, event=0
  405.   temporary object counter: 0
  406. ----------------------------------------
  407. UOL used : 0 locks(used=0, free=0)
  408. KGX Atomic Operation Log 7000004eff96920
  409. Mutex 0(0, 0) idn 0 oper NONE
  410. Library Cache uid 2193 efd 0 whr 0 slp 0
  411. KGX Atomic Operation Log 7000004eff96968
  412. Mutex 0(0, 0) idn 0 oper NONE
  413. Library Cache uid 2193 efd 0 whr 0 slp 0
  414. KGX Atomic Operation Log 7000004eff969b0
  415. Mutex 0(0, 0) idn 0 oper NONE
  416. Library Cache uid 2193 efd 0 whr 0 slp 0
  417. Session 2193: no row
  418. ----------------------------------------
  419. SO: 7000004e77bad80, type: 19, owner: 7000004e846a608, flag: INIT/-/-/0x00
  420.    GES MSG BUFFERS: st=emp chunk=0x0 hdr=0x0 lnk=0x0 flags=0x0 inc=40
  421.     outq=0 sndq=14 opid=31 prmb=0x0
  422.     mbg[i]=(123581 4373604) mbg[b]=(90 996046) mbg[r]=(0 0)
  423.     fmq[i]=(30 74) fmq[b]=(20 0) fmq[r]=(0 0)
  424.     mop[s]=1539856 mop[q]=4661764 pendq=0 zmbq=0
  425.     nonksxp_recvs=0
  426.   ------------process 0x7000004e77bad80--------------------
  427.   proc version      : 0
  428.   Local node        : 1
  429.   pid               : 3572208
  430.   lkp_node          : 1
  431.   svr_mode          : 0
  432.   proc state        : KJP_NORMAL
  433.   Last drm hb acked : 12287
  434.   Total accesses    : 934
  435.   Imm.  accesses    : 933
  436.   Locks on ASTQ     : 0
  437.   Locks Pending AST : 0
  438.   Granted locks     : 0
  439.   AST_Q:
  440.   PENDING_Q:
  441.   GRANTED_Q:
  442.   KJM HIST LMS7:
  443.     7:19 6:1 10:31:1 17:3 16:74 12:3989 7:25 6:1 10:31:1 17:40
  444.     16:100 15:1 12:29340 7:15 6:0 10:31:1 17:3 16:35 15:1 12:29514
  445.     7:13 6:0 10:31:0 17:3 16:30 15:1 12:14537 7:17 6:0 10:31:1
  446.     17:4 16:26 15:2 12:6485 7:18 6:1 10:31:1 17:3 16:40 12:8504
  447.     7:29 6:1 10:31:1 17:3 16:1 15:1 12:5249 7:96 6:1 10:31:1
  448.     2:37 19:1 18:4 17:3 16:364 12:19 7:530 6:1 10:31:1 17:5
  449.     16:1497 15:2 14:44:9 1:0
  450.    DEFER MSG QUEUE ON LMS7 IS EMPTY
  451.    SEQUENCES:
  452.     0:3479782.0  1:0.0
  453. error 481 detected in background process
  454. ORA-00481: LMON 进程因错误而终止
  455. ksuitm: waiting up to [5] seconds before killing DIAG
  456. *** 2012-12-28 10:11:11.927
  457. Instance termination failed to kill one or more processes
  458. ksuitm_check: OS PID=2621624 is still alive
  459. *** 2012-12-28 10:11:11.927
  460. Dumping diagnostic information for oracle@ossdb2:
  461. OS pid = 2621624
  462. loadavg : 2.12 1.84 4.99
  463. swap info: free_mem = 10673.18M rsv = 64.00M
  464.            alloc = 83.45M avail = 16384.00M swap_free = 16300.55M
  465.        F S      UID     PID    PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY  TIME CMD
  466.   250004 Z   oracle 2621624       1   2  61 20                                      0:00 <defunct>

复制代码

回复 只看该作者 道具 举报

3#
发表于 2012-12-28 14:48:58
症状 10.2.0.4 + AIX +RAC  LMON 遇到ORA-481 导致实例crash

LMON触发ORA-481前在做DRM ，DRM未结束前遇到kjfcdrmrfg: SYNC TIMEOUT (48859, 47898, 960), step 31 超时，

LMON的等待事件为gcs remote message ， 且当时5分钟内的负载升高loadavg : 1.89 1.78 5.02


可能的Workaround:

关闭DRM特性 ， 让LMON不会因为DRM而crash instance 方法： 设置隐藏参数_gc_affinity_time=0   

@_gc_affinity_time=0                                 # Only if DB version is 10.1 or 10.2
@_gc_undo_affinity=FALSE                       # Only if Db version is 10.2

回复 只看该作者 道具 举报

返回列表
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 注册

QQ|手机版|Archiver|Oracle数据库数据恢复、性能优化

GMT+8, 2024-11-16 04:27 , Processed in 0.080072 second(s), 24 queries .

Powered by Discuz! X2.5

© 2001-2012 Comsenz Inc.

回顶部
TEL/電話+86 13764045638
Email service@parnassusdata.com
QQ 47079569