arch进程导致11R2 crash

myownstars

OS: Aix 6
DB: oracle 11.2.0.2
症状：

数据库crash，同时alertlog记录如下，一般来说ckpt/pmon/smon终结instance比较常见，但很少见到arch进程会导致down机

Tue May 29 16:11:06 2012
Process W001 died, see its trace file
Tue May 29 16:15:37 2012
ARC2 (ospid: 7405806): terminating the instance due to error 472
Tue May 29 16:15:38 2012
System state dump requested by (instance=1, osid=7405806 (ARC2)), summary=[abnormal instance termination].
System State dumped to trace file /u001/app/oracle/admin/biobank/diag/rdbms/biobank/biobank/trace/biobank_diag_9044050.trc
Dumping diagnostic data in directory=[cdmp_20120529161538], requested by (instance=1, osid=7405806 (ARC2)), summary=[abnormal instance termination].
Instance terminated by ARC2, pid = 7405806

请maclean点化一下吧 谢谢

teapot

*** 2012-05-29 16:15:37.883
ARC2 (ospid: 7405806): terminating the instance due to error 472
ksuitm: waiting up to [5] seconds before killing DIAG(9044050)

472一般是PMON进程异常死掉引起的
diag发生时，进程已经全部DEAD了
$ grep DEAD biobank_diag_9044050.trc
    O/S info: user: , term: , ospid:  (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 8585352 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 8912928 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 6815796 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 7208978 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 9568486 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 4653138 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 6422778 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 3866734 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 9371792 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 7733492 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 8782046 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 8454232 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 8192224 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 4128860 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 7602262 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 1704108 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 2097250 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 8323076 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 6750314 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 5243070 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 13697068 (DEAD)
    O/S info: user: oracle, term: UNKNOWN, ospid: 9699404 (DEAD)
    O/S info: user: , term: , ospid:  (DEAD)

看一下PMON那时的状态：
PROCESS 2: PMON
  ----------------------------------------
  SO: 0x70000027e5aa8d8, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
   proc=0x70000027e5aa8d8, name=process, file=ksu.h LINE:12451 ID:, pg=0
  (process) Oracle pid:2, ser:1, calls cur/top: 0x70000027e982b98/0x70000027e982b98
            flags : (0xe) SYSTEM
            flags2: (0x0),  flags3: (0x0)
            intr error: 0, call error: 0, sess error: 0, txn error 0
            intr queue: empty
    ksudlp FALSE at location: 0
  (post info) last post received: 0 0 45
              last post received-location: ksv2.h LINE:1655 ID:ksvpst: spawncleanup
              last process to post me: 70000027e5bd0b8 3 2
              last post sent: 0 0 26
              last post sent-location: ksa2.h LINE:282 ID:ksasnd
              last process posted by me: 70000027e5bafd8 237 0
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: 0x70000027e64fac8
    O/S info: user: oracle, term: UNKNOWN, ospid: 8585352 (DEAD)
    OSD pid info: Unix process pid: 8585352, image: oracle@sgdebiobdb1 (PMON)

70000027e5bd0b8 是SMCO进程
70000027e5bafd8 已经找不到了，并且与许多后台进程有关系，可能就是ARCH2（diag发生在arch2死掉之后）

SMCO (Space Management Coordinator) For Autoextend On Datafiles And How To Disable/Enable [ID 743773.1]
里讲SMCO与自动空间管理任务有关系，看一下DB的空间情况，可能是引起ARCH2被杀掉的原因。
从alert日志看，W001死掉了，前面不知道发生了什么事情，W001是SMCO的slave进程

[ 本帖最后由 teapot 于 2012-5-31 18:00 编辑 ]