oracle 服务器掉电 重启后观察日志情况

ALLSTARS_ORACLE

大家好

今天机房突然断电了，启动之后oralce警告日志有好多包报错
oracle版本是 11.2.3  安装在IBM p720 操作系统是 AIX 6100-06-08-1216
以下是报错信息，请大家帮忙分析下


SQL>  select group#,sequence#,status from v$log;
    GROUP#  SEQUENCE# STATUS
---------- ---------- ----------------
         1       2197 CURRENT
         2       2194 INACTIVE
         3        189 INACTIVE
         4        190 INACTIVE
         5       2195 INACTIVE
         6       2196 INACTIVE
         7        191 INACTIVE
         8        192 INACTIVE
8 rows selected.
$ cat alert_CWEOL1.log |grep ORA
Starting ORACLE instance (normal)
ORACLE_HOME = /oracle/product/ora11gr2/db_1
ORACLE_BASE from environment = /oracle/product/ora11gr2
Starting ORACLE instance (normal)
ORACLE_HOME = /oracle/product/ora11gr2/db_1
ORACLE_BASE from environment = /oracle/product/ora11gr2
ORA-1109 signalled during: ALTER DATABASE CLOSE NORMAL...
Starting ORACLE instance (normal)
ORACLE_HOME = /oracle/product/ora11gr2/db_1
ORACLE_BASE from environment = /oracle/product/ora11gr2
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00313: open failed for members of log group 5 of thread 1
ORA-00313: open failed for members of log group 5 of thread 1
ORA-00313: open failed for members of log group 5 of thread 1
ORA-00313: open failed for members of log group 6 of thread 1
ORA-00313: open failed for members of log group 6 of thread 1
ORA-00313: open failed for members of log group 6 of thread 1
Starting ORACLE instance (normal)
ORACLE_HOME = /oracle/product/ora11gr2/db_1
ORACLE_BASE from environment = /oracle/product/ora11gr2
Starting ORACLE instance (normal)
ORACLE_HOME = /oracle/product/ora11gr2/db_1
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1
Starting ORACLE instance (normal)
ORACLE_HOME = /oracle/product/ora11gr2/db_1
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1
ORA-01555 caused by SQL statement below (SQL ID: 7dg78f8rg7xgk, Query Duration=0 sec, SCN: 0x0000.0325d6f7):
WARNING: inbound connection timed out (ORA-3136)
Starting ORACLE instance (normal)
ORACLE_HOME = /oracle/product/ora11gr2/db_1
ORACLE_BASE from environment = /oracle/product/ora11gr2
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1
WARNING: inbound connection timed out (ORA-3136)
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1
ORA-27072: File I/O error
ORA-00202: control file: '+DATA/cweol/controlfile/current.260.797570831'
ORA-15081: failed to submit an I/O operation to a disk
ORA-27063: number of bytes read/written is incorrect
ORA-27063: number of bytes read/written is incorrect
ORA-15080: synchronous I/O operation to a disk failed
ORA-15080: synchronous I/O operation to a disk failed
ORA-00345: redo log write error block 93936 count 63
ORA-00312: online log 6 thread 1: '+DATA/cweol/onlinelog/group_6.264.797570853'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00345: redo log write error block 93936 count 63
ORA-00312: online log 6 thread 1: '+RECO/cweol/onlinelog/group_6.260.797570855'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00340: IO error processing online log 6 of thread 1
ORA-00345: redo log write error block 93936 count 63
ORA-00312: online log 6 thread 1: '+DATA/cweol/onlinelog/group_6.264.797570853'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00345: redo log write error block 93936 count 63
ORA-00312: online log 6 thread 1: '+RECO/cweol/onlinelog/group_6.260.797570855'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-1092 : opitsk aborting process
ORA-1092 : opitsk aborting process
Starting ORACLE instance (normal)
ORACLE_HOME = /oracle/product/ora11gr2/db_1
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /oracle/product/ora11gr2

$  cat alert_CWEOL1.log |grep fail
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00313: open failed for members of log group 1 of thread 1
Checker run found 1 new persistent data failures
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00313: open failed for members of log group 5 of thread 1
ORA-00313: open failed for members of log group 5 of thread 1
ORA-00313: open failed for members of log group 5 of thread 1
ORA-00313: open failed for members of log group 6 of thread 1
ORA-00313: open failed for members of log group 6 of thread 1
ORA-00313: open failed for members of log group 6 of thread 1
WARNING: failed to read mirror side 1 of virtual extent 4 logical extent 0 of file 260 in group [1.2634245900] from disk DATA_0002  allocation unit 227 reason error; if possible, will try another mirror side
ORA-15081: failed to submit an I/O operation to a disk
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 45 logical extent 0 of file 264 in group 1 on disk 1 allocation unit 358
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 45 logical extent 0 of file 260 in group 3 on disk 1 allocation unit 467
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Instance termination failed to kill one or more processes

ALLSTARS_ORACLE

本帖最后由 xindahai 于 2012-12-7 18:01 编辑


补充 ：  这是RAC环境，另外一个节点数据库没有启动。  连接的存储是IBM DS5020.
下面是系统的情况



SQL> select * from v$logfile;
    GROUP# STATUS  TYPE    MEMBER                         IS_
---------- ------- ------- ------------------------------ ---
         6         ONLINE  +DATA/cweol/onlinelog/group_6. NO
                           264.797570853
         6         ONLINE  +RECO/cweol/onlinelog/group_6. YES
                           260.797570855
         5         ONLINE  +DATA/cweol/onlinelog/group_5. NO
                           263.797570845
         5         ONLINE  +RECO/cweol/onlinelog/group_5. YES
                           259.797570849
    GROUP# STATUS  TYPE    MEMBER                         IS_
---------- ------- ------- ------------------------------ ---
         2         ONLINE  +DATA/cweol/onlinelog/group_2. NO
                           262.797570841
         2         ONLINE  +RECO/cweol/onlinelog/group_2. YES
                           258.797570843
         1         ONLINE  +DATA/cweol/onlinelog/group_1. NO
                           261.797570835
         1         ONLINE  +RECO/cweol/onlinelog/group_1. YES
    GROUP# STATUS  TYPE    MEMBER                         IS_
---------- ------- ------- ------------------------------ ---
                           257.797570837
         3         ONLINE  +DATA/cweol/onlinelog/group_3. NO
                           267.797571139
         3         ONLINE  +RECO/cweol/onlinelog/group_3. YES
                           261.797571143
         4         ONLINE  +DATA/cweol/onlinelog/group_4. NO
                           268.797571147

    GROUP# STATUS  TYPE    MEMBER                         IS_
---------- ------- ------- ------------------------------ ---
         4         ONLINE  +RECO/cweol/onlinelog/group_4. YES
                           262.797571149
         7         ONLINE  +DATA/cweol/onlinelog/group_7. NO
                           269.797571151
         7         ONLINE  +RECO/cweol/onlinelog/group_7. YES
                           263.797571153
         8         ONLINE  +DATA/cweol/onlinelog/group_8. NO
                           270.797571157
    GROUP# STATUS  TYPE    MEMBER                         IS_
---------- ------- ------- ------------------------------ ---
         8         ONLINE  +RECO/cweol/onlinelog/group_8. YES
                           264.797571161

16 rows selected.




#  fcstat fcs0   

FIBRE CHANNEL STATISTICS REPORT: fcs0

Device Type: 8Gb PCI Express Dual Port FC Adapter (df1000f114108a03) (adapter/pciex/df1000f114108a0)
Serial Number: 1B80313ED5
Option ROM Version: 02781135
ZA: U2D1.10X5
World Wide Node Name: 0x20000000C9A25E70
World Wide Port Name: 0x10000000C9A25E70

FC-4 TYPES:
  Supported: 0x0000012000000000000000000000000000000000000000000000000000000000
  Active:    0x0000010000000000000000000000000000000000000000000000000000000000
Class of Service: 3
Port Speed (supported): 8 GBIT
Port Speed (running):   8 GBIT
Port FC ID: 0x010300
Port Type: Fabric

Seconds Since Last Reset: 31289           

        Transmit Statistics     Receive Statistics
        -------------------     ------------------
Frames: 13281208                16309158        
Words:  4571927296              5622437376      

LIP Count: 0               
NOS Count: 0               
Error Frames:  1               
Dumped Frames: 0               
Link Failure Count: 0               
Loss of Sync Count: 5               
Loss of Signal: 0               
Primitive Seq Protocol Error Count: 0               
Invalid Tx Word Count: 8               
Invalid CRC Count: 1               

IP over FC Adapter Driver Information
  No DMA Resource Count: 0               
  No Adapter Elements Count: 62              

FC SCSI Adapter Driver Information
  No DMA Resource Count: 0               
  No Adapter Elements Count: 62              
  No Command Resource Count: 0               

IP over FC Traffic Statistics
  Input Requests:   0               
  Output Requests:  0               
  Control Requests: 0               
  Input Bytes:  0               
  Output Bytes: 0               

FC SCSI Traffic Statistics
  Input Requests:   2496484         
  Output Requests:  1519369         
  Control Requests: 2353            
  Input Bytes:  21986818303     
  Output Bytes: 17840344576