WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of...
os： rehl 5.9db： 11023 rac
alertlog:
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
Errors in file /usr/oracle/app/oracle/diag/rdbms/xxxxx/xxxxx1/trace/xxxxx1_ora_43933.trc:
ORA-15025: could not open disk "/dev/asm-data2"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group from disk DATADG_0001 allocation unit 920 reason error; if possible, will try another mirror side
Errors in file /usr/oracle/app/oracle/diag/rdbms/xxxxx/xxxxx1/trace/xxxxx1_ora_43933.trc:
ORA-00202: control file: '+DATADG/xxxxx/controlfile/current.260.853170001'
ORA-15081: failed to submit an I/O operation to a disk
2014-08-26 16:17:54.445000 +08:00
Errors in file /usr/oracle/app/oracle/diag/rdbms/xxxxx/xxxxx1/trace/xxxxx1_ora_44247.trc:
ORA-15025: could not open disk "/dev/asm-data2"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
Errors in file /usr/oracle/app/oracle/diag/rdbms/xxxxx/xxxxx1/trace/xxxxx1_ora_44247.trc:
ORA-15025: could not open disk "/dev/asm-data2"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group from disk DATADG_0001 allocation unit 920 reason error; if possible, will try another mirror side
Errors in file /usr/oracle/app/oracle/diag/rdbms/xxxxx/xxxxx1/trace/xxxxx1_ora_44247.trc:
ORA-00202: control file: '+DATADG/xxxxx/controlfile/current.260.853170001'
ORA-15081: failed to submit an I/O operation to a disk
2014-08-26 16:21:21.525000 +08:00
Thread 1 advanced to log sequence 250 (LGWR switch)
Current log# 5 seq# 250 mem# 0: +DATADG/xxxxx/onlinelog/group_5.268.853232459
Current log# 5 seq# 250 mem# 1: +FLASH/xxxxx/onlinelog/group_5.261.853232465
2014-08-26 16:21:29.160000 +08:00
Archived Log entry 339 added for thread 1 sequence 249 ID 0xf106ed8c dest 1:
# vi /usr/oracle/app/oracle/diag/rdbms/xxxxx/xxxxx1/trace/xxxxx1_ora_23336.trc
Trace file /usr/oracle/app/oracle/diag/rdbms/xxxxx/xxxxx1/trace/xxxxx1_ora_23336.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /usr/oracle/app/oracle/product/11.2.0/db_2
System name: Linux
Node name: gtaq11
Release: 2.6.32-431.el6.x86_64
Version: #1 SMP Sun Nov 10 22:19:54 EST 2013
Machine: x86_64
Instance name: xxxxx1
Redo thread mounted by this instance: 1
Oracle process number: 309
Unix process pid: 23336, image: oracle@xxxxxxx1 (TNS V1-V3)
*** 2014-08-27 12:17:00.455
*** SESSION ID:(3664.48931) 2014-08-27 12:17:00.455
*** CLIENT ID:() 2014-08-27 12:17:00.455
*** SERVICE NAME:(SYS$USERS) 2014-08-27 12:17:00.455
*** MODULE NAME:(beremote@xxxxxx1 (TNS V1-V3)) 2014-08-27 12:17:00.455
*** ACTION NAME:() 2014-08-27 12:17:00.455
WARNING: failed to open a disk
ORA-15025: could not open disk "/dev/asm-data2"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
kfk_debug_get_user_groups: uid:2, euid:501, gid:0, egid:504
WARNING: failed to open a disk
ORA-15025: could not open disk "/dev/asm-data2"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
kfk_debug_get_user_groups: uid:2, euid:501, gid:0, egid:504
WARNING: disk locally closed resulting in I/O error
WARNING: Read Failed. group:3 disk:1 AU:920 offset:16384 size:16384
path:Unknown disk
incarnation:0xe968aabb synchronous result:'I/O error'
subsys:Unknown library iop:0x7f71b15c1000 bufp:0x7f71b17a7e00 osderr:0x0 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 260 in group from disk DATADG_0001 allocation unit 920 reason error; if possible, will try another mirror side
DDE: Problem Key 'ORA 202' was flood controlled (0x5) (no incident)
ORA-00202: control file: '+DATADG/xxxxx/controlfile/current.260.853170001'
ORA-15081: failed to submit an I/O operation to a disk
*** 2014-08-27 12:17:00.458
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=1, mask=0x0)
----- Error Stack Dump -----
ORA-00202: control file: '+DATADG/xxxxx/controlfile/current.260.853170001'
ORA-15081: failed to submit an I/O operation to a disk
----- Current SQL Statement for this session (sql_id=6hz7hx94wrdkn) -----
select LOG_MODE from V$DATABASE
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+36 call kgdsdst() 000000000 ? 000000000 ?
7FFFFE34CBC8 ? 000000001 ?
000000001 ? 000000002 ?
ksedst1()+98 call skdstdst() 000000000 ? 000000000 ?
7FFFFE34CBC8 ? 000000001 ?
000000000 ? 000000002 ?
ksedst()+34 call ksedst1() 000000000 ? 000000001 ?
7FFFFE34CBC8 ? 000000001 ?
000000000 ? 000000002 ?
dbkedDefDump()+2741 call ksedst() 000000000 ? 000000001 ?
7FFFFE34CBC8 ? 000000001 ?
000000000 ? 000000002 ?
ksedmp()+36 call dbkedDefDump() 000000001 ? 000000000 ?
7FFFFE34CBC8 ? 000000001 ?
000000000 ? 000000002 ?
kcccsi()+126 call ksedmp() 000000001 ? 000000000 ?
7FFFFE34CBC8 ? 000000001 ?
000000000 ? 000000002 ?
kccrhd()+1626 call kcccsi() 0000000CC ? 000000001 ?
000000001 ? 000000001 ?
000000000 ? 000000002 ?
kccgft_refresh_hdr( call kccrhd() 7F71B178B478 ? 7FFFFE349400 ?
)+112 000000000 ? 000000002 ?
000000000 ? 000000002 ?
kccgftcs()+706 call kccgft_refresh_hdr( 7F71B178B478 ? 7FFFFE349400 ?
) 000000000 ? 000000002 ?
000000000 ? 000000002 ?
kccgft()+59 call kccgftcs() 7F71B178B468 ? 7F71B1780D00 ?
7F71B178B220 ? 000000000 ?
000000000 ? 000000000 ?
kccxdi()+81 call kccgft() 000000000 ? 000000000 ?
7FFFFE349400 ? 000000000 ?
000000000 ? 000000000 ?
qerfxFetch()+521 call kccxdi() 000000000 ? 000000000 ?
7F71B178B220 ? 7F71B1780D00 ?
8C34A7EA8 ? 000000000 ?
rwsfcd()+103 call qerfxFetch() 8C34A8100 ? 7F71B178AE98 ?
000000000 ? 000000000 ?
7F7100000001 ? 000000000 ?
qerjotFetch()+453 call rwsfcd() 8C34A7CA8 ? 7F71B178B868 ?
000000000 ? 000000000 ?
7F7100000001 ? 000000000 ?
opifch2()+2995 call qerjotFetch() 8C34A7A70 ? 7F71B17895E8 ?
00917A034 ? 7FFFFE352F50 ?
000000001 ? 8C34A7998 ?
kpoal8()+2929 call opifch2() 000000089 ? 000000005 ?
7FFFFE353560 ? 7F71B1788268 ?
000000001 ? 8C34A7998 ?
opiodr()+916 call kpoal8() 00000005E ? 00000001C ?
7FFFFE3564C0 ? 7F71B1788268 ?
000000001 ? 100000001 ?
ttcpip()+2242 call opiodr() 00000005E ? 00000001C ?
7FFFFE3564C0 ? 000000000 ?
009B916F0 ? 100000001 ?
opitsk()+1668 call ttcpip() 00BB09C30 ? 0094D6EB0 ?
7FFFFE3564C0 ? 000000000 ?
7FFFFE355F18 ? 7FFFFE3566B4 ?
opiino()+961 call opitsk() 00BB09C30 ? 000000001 ?
7FFFFE3564C0 ? 000000000 ?
7FFFFE355F18 ? 7FFFFE3566B4 ?
opiodr()+916 call opiino() 00000003C ? 000000004 ?
7FFFFE357C38 ? 000000000 ?
7FFFFE355F18 ? 7FFFFE3566B4 ?
opidrv()+565 call opiodr() 00000003C ? 000000004 ?
7FFFFE357C38 ? 000000000 ?
009B911A0 ? 7FFFFE3566B4 ?
sou2o()+98 call opidrv() 00000003C ? 000000004 ?
7FFFFE357C38 ? 000000000 ?
009B911A0 ? 7FFFFE3566B4 ?
opimai_real()+128 call sou2o() 7FFFFE357C10 ? 00000003C ?
000000004 ? 7FFFFE357C38 ?
009B911A0 ? 7FFFFE3566B4 ?
ssthrdmain()+252 call opimai_real() 000000002 ? 7FFFFE357E00 ?
000000004 ? 7FFFFE357C38 ?
009B911A0 ? 7FFFFE3566B4 ?
main()+196 call ssthrdmain() 000000002 ? 7FFFFE357E00 ?
000000001 ? 000000000 ?
009B911A0 ? 7FFFFE3566B4 ?
__libc_start_main() call main() 000000002 ? 7FFFFE357FA8 ?
+253 000000001 ? 000000000 ?
009B911A0 ? 7FFFFE3566B4 ?
_start()+36 call __libc_start_main() 000A0AF64 ? 000000002 ?
7FFFFE357F98 ? 000000000 ?
009B911A0 ? 7FFFFE3566B4 ?
# ls -l /dev/asm*
brw-rw---- 1 grid oinstall 120, 0 Aug 28 15:17 /dev/asm-arc1
brw-rw---- 1 grid oinstall 120, 80 Aug 23 00:03 /dev/asm-arc2
brw-rw---- 1 grid oinstall 120, 128 Aug 28 15:17 /dev/asm-data1
brw-rw---- 1 grid oinstall 120, 144 Aug 28 15:17 /dev/asm-data2
brw-rw---- 1 grid oinstall 120, 64 Aug 28 15:17 /dev/asm-flash
brw-rw---- 1 grid oinstall 120, 48 Aug 28 15:15 /dev/asm-ocr1
brw-rw---- 1 grid oinstall 120, 96 Aug 28 15:17 /dev/asm-ocr2
[ node2~]$ ls -l /dev/asm*
brw-rw---- 1 grid oinstall 120, 0 8月 28 12:44 /dev/asm-arc1
brw-rw---- 1 grid oinstall 120, 80 8月 28 12:44 /dev/asm-arc2
brw-rw---- 1 grid oinstall 120, 128 8月 28 15:20 /dev/asm-data1
brw-rw---- 1 grid oinstall 120, 144 8月 28 15:20 /dev/asm-data2
brw-rw---- 1 grid oinstall 120, 64 8月 28 15:20 /dev/asm-flash
brw-rw---- 1 grid oinstall 120, 48 8月 28 12:44 /dev/asm-ocr1
brw-rw---- 1 grid oinstall 120, 96 8月 28 15:20 /dev/asm-ocr2
日志发现如上，大家给看看到底是什么原因？
mos 里的文档查找类似文档：
WARNINGS: ORA-00204: ORA-00202: ORA-15081 in RDBMS alert log Affects ASM Access (Doc ID 1525250.1) : failed to submit an I/O operation to a disk
》》 strings /dev/asm-data1可以读取设备么
/var/log/messages有无特别信息 本帖最后由 北柏 于 2014-8-28 21:24 编辑
harryzhang 发表于 2014-8-28 17:30 static/image/common/back.gif
: failed to submit an I/O operation to a disk
》》 strings /dev/asm-data1可以读取设备么
节点1：
sdd:
sdc:
sdj: unknown partition table
unknown partition table
unknown partition table
sdk:
sdw:
sdx:
sdad: unknown partition table
unknown partition table
emcpowere: unknown partition table
unknown partition table
emcpowera: unknown partition table
emcpoweri:
sd 5:0:0:9: Very big device. Trying to use READ CAPACITY(16).
unknown partition table
unknown partition table
sdae: unknown partition table
emcpowerj: unknown partition table
emcpowerf: unknown partition table
emcpowerf: unknown partition table
emcpowerf: unknown partition table
emcpowerf: unknown partition table
emcpowerf: unknown partition table
sdb: unknown partition table
sdv: unknown partition table
emcpowerf: unknown partition table
emcpowerd: unknown partition table
emcpowerd: unknown partition table
emcpowerd: unknown partition table
emcpowerd: unknown partition table
emcpowerd: unknown partition table
sdf: unknown partition table
sdz: unknown partition table
emcpowerd: unknown partition table harryzhang 发表于 2014-8-28 17:30 static/image/common/back.gif
: failed to submit an I/O operation to a disk
》》 strings /dev/asm-data1可以读取设备么
节点二：
bnx2 0000:03:00.0: irq 133 for MSI/MSI-X
bnx2 0000:03:00.0: em3: using MSIX
ADDRCONF(NETDEV_UP): em3: link is not ready
bnx2 0000:03:00.0: em3: NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
ADDRCONF(NETDEV_CHANGE): em3: link becomes ready
p4-clockmod: Warning: EST-capable CPU detected. The acpi-cpufreq module offers voltage scaling in addition to frequency scaling. You should use that instead of p4-clockmod, if possible.
bond0: no IPv6 routers present
em3: no IPv6 routers present
emcpowere: unknown partition table
emcpowerf: unknown partition table
emcpowerj: unknown partition table
emcpowerd: unknown partition table
emcpowerg: unknown partition table
emcpowerd: unknown partition table
emcpowerg: unknown partition table
sde: unknown partition table
sdy: unknown partition table
emcpowerg: unknown partition table
emcpowera: unknown partition table
emcpoweri: unknown partition table
emcpowerj: unknown partition table
emcpowere: unknown partition table
sd 3:0:0:9: Very big device. Trying to use READ CAPACITY(16).
sdd:
sdc:
sdj: unknown partition table
unknown partition table
unknown partition table
sdk:
sdw:
sdx:
sdad: unknown partition table
unknown partition table
emcpowere: unknown partition table
unknown partition table
emcpowera: unknown partition table
emcpoweri:
sd 5:0:0:9: Very big device. Trying to use READ CAPACITY(16).
unknown partition table
unknown partition table
sdae: unknown partition table
emcpowerj: unknown partition table
emcpowerf: unknown partition table
emcpowerf: unknown partition table
emcpowerf: unknown partition table
emcpowerf: unknown partition table
emcpowerf: unknown partition table
sdb: unknown partition table
sdv: unknown partition table
emcpowerf: unknown partition table
emcpowerd: unknown partition table
emcpowerd: unknown partition table
emcpowerd: unknown partition table
emcpowerd: unknown partition table
emcpowerd: unknown partition table
sdf: unknown partition table
sdz: unknown partition table
emcpowerd: unknown partition table
页:
[1]