82 积分	0 好友	0 主题

发消息

oracle 11.2.0.1+asm+grid control+rac故障

1^#

发表于 2012-3-15 22:56:38 | 查看: 6561| 回复: 3

asm alert log报错部分如下：

WARNING: IO Failed. group:1 disk(number.incarnation):2.0x9698e5a6 disk_path:ORCL:CRS3
  AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
WARNING: IO Failed. group:1 disk(number.incarnation):2.0x9698e5a6 disk_path:ORCL:CRS3
  AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
WARNING: IO Failed. group:1 disk(number.incarnation):1.0x9698e5a5 disk_path:ORCL:CRS2
  AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
WARNING: IO Failed. group:1 disk(number.incarnation):1.0x9698e5a5 disk_path:ORCL:CRS2
  AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
WARNING: IO Failed. group:1 disk(number.incarnation):0.0x9698e5a4 disk_path:ORCL:CRS1
  AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
WARNING: IO Failed. group:1 disk(number.incarnation):0.0x9698e5a4 disk_path:ORCL:CRS1
  AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
ERROR: no PST quorum in group: required 1, found 0
ERROR: could not heartbeat- no PST found (grp 1)
WARNING: IO Failed. group:2 disk(number.incarnation):0.0x9698e5a7 disk_path:ORCL:FRAVOL1
  AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
WARNING: IO Failed. group:2 disk(number.incarnation):0.0x9698e5a7 disk_path:ORCL:FRAVOL1
  AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
ERROR: no PST quorum in group: required 1, found 0
ERROR: could not heartbeat- no PST found (grp 2)
WARNING: IO Failed. group:3 disk(number.incarnation):0.0x9698e5a8 disk_path:ORCL:FRAV1
  AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
WARNING: IO Failed. group:3 disk(number.incarnation):0.0x9698e5a8 disk_path:ORCL:FRAV1
  AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
ERROR: no PST quorum in group: required 1, found 0
ERROR: could not heartbeat- no PST found (grp 3)
WARNING: IO Failed. group:4 disk(number.incarnation):0.0x9698e5a9 disk_path:ORCL:DATAVOL1
  AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
WARNING: IO Failed. group:4 disk(number.incarnation):0.0x9698e5a9 disk_path:ORCL:DATAVOL1
  AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
ERROR: no PST quorum in group: required 1, found 0
ERROR: could not heartbeat- no PST found (grp 4)
WARNING: IO Failed. group:5 disk(number.incarnation):0.0x9698e5aa disk_path:ORCL:DATAV1
  AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
  result:I/O error process_id:8927
WARNING: IO Failed. group:5 disk(number.incarnation):0.0x9698e5aa disk_path:ORCL:DATAV1
  AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
执行
select name,group_number,state,type from v$asm_diskgroup;
查看磁盘状态为dismounted,并且 crs也down了

dmesg|more磁盘坏道情况
end_request: I/O error, dev sdb, sector 262224
end_request: I/O error, dev sdb, sector 262736
end_request: I/O error, dev sdb, sector 4150
end_request: I/O error, dev sdc, sector 2110
end_request: I/O error, dev sdd, sector 2110
end_request: I/O error, dev sde, sector 4151
end_request: I/O error, dev sdf, sector 208991
end_request: I/O error, dev sdf, sector 209087
end_request: I/O error, dev sdf, sector 217247
end_request: I/O error, dev sdf, sector 4151
end_request: I/O error, dev sdf, sector 5495721
end_request: I/O error, dev sdf, sector 709791
end_request: I/O error, dev sdg, sector 209087
end_request: I/O error, dev sdg, sector 38390703
end_request: I/O error, dev sdg, sector 4151
end_request: I/O error, dev sdg, sector 551849
end_request: I/O error, dev sdh, sector 2111
end_request: I/O error, dev sdh, sector 4151

alert_+ASM2.rar

25.46 KB, 下载次数: 1043

alert_rgszcg1.rar

87.73 KB, 下载次数: 1033

分享0

收藏0 回复只看该作者道具举报

dai_xuej

2^#

发表于 2012-3-15 23:02:19

重新启动服务器又正常了
[root@racnode1 bin]# ./crs_stat -t
Name          Type          Target State    Host
------------------------------------------------------------
ora.CRS.dg    ora....up.type ONLINE ONLINE racnode1
ora.FRA.dg    ora....up.type ONLINE ONLINE racnode1
ora.FRA01.dg ora....up.type ONLINE ONLINE racnode1
ora....ER.lsnr ora....er.type ONLINE ONLINE racnode1
ora....R1.lsnr ora....er.type ONLINE ONLINE racnode1
ora....N1.lsnr ora....er.type ONLINE ONLINE racnode2
ora....DATA.dg ora....up.type ONLINE ONLINE racnode1
ora....TA01.dg ora....up.type ONLINE ONLINE racnode1
ora.asm       ora.asm.type ONLINE ONLINE racnode1
ora.eons    ora.eons.type  ONLINE ONLINE racnode1
ora.gsd       ora.gsd.type ONLINE ONLINE racnode1
ora....network ora....rk.type ONLINE ONLINE racnode1
ora.oc4j    ora.oc4j.type  ONLINE ONLINE racnode2
ora.ons       ora.ons.type ONLINE ONLINE racnode1
ora....SM1.asm application ONLINE ONLINE racnode1
ora....E1.lsnr application ONLINE ONLINE racnode1
ora....E1.lsnr application ONLINE ONLINE racnode1
ora....de1.gsd application ONLINE ONLINE racnode1
ora....de1.ons application ONLINE ONLINE racnode1
ora....de1.vip ora....t1.type ONLINE ONLINE racnode1
ora....SM2.asm application ONLINE ONLINE racnode2
ora....E2.lsnr application ONLINE ONLINE racnode2
ora....E2.lsnr application ONLINE ONLINE racnode2
ora....de2.gsd application ONLINE ONLINE racnode2
ora....de2.ons application ONLINE ONLINE racnode2
ora....de2.vip ora....t1.type ONLINE ONLINE racnode2
ora....ry.acfs ora....fs.type ONLINE ONLINE racnode1
ora.rgszcg.db  ora....se.type ONLINE ONLINE racnode1
ora.scan1.vip  ora....ip.type ONLINE ONLINE racnode2

回复只看该作者道具举报

dai_xuej

3^#

发表于 2012-3-15 23:06:14

运行一段时间，可能一天后，又报如上错，又的重起机器，或者不重起机器，手动mount diskgroup然后再手动启动数据库。

回复只看该作者道具举报

teapot

4^#

发表于 2012-3-16 09:23:04

Bug 10422126 - Misleading "WARNING: IO Failed" alert log messages in ASM alert log [ID 10422126.8]

Bug 10055063 - Excessive trace in ASM "Warning: IO Failed ..." [ID 10055063.8]

操作系统有这些日志，那就表明存储可能有问题了。这些Warning是修复bug引起了另一个bug，会生成大量的trc，问题不大。控制文件没法读那就是系统、存储的问题了

[ 本帖最后由 teapot 于 2012-3-16 09:34 编辑 ]

回复只看该作者道具举报

返回列表

		自动登录	找回密码
密码			注册