3 积分	0 好友	2 主题

发消息

[Exadata硬件] nub备份归档文件时，产生大量IO导致系统HANG，请前辈指点

1^#

发表于 2014-3-27 01:13:10 | 查看: 5086| 回复: 2

操作系统：AIX 6.1
数据库版本：ORACLE 10.2.0.4 RAC
故障现象：当NBU脚本执行到备份归档文件时，SWAP盘瞬间IO 100%，整个系统HANG住，应用无法连接数据库。只有取消掉NBU备份，过一会才能正常，目前一直无法完成备份任务。
观察crsd.log时，发现有如下提示：
32CAAMonitorHandler :: 0:Action Script /oracle/product/10.2.0/crs/bin/racgwrap(check) timed out for ora.imsdb1.vip! (timeout=60)
以为遇到Bug 6196746，但是进程里面没有过多的racgmain累积。因为观察到SWAP空间IO 100%，认为内存不足，但是topas内存使用50%左右，SWAP使用30%左右，但是SWAP盘仍有较多的PI PO
附AWR、LOG、系统参数文件，请前辈给予指点，谢谢！

awrrpt_1_3100_3101.html (424.95 KB, 下载次数: 723)

param.txt (24.98 KB, 下载次数: 1147)

crsd.zip (9.1 KB, 下载次数: 1383)

分享0

收藏0 回复只看该作者道具举报

Liu Maclean(刘相兵

2^#

发表于 2014-3-27 11:48:06

就AWR看解析等待居多：
Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class
latch: library cache    124,490    31,081    250    226.0 Concurrency
cursor: pin S wait on X    1,334,178    13,678    10    99.5 Concurrency
CPU time         3,508         25.5
latch: shared pool    9,581    2,386    249    17.4 Concurrency

可能是LCK进程当时HANG住了，导致其HANG的可能包括 CRS不正常或者资源不足等等。

需要当时的一些SYSTEM级别DUMP，你可以查看alert.log中是否显示生成了此种DUMP

回复只看该作者道具举报

leavesws

3^#

发表于 2014-3-27 14:51:29

alert.log中没有dump信息，要做系统级别dump需要重现故障，会导致应用系统中断太久。看能否申请维护窗口

Mon Mar 24 22:57:14 2014
Thread 1 advanced to log sequence 5827 (LGWR switch)
  Current log# 2 seq# 5827 mem# 0: /dev/rimsh_redo02
Mon Mar 24 23:03:24 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:03:24 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:04:57 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:09:41 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:10:10 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:10:11 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:17:04 2014
Expanded controlfile section 15 from 1302 to 2604 records
Requested to grow by 1302 records; added 6 blocks of records
Mon Mar 24 23:25:47 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:25:58 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:26:01 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:26:02 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:26:13 2014
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 24 23:35:28 2014
Thread 1 advanced to log sequence 5828 (LGWR switch)
  Current log# 1 seq# 5828 mem# 0: /dev/rimsh_redo01
Mon Mar 24 23:40:34 2014
Thread 1 advanced to log sequence 5829 (LGWR switch)
  Current log# 5 seq# 5829 mem# 0: /dev/rimsh_redo03
Mon Mar 24 23:45:17 2014
Thread 1 advanced to log sequence 5830 (LGWR switch)
  Current log# 2 seq# 5830 mem# 0: /dev/rimsh_redo02
Mon Mar 24 23:53:50 2014
Thread 1 advanced to log sequence 5831 (LGWR switch)
  Current log# 1 seq# 5831 mem# 0: /dev/rimsh_redo01
Tue Mar 25 00:03:56 2014
Thread 1 advanced to log sequence 5832 (LGWR switch)
  Current log# 5 seq# 5832 mem# 0: /dev/rimsh_redo03

alert_imscmdb1.zip

87.33 KB, 下载次数: 1371

回复只看该作者道具举报

返回列表

		自动登录	找回密码
密码			注册