Oracle数据库数据恢复、性能优化

找回密码
注册
搜索
热搜: 活动 交友 discuz
Oracle数据库数据恢复、性能优化»论坛 › Oracle › Oracle数据库管理 › 求助ASM加disk出问题了
发新帖
yuyuxp

0

积分

0

好友

3

主题

[Exadata常规管理] 求助ASM加disk出问题了

1#
发表于 2015-2-15 16:22:12 | 查看: 5480| 回复: 3

今天早上把raw7加入了磁盘组DATA1，

成功了。
我继续加入raw8，后来一直半个小时没有反应，业务也一度很卡。
后来状态就成这样了

GROUP_NUMBER DISK_NUMBER MOUNT_STAT HEADER_STATU MODE_ST STATE    FAILGROUP    TOTAL_MB    FREE_MB NAME       PATH
------------ ----------- ---------- ------------ ------- -------- ---------- ---------- ---------- ---------- --------------------
           0           0 CLOSED     FOREIGN      ONLINE  NORMAL                     290          0            /dev/raw/raw1
           0           1 CLOSED     FOREIGN      ONLINE  NORMAL                     196          0            /dev/raw/raw5
           0           2 CLOSED     MEMBER       ONLINE  NORMAL                  307196          0            /dev/raw/raw8
           0           3 CLOSED     FOREIGN      ONLINE  NORMAL                     196          0            /dev/raw/raw3
           0           5 CLOSED     FOREIGN      ONLINE  NORMAL                     196          0            /dev/raw/raw4
           0           4 CLOSED     FOREIGN      ONLINE  NORMAL                     290          0            /dev/raw/raw2
           1           0 CACHED     MEMBER       ONLINE  NORMAL   DATA1_0000     141902       3312 DATA1_0000 /dev/raw/raw6
           1           1 CACHED     MEMBER       ONLINE  NORMAL   DATA1_0001     286714     283398 DATA1_0001 /dev/raw/raw7

SQL> select group_number,block_size,name,allocation_unit_size,state,type,total_mb,free_mb,offline_disks from v$asm_diskgroup;

GROUP_NUMBER BLOCK_SIZE NAME       ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB OFFLINE_DISKS
------------ ---------- ---------- -------------------- ----------- ------ ---------- ---------- -------------
           1       4096 DATA1                   1048576 MOUNTED     EXTERN     428616     286710            


raw8看来是没能成功加入磁盘组。
发现了系统的日志有磁盘报错：
Feb 15 11:36:40 server01 kernel: lpfc_scsi_prep_dma_buf_s3: Too many sg segments from dma_map_sg.  Config 64, seg_cnt 128
Feb 15 11:37:11 server01 last message repeated 1827 times
Feb 15 11:38:12 server01 last message repeated 6778 times
Feb 15 11:39:13 server01 last message repeated 6777 times
Feb 15 11:40:14 server01 last message repeated 6777 times
Feb 15 11:40:35 server01 last message repeated 2344 times
Feb 15 11:40:35 server01 kernel: INFO: task MpxPeriodicCall:3517 blocked for more than 120 seconds.
Feb 15 11:40:35 server01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 15 11:40:35 server01 kernel: MpxPeriodicCa D ffff81000100caa0     0  3517      1          3518  3516 (L-TLB)
Feb 15 11:40:35 server01 kernel:  ffff81041d4559d0 0000000000000046 0000000100000001 ffff81042da34b00
Feb 15 11:40:35 server01 kernel:  0000000000000086 000000000000000a ffff81042e098040 ffff81042ff12100
Feb 15 11:40:35 server01 kernel:  00004e04456192c3 0000000000008cd8 ffff81042e098228 000000018003ddd5
Feb 15 11:40:35 server01 kernel: Call Trace:
Feb 15 11:40:35 server01 kernel:  [<ffffffff80064167>] wait_for_completion+0x79/0xa2
Feb 15 11:40:35 server01 kernel:  [<ffffffff8008e16d>] default_wake_function+0x0/0xe
Feb 15 11:40:35 server01 kernel:  [<ffffffff801459cc>] blk_execute_rq_nowait+0x86/0x9a
Feb 15 11:40:35 server01 kernel:  [<ffffffff80145a78>] blk_execute_rq+0x98/0xc0
Feb 15 11:40:35 server01 kernel:  [<ffffffff885555fd>] :emcp:emcp_scsi_cmd_ioctl+0x2ad/0x400
Feb 15 11:40:35 server01 kernel:  [<ffffffff8855b471>] :emcp:PowerPlatformBottomDispatch+0x461/0x550
Feb 15 11:40:35 server01 kernel:  [<ffffffff8855b5da>] :emcp:PowerSyncIoBottomDispatch+0x7a/0xd0
Feb 15 11:40:35 server01 kernel:  [<ffffffff8855bb23>] :emcp:PowerDispatchX+0x353/0x410
Feb 15 11:40:35 server01 kernel:  [<ffffffff8855e88d>] :emcp:EmsInquiry+0x9d/0x1a0
Feb 15 11:40:35 server01 kernel:  [<ffffffff8877c28a>] :emcpmpx:ClariionKLam_getPathLunStatus+0x8a/0x110
Feb 15 11:40:35 server01 kernel:  [<ffffffff8876c0f1>] :emcpmpx:MpxDefaultTestPath+0x21/0x80
Feb 15 11:40:35 server01 kernel:  [<ffffffff8877b77f>] :emcpmpx:MpxLnxTestPath+0x3f/0x270
Feb 15 11:40:35 server01 kernel:  [<ffffffff88773e84>] :emcpmpx:MpxTestPath+0x244/0x1f50
Feb 15 11:40:35 server01 kernel:  [<ffffffff80063ff8>] thread_return+0x62/0xfe
Feb 15 11:40:35 server01 kernel:  [<ffffffff8877854b>] :emcpmpx:MpxPeriodicTestPath+0x9b/0x230
Feb 15 11:40:35 server01 kernel:  [<ffffffff8006f1f5>] do_gettimeofday+0x40/0x90
Feb 15 11:40:35 server01 kernel:  [<ffffffff887789d2>] :emcpmpx:MpxPeriodicCallout+0x2f2/0x410
Feb 15 11:40:35 server01 kernel:  [<ffffffff887786e0>] :emcpmpx:MpxPeriodicCallout+0x0/0x410
Feb 15 11:40:35 server01 kernel:  [<ffffffff8855c3ad>] :emcp:PowerServiceDaemonQ+0xad/0xd0
Feb 15 11:40:35 server01 kernel:  [<ffffffff8005efb1>] child_rip+0xa/0x11
Feb 15 11:40:35 server01 kernel:  [<ffffffff8855c3d0>] :emcp:PowerDaemonStart+0x0/0x20
Feb 15 11:40:35 server01 kernel:  [<ffffffff8005efa7>] child_rip+0x0/0x11
报错一直持续了半个小时，就是卡住的时间。


asm的alert
Sun Feb 15 11:35:30 2015
SQL> alter diskgroup DATA1 add disk '/dev/raw/raw7'
Sun Feb 15 11:35:30 2015
NOTE: reconfiguration of group 1/0x11f8828e (DATA1), full=1
Sun Feb 15 11:35:30 2015
NOTE: initializing header on grp 1 disk DATA1_0001
NOTE: cache opening disk 1 of grp 1: DATA1_0001 path:/dev/raw/raw7
NOTE: PST update: grp = 1
NOTE: requesting all-instance disk validation for group=1
Sun Feb 15 11:35:30 2015
NOTE: disk validation pending for group 1/0x11f8828e (DATA1)
SUCCESS: validated disks for 1/0x11f8828e (DATA1)
Sun Feb 15 11:35:31 2015
NOTE: PST update: grp = 1
NOTE: requesting all-instance membership refresh for group=1
Sun Feb 15 11:35:31 2015
NOTE: membership refresh pending for group 1/0x11f8828e (DATA1)
SUCCESS: refreshed membership for 1/0x11f8828e (DATA1)
Sun Feb 15 11:35:34 2015
NOTE: requesting all-instance membership refresh for group=1
Sun Feb 15 11:35:34 2015
NOTE: membership refresh pending for group 1/0x11f8828e (DATA1)
SUCCESS: refreshed membership for 1/0x11f8828e (DATA1)
Sun Feb 15 11:35:40 2015
NOTE: starting rebalance of group 1/0x11f8828e (DATA1) at power 1
Starting background process ARB0
ARB0 started with pid=17, OS id=2661
Sun Feb 15 11:35:40 2015
NOTE: assigning ARB0 to group 1/0x11f8828e (DATA1)
Sun Feb 15 11:35:40 2015
NOTE: X->S down convert bast on F1B3 bastCount=2
NOTE: X->S down convert bast on F1B3 bastCount=3
NOTE: X->S down convert bast on F1B3 bastCount=4
NOTE: X->S down convert bast on F1B3 bastCount=5
NOTE: X->S down convert bast on F1B3 bastCount=6
NOTE: X->S down convert bast on F1B3 bastCount=7
NOTE: X->S down convert bast on F1B3 bastCount=8
NOTE: X->S down convert bast on F1B3 bastCount=9
NOTE: X->S down convert bast on F1B3 bastCount=10
NOTE: X->S down convert bast on F1B3 bastCount=11
NOTE: X->S down convert bast on F1B3 bastCount=12
NOTE: X->S down convert bast on F1B3 bastCount=13
NOTE: X->S down convert bast on F1B3 bastCount=14
NOTE: X->S down convert bast on F1B3 bastCount=15
NOTE: X->S down convert bast on F1B3 bastCount=16
NOTE: X->S down convert bast on F1B3 bastCount=17
NOTE: X->S down convert bast on F1B3 bastCount=18
NOTE: X->S down convert bast on F1B3 bastCount=19
NOTE: X->S down convert bast on F1B3 bastCount=20
NOTE: X->S down convert bast on F1B3 bastCount=21
NOTE: X->S down convert bast on F1B3 bastCount=22
NOTE: X->S down convert bast on F1B3 bastCount=23
NOTE: X->S down convert bast on F1B3 bastCount=24
NOTE: X->S down convert bast on F1B3 bastCount=25
NOTE: X->S down convert bast on F1B3 bastCount=26
NOTE: X->S down convert bast on F1B3 bastCount=27
NOTE: X->S down convert bast on F1B3 bastCount=28
NOTE: X->S down convert bast on F1B3 bastCount=29
Sun Feb 15 11:36:39 2015
SQL> alter diskgroup DATA1 add disk '/dev/raw/raw8'
Sun Feb 15 11:36:39 2015
ERROR: ORA-1013 thrown in ARB0 for group number 1
Sun Feb 15 11:36:39 2015
Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_arb0_2661.trc:
ORA-01013: user requested cancel of current operation
Sun Feb 15 11:36:39 2015
NOTE: stopping process ARB0
Sun Feb 15 11:36:40 2015
NOTE: rebalance interrupted for group 1/0x11f8828e (DATA1)
NOTE: reconfiguration of group 1/0x11f8828e (DATA1), full=1
Sun Feb 15 12:24:40 2015
NOTE: initializing header on grp 1 disk DATA1_0002
NOTE: cache opening disk 2 of grp 1: DATA1_0002 path:/dev/raw/raw8
NOTE: requesting all-instance membership refresh for group=1
Sun Feb 15 12:24:40 2015
NOTE: membership refresh pending for group 1/0x11f8828e (DATA1)
SUCCESS: validated disks for 1/0x11f8828e (DATA1)
NOTE: cache closing disk 2 of grp 1: DATA1_0002 path:/dev/raw/raw8
NOTE: cache closing disk 2 of grp 1: DATA1_0002 path:/dev/raw/raw8
SUCCESS: refreshed membership for 1/0x11f8828e (DATA1)
现在该如何操作好？
2#
发表于 2015-2-15 16:52:11
Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_arb0_2661.trc:
ORA-01013: user requested cancel of current operation
Sun Feb 15 11:36:39 2015
NOTE: stopping process ARB0
Sun Feb 15 11:36:40 2015
NOTE: rebalance interrupted for group 1/0x11f8828e (DATA1)
NOTE: reconfiguration of group 1/0x11f8828e (DATA1), full=1
Sun Feb 15 12:24:40 2015
NOTE: initializing header on grp 1 disk DATA1_0002
NOTE: cache opening disk 2 of grp 1: DATA1_0002 path:/dev/raw/raw8
NOTE: requesting all-instance membership refresh for group=1
Sun Feb 15 12:24:40 2015
NOTE: membership refresh pending for group 1/0x11f8828e (DATA1)
SUCCESS: validated disks for 1/0x11f8828e (DATA1)
NOTE: cache closing disk 2 of grp 1: DATA1_0002 path:/dev/raw/raw8
NOTE: cache closing disk 2 of grp 1: DATA1_0002 path:/dev/raw/raw8
SUCCESS: refreshed membership for 1/0x11f8828e (DATA1)


raw8在metadata级别应当已经加入diskgroup ，但  rebalance interrupted  rebalance中断了，其上面可能没有数据

如果数据库现在OK 的话， 先对全库做备份 ， 然后shutdown DB  ,SHUTDOWN ASM ,重启OS， 之后重新MOUNT DISKGROUP吧

回复 只看该作者 道具 举报

3#
发表于 2015-2-15 17:05:19
Liu Maclean(刘相兵 发表于 2015-2-15 16:52
Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_arb0_2661.trc:
ORA-01013: user requested cance ...

感谢刘大。
目前db没有问题，准备晚上rman全备

回复 只看该作者 道具 举报

4#
发表于 2015-2-27 10:43:24
Liu Maclean(刘相兵 发表于 2015-2-15 16:52
Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_arb0_2661.trc:
ORA-01013: user requested cance ...

昨天重启系统也不行，还是原来的状态
后来通过
SQL> Alter diskgroup data1 add disk '/dev/raw/raw8' name  DATA1_0002 force;
强行加上就正常了

回复 只看该作者 道具 举报

返回列表
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 注册

QQ|手机版|Archiver|Oracle数据库数据恢复、性能优化

GMT+8, 2024-12-21 01:11 , Processed in 0.047947 second(s), 20 queries .

Powered by Discuz! X2.5

© 2001-2012 Comsenz Inc.

回顶部
TEL/電話+86 13764045638
Email service@parnassusdata.com
QQ 47079569