在ASM DISKGROUP 创建表空间错误

orasupp

创建TABLESPACE遇到下面的ERROR,ASM DISKGROUP 也有FREE SPACE,重启了ASM也不行。

SQL> create tablespace test datafile '+DG_CS2PRD/cs2prd/datafile/cs2_ct_data_12.dbf' size 10m;
create tablespace test datafile '+DG_CS2PRD/cs2prd/datafile/cs2_ct_data_12.dbf' size 10m
*
ERROR at line 1:
ORA-01119: error in creating database file '+DG_CS2PRD/cs2prd/datafile/cs2_ct_data_12.dbf'
ORA-17502: ksfdcre:4 Failed to create file +DG_CS2PRD/cs2prd/datafile/cs2_ct_data_12.dbf
ORA-00600: internal error code, arguments: [kfdFstAlloc_3], [4], [1], [0], [], [], [], []


SQL> !
[oracle@pmrac01 new1]$ . oraenv
ORACLE_SID = [cs2prd1] ? +ASM1
[oracle@pmrac01 new1]$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Unbal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Name
MOUNTED  EXTERN  N      N         512   4096  1048576     51199    51103                0           51103              0  DG_B2B_ARCH/
MOUNTED  EXTERN  N      N         512   4096  1048576   3244026   421733                0          421733              0  DG_CS2PRD/

orasupp

resize datafile 也出错
SQL> alter database datafile '+DG_CS2PRD/cs2prd/datafile/afw_data_01.dbf' resize 100m;
alter database datafile '+DG_CS2PRD/cs2prd/datafile/afw_data_01.dbf' resize 100m
*
ERROR at line 1:
ORA-01237: cannot extend datafile 8
ORA-01110: data file 8: '+DG_CS2PRD/cs2prd/datafile/afw_data_01.dbf'
ORA-17505: ksfdrsz:1 Failed to resize file to size 12800 blocks
ORA-00600: internal error code, arguments: [kfdFstAlloc_3], [4], [1], [0], [], [], [], []

Maclean Liu(刘相兵

一个好的提问的 基本元素 ： 一个好的标题、 完整的版本信息、 日志信息 、 报错信息。

缺少 版本信息  和 600的完整日志。

orasupp

上传600 TRACE 文件

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
System name:    Linux
Release:        2.6.9-55.ELsmp

Maclean Liu(刘相兵

ODM DATA:

10.2.0.3.0 - 64bit Production+ Linux

ORA-00600: internal error code, arguments: [kfdFstAlloc_3], [4], [1], [0], [], [], [], []


stack call  ksbabs=>kfgbDriver=>kfgbRebalFile=>kfdaExecute=>kffRelocat=>kffsrdReloc=>kfdFstAlloc=>kfdFstAllocInt =>报错



Process global information:
     process: 0x6e92ac68, call: 0x6e4a2390, xact: (nil), curses: 0x6e43d0f0, usrses: 0x6e44b480
  ----------------------------------------
  SO: 0x6e92ac68, type: 2, owner: (nil), flag: INIT/-/-/0x00
  (process) Oracle pid=22, calls cur/top: 0x6e4a2390/0x6e4a2ea8, flag: (2) SYSTEM
            int error: 0, call error: 0, sess error: 0, txn error 0
  (post info) last post received: 0 0 270
              last post received-location: kfclRegOpen
              last process to post me: 6e9235d0 1 6
              last post sent: 0 0 24
              last post sent-location: ksasnd
              last process posted by me: 6e926540 1 6
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: 0x6e9e7358
    O/S info: user: oracle, term: UNKNOWN, ospid: 9352
    OSD pid info: Unix process pid: 9352, image: oracle@pmrac01.cargosmart.com (ARB0)

    SO: 0x6e6ec8d8, type: 11, owner: 0x6e92ac68, flag: INIT/-/-/0x00
    (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 0x6e92ac68,
                       event: 11, last message event: 11,
                       last message waited event: 11, messages read: 0
                       channel: (0x6e706138) scumnt mount lock
                                scope: 1, event: 11, last mesage event: 0,
                                publishers/subscribers: 0/11,
                                messages published: 0
    ----------------------------------------
    SO: 0x6e44b480, type: 4, owner: 0x6e92ac68, flag: INIT/-/-/0x00
    (session) sid: 426 trans: (nil), creator: 0x6e92ac68, flag: (51) USR/- BSY/-/-/-/-/-
              DID: 0000-0016-0000040A, short-term DID: 0000-0000-00000000
              txn branch: (nil)
              oct: 0, prv: 0, sql: (nil), psql: (nil), user: 0/SYS
    last wait for 'GCS lock esc X' blocking sess=0x(nil) seq=3 wait_time=530 seconds since wait started=0
                group=4, obj#=11f, block#=1
    Dumping Session Wait History
     for 'GCS lock esc X' count=1 wait_time=530
                group=4, obj#=11f, block#=1
     for 'enq: AM - rollback COD reservation' count=1 wait_time=235
                name|mode=414d0006, id1=3, id2=40000
     for 'rdbms ipc message' count=1 wait_time=846772
                timeout=12c, =0, =0
    temporary object counter: 0


ASM 后台ARB0 进程在等GCS lock esc X， call指向一个kfgso


  SO: 0x6e4a2ea8, type: 3, owner: 0x6e92ac68, flag: INIT/-/-/0x00
    (call) sess: cur 6e44b480, rec 6e43d0f0, usr 6e44b480; depth: 0
      ----------------------------------------
      SO: 0x6e4a2390, type: 3, owner: 0x6e4a2ea8, flag: INIT/-/-/0x00
      (call) sess: cur 6e43d0f0, rec 6e43d0f0, usr 6e44b480; depth: 1
        ----------------------------------------
        SO: 0x6afbe100, type: 84, owner: 0x6e4a2390, flag: INIT/-/-/0x00
        (kfgso) flags: 00000000 clt: 2 err: 0 hint: 6afbf980
        (kfgpn) rpi: 12341 itrn:(nil) gst:(nil) usrp:(nil)
        busy: 0 rep: 0 grp: 0 check: 0/0 glink: 6afbe180 6afbe180
          ----------------------------------------
          SO: 0x6afbf980, type: 85, owner: 0x6afbe100, flag: INIT/-/-/0x00
          (kfgsc) pwr: 0 flag: 0 op: 0 recur: 0
          (kfgpn) rpi: 8899 itrn:0x6a6896c0 gst:0x6a68e110 usrp:(nil)
          busy: 0 rep: 0 grp: 6afc0960 check: 4/608036412 glink: 6afc09a8 6a49ea10
            kfuitrn: 0x6a6896c0, cur: 0, cnt: 6
              0x0x6afa7160  0x0x6afa7468  0x0x6afa6528  0x0x6afa6830
              0x0x6afa6b38  0x0x6afa6e58



KST TRACE:
C964808B:003197F7    22   426 10005   2 KSL WAIT END [GCS lock esc X] 4/0x4 287/0x11f 1/0x1 time=530
C964808D:003197F8    22   426 10005   3 KSL POST RCVD poster=7 loc='kfclRegOpen' id1=0 id2=0 name=   type=0 fac#=3 facpost=1
C964808E:003197F9    22   426 10495   6 kfcRemovePin: pin=28487 bnum=4025 kfc.c 11013
C964808F:003197FA    22   426 10495  13 kfcGetPin: get: pin=28487 bnum=4025 flags_kfcbh=00000000 compat w/ pins=T compat w/ waits=T head of Queue=T posted=F already pin=F
C964808F:003197FB    22   426 10495   3 kfcAddPin: pin=28487 kfc.c 1695 excl bnum=4025 class=1100
C964809D:003197FC    22   426 10495  23 kfc autopin trigger: dsk=1 blk=1 class=1300 target: fn=1 blk=287
C964809E:003197FD    22   426 10495  10 kfcbpInit: gn=4 fn=1 blk=287 pin=28488 shar current kfc.c 8779
C964809F:003197FE    22   426 10495  13 kfcGetPin: get: pin=28488 bnum=3771 flags_kfcbh=00000000 compat w/ pins=T compat w/ waits=T head of Queue=F posted=F already pin=T
C964809F:003197FF    22   426 10495   3 kfcAddPin: pin=28488 kfc.c 1695 shar bnum=3771 class=300
C96480A0:00319800    22   426 10495  10 kfcbpInit: gn=4 dsk=1 blk=1 pin=28489 shar current kfd.c 8915
C96480A5:00319801    22   426 10495  13 kfcGetPin: get: pin=28489 bnum=3196 flags_kfcbh=00000000 compat w/ pins=T compat w/ waits=T head of Queue=F posted=F already pin=F
C96480A6:00319802    22   426 10495   3 kfcAddPin: pin=28489 kfc.c 1695 shar bnum=3196 class=1300
C96480A9:00319803    22   426 10495  10 kfcbpInit: gn=4 dsk=1 blk=26 pin=28490 excl current kfd.c 8964
C96480AD:00319804    22   426 10495  13 kfcGetPin: get: pin=28490 bnum=3197 flags_kfcbh=00000000 compat w/ pins=T compat w/ waits=T head of Queue=F posted=F already pin=F
C96480AD:00319805    22   426 10495   3 kfcAddPin: pin=28490 kfc.c 1695 excl bnum=3197 class=1400

Maclean Liu(刘相兵

advice:

1. 尝试设置 asm_power_limit=0 禁用asm rebalance

alter system set asm_power_limit=0;

重新尝试create tablespace

or

2. 升级ASM 到10.2.0.5 ，备份DB 到文件系统，之后drop diskgroup ，重建一个diskgrop

orasupp

尝试了第一种方法后，还是同样的错误。上传TRACE文件。

这个DB有2T,第二种方法EFFORT太大且目前没有这么大的文件系统。

Maclean Liu(刘相兵

尝试 在允许的情况下 repair diskgroup :

alter diskgroup  DG_CS2PRD check all repair;

orasupp

上传alert log
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15049: diskgroup "DG_CS2PRD" contains 1125 error(s)

Maclean Liu(刘相兵

ODM DATA:
SQL> alter diskgroup  DG_CS2PRD check all repair
Mon Jun  4 00:28:53 2012
NOTE: starting check of diskgroup DG_CS2PRD
ERROR: file +dg_cs2prd.263.745070755: F263 PX11656 => D1 A0 => F263 PX9824: xnum mismatch
ERROR: file +dg_cs2prd.263.745070755: F263 PX11662 => D1 A1 => F263 PX9830: xnum mismatch
ERROR: file +dg_cs2prd.263.745070755: F263 PX11668 => D1 A2 => F263 PX9836: xnum mismatch
ERROR: file +dg_cs2prd.263.745070755: F263 PX11674 => D1 A3 => F263 PX9842: xnum mismatch
Mon Jun  4 00:30:04 2012
ERROR: disk CS2PRD_DATA_06, AT 23: D1 A10304 => F263 X9824 => D7 A9014: disk mismatch
ERROR: disk CS2PRD_DATA_06, AT 23: D1 A10305 => F263 X9830 => D4 A9039: disk mismatch
ERROR: disk CS2PRD_DATA_06, AT 23: D1 A10306 => F263 X9836 => D6 A9053: disk mismatch
ERROR: disk CS2PRD_DATA_06, AT 23: D1 A10307 => F263 X9842 => D5 A9077: disk mismatch

ODM FINDING:

AU 352, while getting deallocated, AT was referred to, but extent# 1109 and
File# 258 was obtained, resulting in the discrepancy and the assert.

_asm_ausize              = 16777216
_asm_stripesize          = 1048576

'check all norepair' is reporting:

ERROR: file +dg_data1.256.694022821: F256 PX91 => D6 A284 => F257 PX40: fnum
mismatch

This file 256 is not there in the db side ( v$datafile/dba_data_files ) also
not present physically ( asmcmd ). So is an orphan entry.

'check all repair' is not able to fix this.


ASM diskgroup 存在较多讹误，就现有的metalink note看 不存在此问题的其他workaround方案， 建议重建diskgroup