ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0], [1]

newcom

系统版本： redhat 6.4
Oracle版本： Oracle 11.2.0.1.0 64bit

错误信息：
alert文件
Mon Dec 23 20:22:06 2013
Errors in file /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/trace/gsjdata_m000_4313.trc  (incident=177451):
ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0], [1], [], [], [], [], [], [], [], []
Incident details in: /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/incident/incdir_177451/gsjdata_m000_4313_i177451.trc
Mon Dec 23 20:23:00 2013
Trace dumping is performing id=[cdmp_20131223202300]
Mon Dec 23 20:23:00 2013
Errors in file /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/trace/gsjdata_m000_4313.trc  (incident=177452):
ORA-00600: internal error code, arguments: [kewrose_1], [600], [ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk
], [0], [0], [1], [], [], [], [], [], [], [], []
], [], [], [], [], [], [], [], [], []
Incident details in: /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/incident/incdir_177452/gsjdata_m000_4313_i177452.trc
Trace dumping is performing id=[cdmp_20131223202302]
Suspending MMON slave action kewrmafsa_ for 82800 seconds
Errors in file /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/trace/gsjdata_m000_4313.trc:
ORA-00600: internal error code, arguments: [kewrose_1], [600], [ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk
], [0], [0], [1], [], [], [], [], [], [], [], []
], [], [], [], [], [], [], [], [], []

Trac 文件
[oracle@gzszdb ~]$ more /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/trace/gsjdata_m000_4313.trc
Trace file /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/trace/gsjdata_m000_4313.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /app/ora11g/oracle/product/11.2.0/db_1
System name:    Linux
Node name:      gzszdb
Release:        2.6.18-308.el5
Version:        #1 SMP Tue Feb 21 20:06:06 EST 2012
Machine:        x86_64
Instance name: gsjdata
Redo thread mounted by this instance: 1
Oracle process number: 27
Unix process pid: 4313, image: oracle@gzszdb (M000)


*** 2013-12-23 20:22:05.633
*** SESSION ID:(300.36948) 2013-12-23 20:22:05.633
*** CLIENT ID:() 2013-12-23 20:22:05.633
*** SERVICE NAME:(SYS$BACKGROUND) 2013-12-23 20:22:05.633
*** MODULE NAME:(MMON_SLAVE) 2013-12-23 20:22:05.633
*** ACTION NAME:(Auto-Flush Slave Action) 2013-12-23 20:22:05.633


*** SESSION ID:(300.36948) 2013-12-23 20:22:05.628
OBJD MISMATCH typ=35, seg.obj=-2, diskobj=671894, dsflg=0, dsobj=676581, tid=676581, cls=4

*** 2013-12-23 20:22:06.172
Incident 177451 created, dump file: /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/incident/incdir_177451/gsjdata_m000_4313_i177451.trc
ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0], [1], [], [], [], [], [], [], [], []

*** KEWROCISTMTEXEC - encountered error: (ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0], [1], [],
[], [], [], [], [], [], []
)  
  *** SQLSTR: total-len=2100, dump-len=240,
      STR={insert into wrh$_seg_stat   (snap_id, dbid, instance_number, ts#, obj#, dataobj#, logical_reads_total,    logical_rea
ds_delta, buffer_busy_waits_total, buffer_busy_waits_delta,    db_block_changes_total, db_block_changes_delta, physical_rea}
Incident 177452 created, dump file: /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/incident/incdir_177452/gsjdata_m000_4313_i177451.trc
ORA-00600: internal error code, arguments: [kewrose_1], [600], [ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk
], [0], [0], [1], [], [], [], [], [], [], [], []
], [], [], [], [], [], [], [], [], []

*** KEWRAFM1: Error=600 encountered by kewrfteh
*** KEWRAFM1: Error=600 encountered by kewrfteh
*** KEWRAFS: Error=600 encountered by Auto Flush Slave.
KEBM: MMON slave action policy violation. kewrmafsa_; viol=0; err=600

*** 2013-12-23 20:23:02.328
ORA-00600: internal error code, arguments: [kewrose_1], [600], [ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk
], [0], [0], [1], [], [], [], [], [], [], [], []
], [], [], [], [], [], [], [], [], []
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process

dump 文件
[oracle@gzszdb ~]$ more  /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/incident/incdir_177451/gsjdata_m000_4313_i177451.trc
Dump file /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/incident/incdir_177451/gsjdata_m000_4313_i177451.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /app/ora11g/oracle/product/11.2.0/db_1
System name:    Linux
Node name:      gzszdb
Release:        2.6.18-308.el5
Version:        #1 SMP Tue Feb 21 20:06:06 EST 2012
Machine:        x86_64
Instance name: gsjdata
Redo thread mounted by this instance: 1
Oracle process number: 27
Unix process pid: 4313, image: oracle@gzszdb (M000)


*** 2013-12-23 20:22:06.173
*** SESSION ID:(300.36948) 2013-12-23 20:22:06.173
*** CLIENT ID:() 2013-12-23 20:22:06.173
*** SERVICE NAME:(SYS$BACKGROUND) 2013-12-23 20:22:06.173
*** MODULE NAME:(MMON_SLAVE) 2013-12-23 20:22:06.173
*** ACTION NAME:(Auto-Flush Slave Action) 2013-12-23 20:22:06.173

Dump continued from file: /app/ora11g/oracle/diag/rdbms/gsjdata/gsjdata/trace/gsjdata_m000_4313.trc
ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0], [1], [], [], [], [], [], [], [], []

========= Dump for incident 177451 (ORA 600 [ktecgsc:kcbz_objdchk]) ========
----- Beginning of Customized Incident Dump(s) -----
Input data (nil), 0, 0
Formatted dump of block:
buffer tsn: 2 rdba: 0x00c07c8b (3/31883)
scn: 0x0015.c6026c95 seq: 0x01 flg: 0x04 tail: 0x6c952301
frmt: 0x02 chkval: 0x2d37 type: 0x23=PAGETABLE SEGMENT HEADER
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x00000001DF4B4000 to 0x00000001DF4B6000
1DF4B4000 0000A223 00C07C8B C6026C95 04010015  [#....|...l......]
1DF4B4010 00002D37 00000000 00000000 00000000  [7-..............]
1DF4B4020 00000000 00000007 00000038 00000A9C  [........8.......]
1DF4B4030 00000006 00000008 00000008 00C08679  [............y...]

确认object

SQL> select * from v$tablespace where ts#=2;

       TS# NAME                           INCLUDED_IN_DATABASE_BACKUP BIGFILE FLASHBACK_ON ENCRYPT_IN_BACKUP
---------- ------------------------------ --------------------------- ------- ------------ -----------------
         2 SYSAUX                         YES                         NO      YES         

SQL> select * from seg$ where file#=3 and block#=31883;

     FILE#     BLOCK#      TYPE#        TS#     BLOCKS    EXTENTS    INIEXTS    MINEXTS    MAXEXTS    EXTSIZE     EXTPCT      USER#      LISTS     GROUPS BITMAPRANGES  CACHEHINT   SCANHINT    HWMINCR     SPARE1     SPARE2
---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ------------ ---------- ---------- ---------- ---------- ----------
         3      31883          5          2         64          8          8          1 2147483645        128          0          0          0          0            0          0          0     676581     131329

SQL>
SQL> select object_id, data_object_id, owner, object_name, object_type from dba_objects
  2  where data_object_id in (671894, 676581) or object_id in (671894, 676581);

OBJECT_ID DATA_OBJECT_ID OWNER                          OBJECT_NAME                                                                      OBJECT_TYPE
---------- -------------- ------------------------------ -------------------------------------------------------------------------------- -------------------
    676581         676581 SYS                            WRH$_SEG_STAT                                                                    TABLE PARTITION

通过分析日志可以知道是在MMON进程在更新WRH$_SEG_STAT表的统计信息是报错
  *** SQLSTR: total-len=2100, dump-len=240,
      STR={insert into wrh$_seg_stat   (snap_id, dbid, instance_number, ts#, obj#, dataobj#, logical_reads_total,    logical_rea
ds_delta, buffer_busy_waits_total, buffer_busy_waits_delta,    db_block_changes_total, db_block_changes_delta, physical_rea}

处理思路：由于是MMON在更新统计信息是报错，而且对象是保存在sysaux表空间。所以把改对象WRH$_SEG_STAT删除重建。
参考刘大《Script:诊断SYSAUX表空间使用情况》http://www.oracledatabase12g.com/archives/tag/sysaux

”鉴于该操作过于危险，只适合于深入了解Oracle的人士，所以具体操作步骤移除。“  就没了。
         
请教高手帮忙分析以上分析方法和处理方法是否正确，能否提供参考链接及可操作性步骤。

Liu Maclean(刘相兵

可以尝试truncate SYS                            WRH$_SEG_STAT        这些SYSAUX上的非关键表

newcom

谢谢刘大的恢复！
SQL> show user
USER is "SYS"
SQL> truncate table SYS.WRH$_SEG_STAT;
truncate table SYS.WRH$_SEG_STAT
                   *
ERROR at line 1:
ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0],
[1], [], [], [], [], [], [], [], []

不能直接truncate。能否删除重建呢？

newcom

只有创建catawrtb.sql包
create table WRH$_SEG_STAT
没有找到删除删除包

newcom

在catnoawr.sql  包里找到了。

请问如果运行了catnoawr.sql
再跑catawrtb.sql

是否可以解决问题呢？
如果只有操作将会对系统有什么后果呢？

Liu Maclean(刘相兵

ODM FINDING:

        ORA-00600 [ktecgsc:kcbz_objdchk] on 11.2 (Doc ID 1562473.1)

Oracle Database - Enterprise Edition - Version 11.2.0.2 and later
Information in this document applies to any platform.
Symptoms

    The following error is raised by a W### process:
    *** 2013-05-23 12:20:02.137
    *** SESSION ID:(426.38291) 2013-05-23 12:20:02.137
    *** CLIENT ID:() 2013-05-23 12:20:02.137
    *** SERVICE NAME:(SYS$BACKGROUND) 2013-05-23 12:20:02.137
    *** MODULE NAME:(KTSJ) 2013-05-23 12:20:02.137
    *** ACTION NAME:(KTSJ Slave) 2013-05-23 12:20:02.137

    Dump continued from file: /paic/sx/ics/data/app/oracle/diag/rdbms/ics/ics/trace/ics_w006_25483.trc
    ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0], [1], [], [], [], [], [], [], [], []

    ========= Dump for incident 104147 (ORA 600 [ktecgsc:kcbz_objdchk]) ========
    ----- Beginning of Customized Incident Dump(s) -----
    Input data 0x0, 0, 0
    Formatted dump of block:
    buffer tsn: 4 rdba: 0x01002e4a (4/11850)
    scn: 0x082a.f00dd046 seq: 0x01 flg: 0x04 tail: 0xd0462301
    frmt: 0x02 chkval: 0xcea6 type: 0x23=PAGETABLE SEGMENT HEADER
    Hex dump of block: st=0, typ_found=1


    Review of the generated tracefiles reveals a call stack similar to:
    ... <- _$c1A.kcbz_check_objd_typ <- kcbzib <- kcbgcur <- ktecgsc <- ktecgetsh <- ktecgshx <- ktssupd_segment_extblks_bkg <- ktsj_execute_task <- ktsj_slave_main <- ksvrdp <- opirip <- ...

Cause

The cause of this problem has been identified in:
unpublished Bug:10237773 - 11202-NT_2K8_X64:ORA-00600, ARGUMENTS: [KCBZ_CHECK_OBJD_TYP], [0], [0], [1], [],

It is caused by a process trying to update SEG$ contents with on-disk info where the segment has undergone an object ID change (due to a TRUNCATE occurring).

Bug:10237773 is fixed in RDBMS 12.1.
Solution

To implement a solution for Bug:10237773, please execute any of the below alternative solutions:

    Upgrade to 12.1

    OR

    Apply patchset release 11.2.0.3 in which Bug:10237773 is fixed.

    OR

    Download and apply interim Patch:10237773, if available for your platform and RDBMS release. To check for conflicting patches, please use the MOS Patch Planner Tool. If no patch is available, file a Service Request through My Oracle Support for your specific Oracle version and platform.

Liu Maclean(刘相兵

action plan:


alter system flush buffer_cache;

show parameter db_block

newcom

非常谢谢刘大！
通过metalink查询过，但由于版本不一致。所以不敢贸然使用该补丁包。
SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE    11.2.0.1.0      Production
TNS for Linux: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production

如果是这样的话，是否可以直接升级到11.2.0.3版本呢？