如何分析trace文件

2^#

发表于 2012-3-1 17:06:25

trace2

tracefile2

trace在一楼

回忆未来

3^#

发表于 2012-3-1 17:27:49

解压出来竟然将近1.8G....

[ 本帖最后由回忆未来于 2012-3-1 17:31 编辑 ]

未命名.jpg (79.12 KB, 下载次数: 516)

未命名.jpg

4^#

发表于 2012-3-1 17:59:38

最后100行？，怎么分析？

5^#

发表于 2012-3-1 19:24:21

这个trace 文件过于庞大了，学习如何阅读trace 从较小的size 开始。

6^#

发表于 2012-3-1 19:54:22

ODM Finding:

/oracle/app1/oracle/admin/crmint/udump/crmint1_ora_6455750.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/app1/oracle/product/10.2.0/
System name: AIX
Node name: crm_int_db1
Release: 3
Version: 5
Machine: 00C6E6644C00
Instance name: crmint1
Redo thread mounted by this instance: 1
Oracle process number: 203
Unix process pid: 6455750, image: oracle@crm_int_db1
*** ACTION NAME:() 2012-02-24 23:30:18.472
*** MODULE NAME:(NewBillTfjClientA@crmapp11 (TNS V1-V3)) 2012-02-24 23:30:18.472
*** SERVICE NAME:(crmint) 2012-02-24 23:30:18.472
*** SESSION ID:(2950.56667) 2012-02-24 23:30:18.472
*** 2012-02-24 23:30:18.472
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [25027], [7], [4063256], [], [], [], [], []
Current SQL statement for this session:
INSERT INTO TFJ_WORK_ORDER (ORDER_SERIAL_NBR, PROD_SPEC_ID, SERV_ID, ACC_NBR, ACTION, ORDER_STATE, SOURCE, CREATE_DATE, STATE, STATE_DATE, LATN_ID, BUREAU_ID, IF_QUOTA
, PRIORITY, STAFF_ID, COMMENTS, AREA_ID, MDSE_SPEC_ID, SWD_ID, INSERT_DATE, EXCH_ID, HLR_NBR) SELECT A.ORDER_SERIAL_NBR, A.PROD_SPEC_ID, A.SERV_ID, DECODE(B.ACCOUNT_FL
AG, 'N', A.ACC_NBR, 'Y', A.SERVICE_CODE), A.ACTION, A.ORDER_STATE, SOURCE, A.CREATE_DATE, '5SN', A.STATE_DATE, A.LATN_ID, A.BUREAU_ID, A.IF_QUOTA, A.PRIORITY, A.STAFF_
ID, A.COMMENTS, A.AREA_ID, A.MDSE_SPEC_ID, A.SWD_ID, SYSDATE, A.EXCH_ID, A.HLR_NBR FROM WORK_ORDER A, TFJ_PROD_SPEC_CONFIG B WHERE A.ORDER_SERIAL_NBR = :B1 AND A.PROD_
SPEC_ID = B.PROD_SPEC_ID
----- PL/SQL Call Stack -----
object line object
handle number name
7000004d9490230 218 procedure BASEJK.NEWBILLTFJSEND
70000060d968880 1 anonymous block
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst+001c bl ksedst1 088404844 ? 041104844 ?
ksedmp+0290 bl ksedst 10453B3A8 ?
ksfdmp+0018 bl 03F3AB94
kgeriv+0108 bl _ptrgl
kgesiv+0080 bl kgeriv 000000000 ? 000000002 ?
000000000 ? 000000000 ?
000000000 ?
ksesic2+0060 bl kgesiv 000000002 ? 11016E478 ?
FFFFFFFFFFDE8F0 ? 1051018FC ?
000024000 ?
krtd2abh+040c bl ksesic2 61C3000061C3 ? 000000000 ?
000000007 ? 000000000 ?
0003E0018 ? 000000320 ?
105101038 ? 000355D00 ?
kcbget+1a84 bl krtd2abh 1002A062C ? 10510156C ?
0FFFFF7FF ? 1101FBD60 ?
105101578 ?
ktbxchg+0110 bl kcbget 7000005E74A45F0 ? 000000000 ?
FFFFFFFFFFE4660 ? 11019A468 ?
kdifind+0f50 bl ktbxchg 7000005E4A81590 ? 000000000 ?
7000005E52CA508 ? 000000000 ?
kdiins0+0cc4 bl kdifind 389CB4C00000000 ?
700000398AB08B0 ?
FFFFFFFFFFE4660 ?
FFFFFFFFFFDFD90 ?
FFFFFFFFFFDF8E0 ? 000000000 ?
000000000 ? 000000000 ?
kdiinsp+0070 bl kdiins0 700000398AB08A8 ? 104BBCFB0 ?

复制代码

10.2.0.4 on AIX 5.3

ORA-00600: internal error code, arguments: [25027], [7], [4063256], [], [], [], [], []

stack call kauxsin -> kdiinsp -> kdiins0 -> kdifind -> ktbxchg -> kcbget -> krtd2abh-> ksesic2-> kgesiv

ORA-600 [26027] Note:

ORA-600 [25027] [ID 284433.1]
PURPOSE:
This article represents a partially published OERI note.
It has been published because the ORA-600 error has been
reported in at least one confirmed bug.
Therefore, the SUGGESTIONS section of this article may help
in terms of identifying the cause of the error.
This specific ORA-600 error may be considered for full publication
at a later date. If/when fully published, additional information
will be available here on the nature of this error.
ERROR:
Format: ORA-600 [25027] [a] [b]
VERSIONS:
versions 9.2 and above
ARGUMENTS:
Arg [a] Tablespace Number (TSN)
Arg [b] Decimal Relative Data Block Address (RDBA)
SUGGESTIONS:
1. If the Arg [b] (the RDBA) is 0 (zero), then this could be due to fake indexes.
The following query will list fake indexes:
select do.owner,do.object_name, do.object_type,sysind.flags
from dba_objects do, sys.ind$ sysind
where do.object_id = sysind.obj#
and bitand(sysind.flags,4096)=4096;
If the above query returns any rows, check the objects involved and consider
dropping them as they can cause this error.
2. Run analyze table validate structure on the table referenced in the Current SQL statement in
the related trace file.
If the Known Issues section below does not help in terms of identifying
a solution, please submit the trace files and alert.log to Oracle
Support Services for further analysis.

复制代码

7^#

发表于 2012-3-1 20:26:18

ORA-00600: internal error code, arguments: [25027], [7], [4063256], [], [], [], [], []

Arg a  Tablespace Number (TSN)                            => 7
Arg b  Decimal Relative Data Block Address (RDBA)          => 4063256

GLOBAL CACHE ELEMENT DUMP (address: 700000441fd1cd0):
  id1: 0x74c51 id2: 0x300000 obj: 265103 block: (48/478289)

  LIST OF BUFFERS LINKED TO THIS GLOBAL CACHE ELEMENT:
flg: 0x02002001 state: XCURRENT mode: EXCL
   pin: 'kdiwh22: kdifind'
   addr: 700000211efe018 obj: 265103 cls: DATA bscn: 0xb87.8e06d715
      buffer tsn: 7 rdba: 0x0c074c51 (48/478289)
      scn: 0x0b87.8e06d715 seq: 0x01 flg: 0x00 tail: 0xd7150601
      frmt: 0x02 chkval: 0x0000 type: 0x06=trans data

Block header dump:  0x0c074c51
Object id on Block? Y
seg/obj: 0x40b8f  csc: 0xb87.8e06d715  itc: 0  flg: O  typ: 2 - INDEX
   brn: 8  bdba: 0xc074a8a ver: 0x01 opc: 112
   inc: 0  exflg: 0

Itl          Xid                Uba       Flag  Lck       Scn/Fsc

Branch block dump

Branch block dump
=================
header address 504403167140642868=0x700000211000034
kdxcolev 2
KDXCOLEV Flags = - - -
kdxcolok 0
kdxcoopc 0x0: opcode=0: iot flags=--- is converted=-
kdxconco 251
kdxcosdc 1312293376
kdxconro -32768
kdxcofbo 2902=0xb56
kdxcofeo -28623=0xffff9031
kdxcoavs 1472
kdxbrlmc 4063256=0x3e0018
kdxbrsno 94
.....................
row#1015[16] dba: 4063256=0x3e0018

row#27[16384] **** invalid row offset ****          => 说明偏移量不正确，逻辑讹误
row#28[8032] dba: 16777921=0x10002c1       =》偏移量不正确会导致row header的dba 不正确

kdxbrlmc 4063256=0x3e0018

kdxbrlmc 是小于本branch block 中row# 0的值的 branch block的 rdba address

kdxcolev: index level (0 represents leaf blocks)
kdxcolok: denotes whether structural block transaction is occurring
kdxcoopc: internal operation code
kdxconco: index column count
kdxcosdc: count of index structural changes involving block
kdxconro: number of index entries (does not include kdxbrlmc pointer)
kdxcofbo: offset to beginning of free space within block
kdxcofeo: offset to the end of free space (ie. first portion of block containing index data)
kdxcoavs: available space in block (effectively area between the two fields above)
kdxbrlmc: block address if index value is less than the first (row#0) value
kdxbrsno: last index entry to be modified
kdxbrbksz: size of usable block space

以上dump了 file# 48  block# 478289，该块属于一个索引对象号obj_number  为 0x40b8f = 265103

该块可能存在逻辑讹误(logical corruption) ，提供了错误的偏移量，这导致 branch block中记录的DBA存在错误，从而引发ORA-600[25027]错误：
Ora-600 [25027] While Inserting Into a Table [ID 560103.1]

Applies to:

Oracle Server - Enterprise Edition - Version: 10.1.0.2 to 10.2.0.4 - Release: 10.1 to 10.2

Information in this document applies to any platform.

Checked for relevance on 22-Feb-2012

Symptoms

While inserting into a table ORA-00600 [25027][177][0] is encountered.

Cause

One of the table indexes is corrupted.

The following is found in the ORA-600 [25027] trace file.

Current SQL statement for this session:

INSERT

INTO

dtktgt.loc_meas_dly_pull

.....

Block header dump: 0x0b823083

Object id on Block? Y

seg/obj: 0x4c1cec csc: 0x08.20c997dd itc: 0 flg: O typ: 2 - INDEX

brn: 0 bdba: 0x0 ver: 0x01 opc: 80

inc: 0 exflg: 0

Itl Xid Uba Flag Lck Scn/Fsc

Branch block dump

=================

header address 17620123700=0x41a3dc034

kdxcolev 9

KDXCOLEV Flags = R T -

kdxcolok 0

kdxcoopc 0x0: opcode=0: iot flags=--- is converted=-

kdxconco 224

kdxcosdc 4022075648

kdxconro -16384

kdxcofbo 8=0x8

kdxcofeo 8393=0x20c9

kdxcoavs 20868

kdxbrlmc 0=0x0

kdxbrsno 0

-16384 exceeds Max number of rows

row#0[0] **** invalid row offset ****

row#1[0] **** invalid row offset ****

The last three lines should not be there in a valid block and show that the contents of the block have been corrupted.

Solution

1) SQL> Analyze table <name> validate structure cascade ;

2) query dba_indexes to get all indexes for failed table 'table_name'

SQL> select index_name from user_indexes where table_name = 'TABLE_NAME';

3) validate table indexes :

SQL > Analyze index <index name > validate structure ;

If the analyze indicates corruption and that corruption is within an index, drop and recreate the index.
复制代码
建议你使用 Analyze index <index name > validate structure ;分析 bj_number  为 0x40b8f = 265103的索引，验证索引结构，  若确认存在讹误，则重建索引( drop 后再build , rebuild 可能无法cover 逻辑讹误)。

8^#

发表于 2012-3-1 23:19:16

谢谢刘老大，非常专业，事实证明在这里提问比MOS更加快，回答更加专业。我还有一个问题，刚开始我也是差MOS得到DOC ORA-600 [25027] [doc ID 284433.1]，说ORA-600 [25027] [a] 中的是RDBA，然后就用dbms_utility去convert，但是结果得到file_id=0，这明显不符合，就疑惑了，难道doc ID 284433.1 有错？
-------
PURPOSE:
  This article represents a partially published OERI note.

  It has been published because the ORA-600 error has been
  reported in at least one confirmed bug.

  Therefore, the SUGGESTIONS section of this article may help
  in terms of identifying the cause of the error.

  This specific ORA-600 error may be considered for full publication
  at a later date. If/when fully published, additional information
  will be available here on the nature of this error.

ERROR:

  Format: ORA-600 [25027] [a]

VERSIONS:
  versions 9.2 and above

ARGUMENTS:
  Arg [a]  Tablespace Number (TSN)
  Arg   Decimal Relative Data Block Address (RDBA)

9^#

发表于 2012-3-2 13:12:32

“DOC ORA-600 [25027] [doc ID 284433.1]，说ORA-600 [25027] [a] 中的是RDBA，然后就用dbms_utility去convert，但是结果得到file_id=0，这明显不符合，就疑惑了，难道doc ID 284433.1 有错？”

  Arg  b Decimal Relative Data Block Address (RDBA) 4063256 =>0x3E0018

Oracle中最小的RDBA是 1号文件上的数据块

buffer tsn: 0 rdba: 0x00400009 (1/9)       =》　1号文件第9个块为 0x400009

1号文件第一个块为0x400001

0x400000  》　0x3E0018，　　显然说明了0x3E0018　这个RDBA是讹误后的结果。

aaaaaaaa2000

10^#

发表于 2012-3-2 14:15:45

刘总

stack call kauxsin -> kdiinsp -> kdiins0 -> kdifind -> ktbxchg -> kcbget -> krtd2abh-> ksesic2-> kgesiv
这块是从那得到的？这些都是干啥啊

11^#

发表于 2012-3-2 14:33:59

回复 10# 的帖子

一般600 trace的开头部分就会有stack call trace，如他的trace里：

----- Call Stack Trace -----
calling             call    entry             argument values in hex
location          type    point             (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst+001c       bl    ksedst1             088404844 ? 041104844 ?
ksedmp+0290       bl    ksedst             10453B3A8 ?
ksfdmp+0018       bl    03F3AB94
kgeriv+0108       bl    _ptrgl
kgesiv+0080       bl    kgeriv             000000000 ? 000000002 ?
                                                000000000 ? 000000000 ?
                                                000000000 ?
ksesic2+0060       bl    kgesiv             000000002 ? 11016E478 ?
                                                FFFFFFFFFFDE8F0 ? 1051018FC ?
                                                000024000 ?
krtd2abh+040c       bl    ksesic2             61C3000061C3 ? 000000000 ?
                                                000000007 ? 000000000 ?
                                                0003E0018 ? 000000320 ?
                                                105101038 ? 000355D00 ?
kcbget+1a84       bl    krtd2abh          1002A062C ? 10510156C ?
                                                0FFFFF7FF ? 1101FBD60 ?
                                                105101578 ?
ktbxchg+0110       bl    kcbget             7000005E74A45F0 ? 000000000 ?
                                                FFFFFFFFFFE4660 ? 11019A468 ?
kdifind+0f50       bl    ktbxchg             7000005E4A81590 ? 000000000 ?

注意看每一个function 的顺序，是从下到上的，最上部的function kge部分是无意义的，他们是 oracle 错误机制的代码。

可以参考ORA-00600: http://www.oracledatabase12g.com/archives/ora-00600.html

关于Unix 上的stack call你可以去读一下 <unix环境高级编程>这本书

aaaaaaaa2000

12^#

发表于 2012-3-2 14:34:55

还有
block# 478289-这个块号是根据4063256的出来的吗？

row#28[8032] dba: 16777921=0x10002c1 =》偏移量不正确会导致row header的dba 不正确
kdxbrlmc 4063256=0x3e0018
kdxbrlmc 是小于本branch block 中row# 0的值的 branch block的 rdba address-----这一段还不是很明白
望刘总解答，多谢

13^#

发表于 2012-3-2 14:37:46

回复 12# 的帖子

kdxbrlmc: block address if index value is less than the first (row#0) value 你可以理解为上一个branch block的 block address

14^#

发表于 2012-3-3 23:40:57

谢谢刘老大

ricky

15^#

发表于 2012-7-4 15:30:59

哇咔咔~~终于在这里找到了，还让我纠结了很久~~这个经典的帖子我的好好嚼嚼~~~~

ricky

16^#

发表于 2012-7-4 15:32:51

回复 11# 的帖子

关于Unix 上的stack call你可以去读一下 <unix环境高级编程>这本书~~
这个trace文件里面的调用就是os层面的知识是吧？？刘大能不能就这个<unix环境高级编程>简单讲解一二~~

chen28

17^#

发表于 2012-7-27 14:46:48

回复 11# 的帖子

hi
Ml
我按照你的
http://www.oracledatabase12g.com/archives/ora-00600.html
方法去(和你帖子一样的stack call)
metalink
搜索
对应的call stack 没有没有记录呢?

18^#

发表于 2012-7-27 21:54:04

回复 17# 的帖子

你选择的搜索范围是什么？

chen28

19^#

发表于 2012-7-30 09:21:31

选择的范围是这样的
ksedst()+40
ksedmp()+168
ksfdmp()+32
kgerinv()+152
kgeasnmierr()+88
kcbassertbd3()+204
kcbz_check_objd_typ
kcbzib()+
kcbgtcr()+
ktecgsc()+168
ktecgetsh()+196
ktecgshx()+40
kteinicnt1()+648
ktssdrbm_segment()+
ktssdro_segment()+3
ktssdt_segs()+1128
ktmmon()+3500
ktmSmonMain()+64
ksbrdp()+1276
opirip()+
opidrv()+1088
sou2o()+120
opimai_real()+496
main()+240
$START$()+

20^#

发表于 2012-7-30 10:17:31

我在这篇文章里说明了 http://www.oracledatabase12g.com/archives/ora-00600.html

"ksedst()+40
ksedmp()+168
ksfdmp()+32
kgerinv()+152
kgeasnmierr()+88
kcbassertbd3()+204
kcbz_check_objd_typ
kcbzib()+
kcbgtcr()+
ktecgsc()+168
ktecgetsh()+196
ktecgshx()+40
kteinicnt1()+648
ktssdrbm_segment()+
ktssdro_segment()+3
ktssdt_segs()+1128
ktmmon()+3500
ktmSmonMain()+64
ksbrdp()+1276
opirip()+
opidrv()+1088
sou2o()+120
opimai_real()+496
main()+240
$START$()+

注意以上stack call中只有 ktmSmonMain -> kcbassertbd3 这部分是有意义的，开始部分的main()-> ksbrdp() 是很普通的入口函数，而从kgeasnmierr (Kernel generic Error ) 开始的代码是Oracle 报错层使用的函数，都是对定位问题没有帮助的。将这部分有用的stack call 填入Metalink <ORA-600/ORA-7445 Error Look-up Tool [ID 153788.1]> 600问题诊断页面的 stack call 栏会以较严格的筛选条件找出问题相关的Note:"

你上面的stack call 中只有 ksbrdp => kcbassertbd3 这部分是有意义的

最下面 main() 到opirip 是程序的入口，每个 stack call几乎有这样

kgeasnmierr=> 是负责报错的函数

chen28

21^#

发表于 2012-7-30 10:50:22

不好意思 ML
trace stack copy错了
如下图
请帮忙谢谢

查询的.jpg (83.6 KB, 下载次数: 506)

查询的.jpg

结果.jpg (31.48 KB, 下载次数: 492)

结果.jpg