ricky

158 积分	1 好友	8 主题

发消息

[常规总体性能调优] ORA-600 [kclexpandlock_1]

1^#

发表于 2013-2-6 16:11:11 | 查看: 5318| 回复: 8

本帖最后由 ricky 于 2013-2-6 16:14 编辑

环境：HP_UX
数据库：rac
version :10.2.0.4.12

SQL>  select version from v$instance;

VERSION
-----------------
10.2.0.4.0

打的补丁：./opatch lsinventory

Patch  12879933

Patch  9352164
Patch  7493592

问题描述：
节点一ods1 正常。
节点二ods2会重启；

节点一alert：
Wed Feb  6 12:07:30 2013
Thread 1 advanced to log sequence 23923 (LGWR switch)
  Current log# 1 seq# 23923 mem# 0: /dev/vg001/rlv_redo_11_4g_001
  Current log# 1 seq# 23923 mem# 1: /dev/vg002/rlv_redo_11_4g_002
Wed Feb  6 14:48:23 2013
Trace dumping is performing id=[cdmp_20130206144823]
Wed Feb  6 14:48:31 2013
Reconfiguration started (old inc 8, new inc 10)
List of nodes:
0
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Wed Feb  6 14:48:31 2013
LMS 7: 0 GCS shadows cancelled, 0 closed
Wed Feb  6 14:48:31 2013
LMS 5: 0 GCS shadows cancelled, 0 closed
Wed Feb  6 14:48:31 2013
LMS 6: 0 GCS shadows cancelled, 0 closed
Wed Feb  6 14:48:31 2013
LMS 1: 0 GCS shadows cancelled, 0 closed
Wed Feb  6 14:48:31 2013
LMS 3: 0 GCS shadows cancelled, 0 closed
Wed Feb  6 14:48:31 2013
LMS 2: 0 GCS shadows cancelled, 0 closed
Wed Feb  6 14:48:31 2013
LMS 0: 21 GCS shadows cancelled, 0 closed
Wed Feb  6 14:48:31 2013
LMS 4: 2 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Wed Feb  6 14:48:35 2013
Instance recovery: looking for dead threads
Wed Feb  6 14:48:35 2013
Beginning instance recovery of 1 threads
Wed Feb  6 14:48:37 2013
LMS 6: 399997 GCS shadows traversed, 0 replayed
Wed Feb  6 14:48:37 2013
LMS 5: 400619 GCS shadows traversed, 0 replayed
Wed Feb  6 14:48:37 2013
LMS 4: 417095 GCS shadows traversed, 0 replayed
Wed Feb  6 14:48:37 2013
LMS 7: 398648 GCS shadows traversed, 0 replayed
Wed Feb  6 14:48:37 2013
LMS 3: 404854 GCS shadows traversed, 0 replayed
Wed Feb  6 14:48:37 2013
.........
.........
Wed Feb  6 14:50:28 2013
LMS 3: 4970 GCS shadows traversed, 4001 replayed
LMS 3: 4961 GCS shadows traversed, 4001 replayed
Wed Feb  6 14:50:28 2013
LMS 1: 4997 GCS shadows traversed, 4001 replayed
LMS 1: 4986 GCS shadows traversed, 4001 replayed
LMS 1: 5010 GCS shadows traversed, 4001 replayed
LMS 1: 4944 GCS shadows traversed, 4001 replayed
Wed Feb  6 14:50:28 2013
LMS 3: 2479 GCS shadows traversed, 1965 replayed
Wed Feb  6 14:50:28 2013
LMS 1: 4937 GCS shadows traversed, 4001 replayed
LMS 1: 5034 GCS shadows traversed, 4001 replayed
LMS 1: 4940 GCS shadows traversed, 4001 replayed
LMS 1: 3047 GCS shadows traversed, 2420 replayed

节点二alert：

Wed Feb  6 14:44:57 2013
ORA-01555 caused by SQL statement below (SQL ID: b5m84q8mrdppa, Query Duration=12212 sec, SCN: 0x0bfe.d71b9954):
Wed Feb  6 14:44:57 2013
SELECT
  SERV_ID,'20130206112633' ETL_TIME
FROM ODS.SERV a
WHERE RELA_ADDR IS NULL
--and rownum<10000
Wed Feb  6 14:48:21 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lms0_18714.trc:
ORA-00600: internal error code, arguments: [kclexpandlock_1], [101954], [], [], [], [], [], []
Wed Feb  6 14:48:23 2013
Trace dumping is performing id=[cdmp_20130206144823]
Wed Feb  6 14:48:23 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lms0_18714.trc:
ORA-00600: internal error code, arguments: [kclexpandlock_1], [101954], [], [], [], [], [], []
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lms0_18714.trc:
ORA-00600: internal error code, arguments: [kclexpandlock_1], [101954], [], [], [], [], [], []
Wed Feb  6 14:48:27 2013
LMS0: terminating instance due to error 484
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lms3_18720.trc:
ORA-00484: LMS* process terminated with error
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lms4_18727.trc:
ORA-00484: LMS* process terminated with error
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lms5_18729.trc:
ORA-00484: LMS* process terminated with error
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lms2_18718.trc:
ORA-00484: LMS* process terminated with error
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lmon_18706.trc:
ORA-00484: LMS* process terminated with error
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_pmon_17899.trc:
ORA-00484: LMS* process terminated with error
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lmd0_18712.trc:
ORA-00484: LMS* process terminated with error
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lms1_18716.trc:
ORA-00484: LMS* process terminated with error
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lms7_18733.trc:
ORA-00484: LMS* process terminated with error
Wed Feb  6 14:48:27 2013
Errors in file /oracle/product/oracle10g/admin/odsdb/bdump/odsdb2_lms6_18731.trc:
ORA-00484: LMS* process terminated with error
Wed Feb  6 14:48:28 2013
Shutting down instance (abort)
License high water mark = 471
Wed Feb  6 14:48:32 2013
Instance terminated by LMS0, pid = 18714
Wed Feb  6 14:48:33 2013
Instance terminated by USER, pid = 19733
Wed Feb  6 14:48:54 2013
Starting ORACLE instance (normal)

查询metlink发现是bug：
Bug 7680907  ORA-600 [kclexpandlock_2] in LMS / instance crash. Incorrect locks in RAC. ORA-600 [3020] in recovery

In a RAC environment non contiguous multiblock reads with prefetching
across multiple files may not get the correct locks leading to
internal errors such as ORA-600 [kclexpandlock_2].

问题：是不是只有升级才能解决该问题？
      为什么只有节点二回出现呢，相同的版本，节点一怎么不会出现呢？
      有没有什么避免的方法？
  trace文件稍后上传

分享0

收藏0 回复只看该作者道具举报

ricky

2^#

发表于 2013-2-6 16:18:58

这些是trace文件。谢谢

odsdb2_lms0_18714.txt

7.02 MB, 下载次数: 715

odsdb2_lms3_18720.txt

144.51 KB, 下载次数: 686

odsdb2_lms4_18727.txt

82.1 KB, 下载次数: 711

回复只看该作者道具举报

Maclean Liu(刘相兵

3^#

发表于 2013-2-6 16:25:23

你不能压缩下trace文件吗？你上传的不辛苦吗？

回复只看该作者道具举报

ricky

4^#

发表于 2013-2-6 16:28:30

Maclean Liu(刘相兵发表于 2013-2-6 16:25
你不能压缩下trace文件吗？你上传的不辛苦吗？

哦不好意思~~网速还行，不太辛苦。下次我会注意!trace文件太多，我就上传这3个典型的了。/tp

回复只看该作者道具举报

Maclean Liu(刘相兵

5^#

发表于 2013-2-6 16:52:44

kjmpmsgi fatal error on 32
MSG [32:KJX_B_OCR] [0x12905.17d0000] inc=8 len=296 sender=(0,0) seq=69701158
   fg=s stat=KJUSERSTAT_DONE spid=28312 flg=x4
flow ctrl: ver=1 flag=33 len=344 tkts=0 seq=69701158 wrp=0
            sndr=0 dest=1 rcvr=1
FUSION MSG 9fffffffbd694280,32 from[0,28312] ver[8,4418] ln 296 sq[0,0]
      OCR [0x12905.17d0000,101954] lockp[c000000a0bfd6a70,517] shadow[0000000000000000,0]
      reqid 436 seq 0x1 fl 0xf md 1 infosz 136 opt 0x0
      seq 0x1 change-scn scn: 0x0000.00000000 xid 0x0000.000.00000000
      msgRM# 4418 bkt# 58656 drmbkt# 58656
   pkey 101954, stat 0, masters[32767, 1->1], reminc 8, RM# 4079 flg 0x0
   hv 82 [stat 0x0, 1->1, wm 32767, RMno 0, reminc 8, dom 0]
   kjga st 0x4, step 0.0.0, cinc 8, rmno 4418, flags 0x0
   lb 0, hb 0, myb 58656, drmb 58656, apifrz 0

查下 data_object_id=101954的对象

select object_name,object_type from dba_objects where data_object_id=101954;

回复只看该作者道具举报

ricky

6^#

发表于 2013-2-6 17:07:34

Maclean Liu(刘相兵发表于 2013-2-6 16:52
kjmpmsgi fatal error on 32
MSG [32:KJX_B_OCR] [0x12905.17d0000] inc=8 len=296 sender=(0,0) seq=69701 ...

SQL> select object_name,object_type from dba_objects where data_object_id=101954;

no rows selected

SQL>

回复只看该作者道具举报

Maclean Liu(刘相兵

7^#

发表于 2013-2-6 17:12:17

就trace看当时lms在等ges remote message ，相关的信息指向 pkey 101954

是否有该对象被drop了的可能性？

为什么只有2节点发生该问题==》可能涉及到的gcs resource master在1节点上导致2节点访问时发生该问题

==》

如何解决？

1. 打补丁 Unpublished bug 5843468 is fixed in 10.2.0.5 and 11g.

2. 找出相关对象想办法workaround

回复只看该作者道具举报

ricky

8^#

发表于 2013-2-6 17:26:20

因为是生产系统，所以暂时不能停机打补丁。请问有什么好的方法可以找到这个对象吗？

回复只看该作者道具举报

ricky

9^#

发表于 2013-2-6 17:27:47

这个问题在4号也出现过一次。之前一直没出现过。

回复只看该作者道具举报

返回列表

		自动登录	找回密码
密码			注册