ORA-07445 core dump [0000F4F4] [SIGSEGV]错误，导致ogg连节oracle中断

clot09

hi 各位,
今天早上ogg报连不上oracle，然后中断了；
数据库报了个ORA-07445: exception encountered: core dump [0000F4F4] [SIGSEGV] [Invalid permissions for mapped object] [0x11028EFFC] [] []；
基本信息如下：

1. $ sqlplus / as sysdba
   
2. 
   
3. SQL*Plus: Release 10.2.0.4.0 - Production on Fri Dec 27 12:41:18 2013
   
4. 
   
5. Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.
   
6. 
   
7. 
   
8. Connected to:
   
9. Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
   
10. With the Partitioning, OLAP, Data Mining and Real Application Testing options
    
11. 
    
12. SQL> select * from v$version;
    
13. 
    
14. BANNER
    
15. ----------------------------------------------------------------
    
16. Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi
    
17. PL/SQL Release 10.2.0.4.0 - Production
    
18. CORE    10.2.0.4.0      Production
    
19. TNS for IBM/AIX RISC System/6000: Version 10.2.0.4.0 - Productio
    
20. NLSRTL Version 10.2.0.4.0 - Production
    
21. 
    
22. 
    
23. 
    
24. $ ggsci
    
25. 
    
26. Oracle GoldenGate Command Interpreter for Oracle
    
27. Version 11.1.1.0.28 Build 001
    
28. AIX 5L, ppc, 64bit (optimized), Oracle 10.2 on Mar 12 2011 19:21:59
    
29. 
    
30. Copyright (C) 1995, 2010, Oracle and/or its affiliates. All rights reserved.
    

复制代码

ogg日志报错如下：

1. 2013-12-27 09:00:12  WARNING OGG-00869  Oracle GoldenGate Delivery for Oracle, reperp.prm:  OCI Error ORA-03113: end-of-file on comm
   
2. unication channel (status = 3113), SQL <UPDATE /*+ RESTRICT_ALL_REF_CONS */ "SAPR3"."ZERMS_1" SET "JSON" = :a5,"METADATA" = :a6,"XML
   
3. STR" = :a7,"ZDATE" = :a8,"EFILE_TITLE" = :a9,"EFILE_TIME" = :a10,"DOC_OPERATION" = :a11,"DOC_CREATOR" = :a12>.
   
4. 2013-12-27 09:00:12  WARNING OGG-01004  Oracle GoldenGate Delivery for Oracle, reperp.prm:  Aborted grouped transaction on 'SAPR3.ZE
   
5. RMS_1', Database error 3114 (ORA-03114: not connected to ORACLE).

复制代码

alert日志如下：

1. Thread 1 advanced to log sequence 183196 (LGWR switch)
   
2.   Current log# 4 seq# 183196 mem# 0: /oracle/PRD/origlogB/log_g14m1.dbf
   
3.   Current log# 4 seq# 183196 mem# 1: /oracle/PRD/mirrlogB/log_g14m2.dbf
   
4. Fri Dec 27 08:54:32 2013
   
5. Completed checkpoint up to RBA [0x2cb9a.2.10], SCN: 1669885094
   
6. Fri Dec 27 08:55:20 2013
   
7. Completed checkpoint up to RBA [0x2cb9b.2.10], SCN: 1669888343
   
8. Fri Dec 27 08:58:05 2013
   
9. Errors in file /oracle/PRD/saptrace/usertrace/prd_ora_360586.trc:
   
10. ORA-07445: exception encountered: core dump [0000F4F4] [SIGSEGV] [Invalid permissions for mapped object] [0x11028EFFC] [] []
    
11. Fri Dec 27 08:58:27 2013
    
12. Completed checkpoint up to RBA [0x2cb9c.2.10], SCN: 1669891863
    
13. Fri Dec 27 09:17:18 2013
    
14. Beginning log switch checkpoint up to RBA [0x2cb9d.2.10], SCN: 1669896877
    
15. Fri Dec 27 09:17:18 2013
    
16. Thread 1 advanced to log sequence 183197 (LGWR switch)
    
17.   Current log# 1 seq# 183197 mem# 0: /oracle/PRD/origlogA/log_g11m1.dbf
    
18.   Current log# 1 seq# 183197 mem# 1: /oracle/PRD/mirrlogA/log_g11m2.dbf

复制代码

附件中为日志和trc文件，请各位帮忙，谢谢！

kevinlin.ora

按call stack搜索，比如：
000044F0             ?        00000000            
upduaw+00a4          bl       updgrh               FFFFFFFFFFEA680 ?
                                                   700000226EEA0C8 ?
kdusru+09f8          bl       01F93874            
kauupd+0230          bl       kdusru               700000210B6FA08 ?
                                                   100400000000 ? 000000000 ?
                                                   000000000 ?
updrow+10fc          bl       kauupd               11050F940 ? 000000000 ?
                                                   110511FA8 ?
                                                   4004828010000FD8 ?
                                                   700000217F99588 ?
                                                   1A000000000000 ?
                                                   48523FFFF6870 ?
                                                   700000226B555E0 ?
qerupRowProcedure+0  bl       updrow               000000000 ? 7FFF00000000 ?
050                                                
qerupFetch+053c      bl       03F24EB4            
updaul+0e1c          bl       01F93874            
updThreePhaseExe+0f  bl       updaul               700000217F9D5E8 ?
10                                                 FFFFFFFFFFF7FA8 ? 000000000 ?
updexe+02f8          bl       updThreePhaseExe     07FFFFFF8 ? 104A27300 ?
                                                   104A272E4 ? 000000000 ?
opiexe+27d8          bl       updexe               110D72750 ? 3FFFF86B8 ?
kpoal8+0edc          bl       opiexe               1100DD628 ? 000000018 ?

clot09

kevinlin.ora 发表于 2013-12-27 14:49
按call stack搜索，比如：
000044F0             ?        00000000            
upduaw+00a4          bl ...

额，水平比较菜，没理解，麻烦说的详细点吧，谢谢了～

kevinlin.ora

MOS里的7445文档工具里，左边是按版本及错误的argument查找，右边有按stack搜索的。

clot09

kevinlin.ora 发表于 2013-12-27 15:03
MOS里的7445文档工具里，左边是按版本及错误的argument查找，右边有按stack搜索的。 ...

根据你提供的方法，试了下，似乎只找到了
Bug 4904890 - Dump (upduaw) from UPDATE (文档 ID 4904890.8)
thanks！

Liu Maclean(刘相兵

Exception signal: 11 (SIGSEGV), code: 51 (Invalid permissions for mapped object), addr: 0x11028effc, PC: [0xf4f4, 0000F4F4]
Registers:

UPDATE /*+ RESTRICT_ALL_REF_CONS */ "SAPR3"."ZERMS_1" SET "JSON" = :a5,"METADATA" = :a6,"XMLSTR" = :a7,"ZDATE" = :a8,"EFILE_TITLE" = :a9,"EFILE_TIME" = :a10,"DOC_OPERATION" = :a11,"DOC_CREATOR" = :a12,"TS_STATUS" = :a13,"TS_TIME" = :a14,"ERMS_EFILE_ID" = :a15,"RET_RESULT_ID" = :a16,"RET_MESSAGE" = :a17,"CXTS_NUM" = :a18,"REMARK" = :a19,"RESERVE1" = :a20,"RESERVE2" = :a21,"RESERVE3" = :a22,"RESERVE4" = :a23,"RESERVE5" = :a24 WHERE "MANDT" = :b0 AND "UNIT_ID" = :b1 AND "SYS_NAME" = :b2 AND "TP_ID" = :b3 AND "EFILE_ID" = :b4


qerupFetch qerupRowProcedure updrow kauupd kdusru upduaw ssexhd

Liu Maclean(刘相兵

ODM FINDING:

Symptoms

You could be getting either of the following errors in the alert.log or all of them while running an update statement:


ORA-07445: exception encountered: core dump [_memmove()+2369] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF200000] [] []

ORA-07445: exception encountered: core dump [kgidum()+1536] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF200000] [] []

ORA-07445: exception encountered: core dump [kgscDump()+1376] [SIGSEGV] [Address not mapped to object] [0xB38F00000000B1] []

ORA-7445: exception encountered: core dump [_memcpy()+2369] [SIGSEGV]
[Invalid permissions for mapped object] [0x9FFFFFFFBF448028] [] []

ORA-07445: exception encountered: core dump[_intel_fast_memcpy.A()+10] [SIGSEGV] [Invalid permissions for mapped object] [0x2A9768C000] [] []


The call stack extracted from the trace file for the above error(s) will look similar to the one below:


updgrh upduaw kdusru kauupd updrow qerupRowProcedure qerupFetch updaul updThreePhaseExe updexe opiexe kpoal8 opiodr ttcpip opitsk opiino opiodr opidrv sou2o opimai_real main main_opd_entry


If you enable the 1551 event tracing, a trace file is generated at the time of the internal error reporting the following:

Errors in file /u10/app/oracle/admin/sid/udump/sid_ora_x.trc:
ORA-01551: extended rollback segment, pinned blocks released
Changes

Upgraded to 9.2.0.8 or 10.2.0.4
Cause

This problem is due to unpublished BUG 5868257 : Dump / memory corruption from UPDATE DML

Details:
There is a problem with the fix from Bug 4549673 which is included in 9.2.0.8 and 10.2.0.4. With that fix applied it is possible to get a dump under updgrh and/or private memory corruption. This issue can also cause malformed RCI information in the redo stream causing problems for log miner and Streams.
eg:
Corruption / dumps can occur when updating a migrated row where one of the columns is to be updated to NULL and an internal 1551 error occurs during the DML.(1551 errors are not visible to customers and are caught and handled internally)

This problem is raised in the following bugs which are closed as Duplicate of

Unpublished BUG 5868257.

BUG 7138239 - ORA-07445[_INTEL_FAST_MEMCPY.A()+10] FOR UPDATE
BUG 7118779 - VARIOUS ORA-600 / ORA-7445 ERRORS FROM REMOTE APPLY SIDE
BUG 7136414 - ORA-07445 [_MEMMOVE()+2464] ERROR RUNNING UPDATE
BUG 7423332 - MEMORY CORRUPTIONS WITH VARIOUS ORA-600 AND ORA-7445'S UNTIL PMON FAILURE
Bug 7390551 ORA-7445 EXCEPTION ENCOUNTERED: CORE DUMP [FFFFFFFF7F40079C]

Solution

1. Unpublished Bug 5868257 is fixed in patchsets 10.2.0.5 and 11.1.0.6 and higher. Therefore upgrade to one of the Oracle Versions where the bug has been fixed.

OR

2. Apply the one-off Patch 5868257 if available on My Oracle Support for your Oracle Version and platform.

If your database version is 10.2.0.4 on HP-UX Itanium, then make sure you apply the patch which is dated 23-OCT-2008 (or later) because previous versions of this patch on this platform & DB version showed some problems as seen in Bug 7423332.


OR

3. As a workaround, you can help to avoid getting migrated rows by ensuring that PCTFREE is set sufficiently high such that subsequent UPDATE of rows in a data block has space to fit. It seems likely that you could minimize the risk of this issue by rebuilding the main affected table with a sufficient PCTFREE to avoid row migration.
References

Liu Maclean(刘相兵

. Apply the one-off Patch 5868257 if available on My Oracle Support for your Oracle Version and platform.

clot09

Liu Maclean(刘相兵 发表于 2013-12-27 17:35
Exception signal: 11 (SIGSEGV), code: 51 (Invalid permissions for mapped object), addr: 0x11028effc, ...

刘大，请问这是使用到的函数吧？那根据Bug 4904890 - Dump (upduaw) from UPDATE (文档 ID 4904890.8)。能推断这是bug引起的吧？

clot09

Liu Maclean(刘相兵 发表于 2013-12-27 17:39
. Apply the one-off Patch 5868257 if available on My Oracle Support for your Oracle Version and plat ...

好的，谢谢刘大