[常规总体性能调优] $ORACLE_HOME/dbs下产生一个20多G的core ora-07445

2^#

发表于 2013-2-6 18:01:49

运行一下

su - oracle

ls -al $ORACLE_HOME/bin/oracle >> /tmp/support.txt
oslevel -s
whoami >> /tmp/support.txt
ulimit -a >> /tmp/support.txt
svmon -O unit=MB >> /tmp/support.txt
/usr/sbin/lsps -a >> /tmp/support.txt
/usr/sbin/lsattr -HE -l sys0 -a realmem >> /tmp/support.txt
ipcs -m >> /tmp/support.txt

cat /tmp/support.txt

ulimit -a

SQL语句：

select name,value from v$system_parameter where ISDEFAULT!='TRUE' order by 1;

3^#

发表于 2013-2-6 18:11:45

oslevel :6100-06-09-1228

4^#

发表于 2013-2-6 19:10:42

opatch lsinventory看一下

5^#

发表于 2013-2-6 22:09:05

Invoking OPatch 10.2.0.5.1

Oracle Interim Patch Installer version 10.2.0.5.1
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

Oracle Home    : /oracle/product/10.2.0/db_1
Central Inventory : /oracle/oraInventory
from          : /etc/oraInst.loc
OPatch version : 10.2.0.5.1
OUI version    : 10.2.0.5.0
OUI location    : /oracle/product/10.2.0/db_1/oui
Log file location : /oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2013-02-06_22-06-59PM.log

Patch history file: /oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /oracle/product/10.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2013-02-06_22-06-59PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (2):

Oracle Database 10g                                                 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 4                         10.2.0.5.0
There are 2 products installed in this Oracle Home.

Interim patches (2) :

Patch  14275629    : applied on Thu Jan 17 10:57:00 CST 2013
Unique Patch ID:  15415102
Created on 4 Oct 2012, 19:33:55 hrs PST8PDT
Bugs fixed:
   8865718, 11790175, 13489660, 9020537, 9772888, 8650138, 8664189, 10091698
   14275629, 10092858, 12551710, 7519406, 13349665, 8771916, 7509714
   8822531, 10139235, 10159846, 13257247, 8350262, 11792865, 7119382
   13632738, 11724962, 8966823, 9320130, 13775862, 11674645, 7026523
   9399589, 9672816, 13503598, 9499302, 9150282, 9448311, 9659614, 13632743
   9949948, 10327179, 8882576, 7612454, 7111619, 9711859, 9714832, 9735237
   9952230, 12780098, 14665116, 8660422, 11066597, 14105702, 9713537
   14105703, 14105704, 13483152, 13737773, 13737775, 14269955, 12925532
   12748240, 9694101, 14390396, 12862186, 12862187, 10249537, 9586877
   8211733, 6694396, 9548269, 7115910, 7710224, 9337325, 8354642, 7602341
   14076510, 10157402, 12565867, 6402302, 10327190, 10269717, 11693109
   10017048, 8546356, 8394351, 9024850, 8224558, 10235783, 9770451, 9360157
   8488233, 9109487, 10132870, 9171933, 10173237, 9532911, 10068982, 10306945
   7361418, 11725006, 8666117, 6157713, 9184754, 10214450, 8544696, 9767674
   9323583, 8277300, 9726739, 13343467, 8412426, 10326338, 10165083
   6651220, 12419392, 10208905, 9145204, 13554409, 11076894, 7450366
   11893577, 8970313, 14492313, 6011045, 14492314, 10162036, 11814891
   14492315, 10248542, 14492316, 9469117, 13359623, 9952270, 13343471
   9842573, 10324526, 12419258, 9322219, 8636407, 10010310, 12828105
   9689310, 9390484, 13736501, 13736502, 9824435, 13736503, 13736504
   13736505, 13736506, 9963497, 9032322, 13736507, 12551700, 12551701
   12551702, 11858315, 12551703, 12551704, 12551705, 10076669, 12551706
   14040433, 12551707, 6076890, 9308296, 12551708, 13916709, 12827745
   12880299, 14038805, 13923855, 8528171, 9072105, 11737047

Patch  9952245    : applied on Thu Jan 17 10:38:10 CST 2013
Unique Patch ID:  13348225
Created on 17 Jan 2011, 04:22:06 hrs UTC
Bugs fixed:
   9952245, 9879181, 9610672, 9872299, 9577921, 8308495, 9103966, 8996060
   6899183, 9261837, 8487143, 9646836

--------------------------------------------------------------------------------

OPatch succeeded.

6^#

发表于 2013-2-6 22:21:09

你已经打了10.2.0.5.9的PSU ；

检查下 IZ56903 这个OS APAR打了吗？

7^#

发表于 2013-2-7 08:50:17

IZ56903 这个OS APAR在系统中没有查到

8^#

发表于 2013-2-18 18:21:28

今天又产生了一个20g的core

9^#

发表于 2013-2-18 19:24:57

trace文件：
Dump file /oracle/admin/pmdb/udump/pmdb1_ora_62522046.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /oracle/product/10.2.0/db_1
System name: AIX
Node name: nxyxdb1
Release: 1
Version: 6
Machine: 00F8099D4C00
Instance name: pmdb1
Redo thread mounted by this instance: 1
Oracle process number: 563
Unix process pid: 62522046, image: oracle@nxyxdb1

Exception signal: 11 (SIGSEGV), code: 50 (Address not mapped to object), addr: 0x6d756c74697a6f6e, PC: [0x9000000000a3858, demote+01f8]
Registers:
iar: 09000000000a3858, msr: a00000000000d032
lr: 09000000000a4f64,  cr: 000000008444244b
r00: 0000000065000000, r01: 0fffffffffffa4e0, r02: 09001000a00827f0,
r03: 0fffffffffffa528, r04: 0000000000000080, r05: 6d756c74697a6f6e,
r06: 0fffffffffffa538, r07: 00000000000000a0, r08: 000000011056bcf2,
r09: 0000000000020000, r10: 000000011056bcea, r11: 000000011056bcea,
r12: 09001000a00a9708, r13: 0000000110287580, r14: 000000000000000e,
r15: 0000000000000000, r16: 0000000000000009, r17: 000000000000000a,
r18: 09001000a00a96f8, r19: 000000011063bfa0, r20: 09001000a00a96d8,
r21: 09001000a0019338, r22: 09001000a00a9718, r23: 0000000000000000,
r24: 00000000000000c0, r25: 000000011063bde0, r26: 000000011063bea0,
r27: 00000000000000c0, r28: 09001000a00a82e0, r29: 09001000a00a82e0,
r30: 0000000000000000, r31: 000000011063bde0,
Exception signal: 11 (SIGSEGV), code: 50 (Address not mapped to object), addr: 0x6d756c74697a6f76, PC: [0x9000000000a412c, leftmost+000c]
Registers:
iar: 09000000000a412c, msr: a00000000000d032
lr: 09000000000a6908,  cr: 0000000044442484
r00: 6500000000020000, r01: 0fffffffffff8b30, r02: 09001000a00827f0,
r03: 00000000000000c0, r04: 09001000a00a82e0, r05: 000000006d756c74,
r06: 00000000697a6f6e, r07: 0000000065000000, r08: 0000000000020000,
r09: 6d756c74697a6f6e, r10: 0fffffffffff8ee0, r11: 09001000a00a96d8,
r12: 09001000a0071f28, r13: 0000000110287580, r14: 0fffffffffff9ff0,
r15: 00000000000000b1, r16: 0000000000000002, r17: 00000001101b9460,
r18: 09001000a00a96f8, r19: 000000011063bfa0, r20: 000000000000000b,
r21: 000000000000000b, r22: 0000000000000007, r23: 0000000000000001,
r24: 000000000000000b, r25: 0000000000000001, r26: 0000000000000000,
r27: 09001000a00a9718, r28: 09001000a00a82e0, r29: 0000000000000000,
r30: 00000000000000c0, r31: 00000000000000a0,
Exception signal: 11 (SIGSEGV), code: 50 (Address not mapped to object), addr: 0x6d756c74697a6f76, PC: [0x9000000000a412c, leftmost+000c]
Registers:
iar: 09000000000a412c, msr: a00000000000d032
lr: 09000000000a6908,  cr: 0000000044442484
r00: 6500000000020000, r01: 0fffffffffff7180, r02: 09001000a00827f0,
r03: 00000000000000c0, r04: 09001000a00a82e0, r05: 000000006d756c74,
r06: 00000000697a6f6e, r07: 0000000065000000, r08: 0000000000020000,
r09: 6d756c74697a6f6e, r10: 0fffffffffff7530, r11: 09001000a00a96d8,
r12: 09001000a0071f28, r13: 0000000110287580, r14: 0fffffffffff8640,
r15: 00000000000000b1, r16: 0000000000000002, r17: 00000001101b9460,
r18: 09001000a00a96f8, r19: 000000011063bfa0, r20: 000000000000000b,
r21: 000000000000000b, r22: 0000000000000007, r23: 0000000000000001,
r24: 000000000000000d, r25: 0000000000000001, r26: 0000000000000000,
r27: 09001000a00a9718, r28: 09001000a00a82e0, r29: 0000000000000000,
r30: 00000000000000c0, r31: 00000000000000a0,
Exception signal: 11 (SIGSEGV), code: 50 (Address not mapped to object), addr: 0x6d756c74697a6f76, PC: [0x9000000000a412c, leftmost+000c]
Registers:
iar: 09000000000a412c, msr: a00000000000d032
lr: 09000000000a6908,  cr: 0000000044442484
r00: 6500000000020000, r01: 0fffffffffff57d0, r02: 09001000a00827f0,
r03: 00000000000000c0, r04: 09001000a00a82e0, r05: 000000006d756c74,
r06: 00000000697a6f6e, r07: 0000000065000000, r08: 0000000000020000,
r09: 6d756c74697a6f6e, r10: 0fffffffffff5b80, r11: 09001000a00a96d8,
r12: 09001000a0071f28, r13: 0000000110287580, r14: 0fffffffffff6c90,
r15: 00000000000000b1, r16: 0000000000000002, r17: 00000001101b9460,
r18: 09001000a00a96f8, r19: 000000011063bfa0, r20: 000000000000000b,
r21: 000000000000000b, r22: 0000000000000007, r23: 0000000000000001,
r24: 000000000000000d, r25: 0000000000000001, r26: 0000000000000000,
r27: 09001000a00a9718, r28: 09001000a00a82e0, r29: 0000000000000000,
r30: 00000000000000c0, r31: 00000000000000a0, ssexhd: crashing the process...
Shadow_Core_Dump = PARTIAL
Exception signal: 11 (SIGSEGV), code: 50 (Address not mapped to object), addr: 0x6d756c74697a6f76, PC: [0x9000000000a3f18, splay+0398]
Registers:
iar: 09000000000a3f18, msr: a00000000000d032
lr: 09000000000a4ad4,  cr: 000000008844222b
r00: 6500000000020000, r01: 0fffffffffff4560, r02: 09001000a00827f0,
r03: 00000001104396a0, r04: 00000001104396e0, r05: 6d756c74697a6f6e,
r06: 00000000697a6f6e, r07: 0000000065000000, r08: 0000000000020000,
r09: 09001000a00a9708, r10: 0000000000000000, r11: 0000000000000000,
r12: 09001000a00a9708, r13: 0000000110287580, r14: 0fffffffffff52e0,
r15: 00000000000000b1, r16: 0000000000000002, r17: 00000001101b9460,
r18: 09001000a00a96f8, r19: 000000011063bfa0, r20: 09001000a00a96d8,
r21: 09001000a0019338, r22: 09001000a00a9718, r23: 0000000000000000,
r24: 0000000000000040, r25: 00000001104396a0, r26: 00000001104396e0,
r27: 0000000000000040, r28: 09001000a00a82e0, r29: 09001000a00a96d8,
r30: 0000000000000000, r31: 09001000a00a82e0, ssexhd: crashing the process...

10^#

发表于 2013-2-19 10:55:49

这次生产20G的core时，alert里面并没有7445的报错
errpt的报错如下：
---------------------------------------------------------------------------
LABEL:       CORE_DUMP
IDENTIFIER:    A924A5FC

Date/Time:    Mon Feb 18 16:26:18 GMT+08:00 2013
Sequence Number: 884
Machine Id:    00F8099D4C00
Node Id:       nxyxdb1
Class:          S
Type:          PERM
WPAR:          Global
Resource Name: SYSPROC

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

      Recommended Actions
      CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

      Recommended Actions
      RERUN THE APPLICATION PROGRAM
      IF PROBLEM PERSISTS THEN DO THE FOLLOWING
      CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
         6
USER'S PROCESS ID:
            62522046
FILE SYSTEM SERIAL NUMBER
      14
INODE NUMBER
         0    251033
CORE FILE NAME
/oracle/product/10.2.0/db_1/dbs/core_62522046/core
PROGRAM NAME
oracle
STACK EXECUTION DISABLED
         0
COME FROM ADDRESS REGISTER
ksdwrf 40

PROCESSOR ID
  hw_fru_id: 2
  hw_cpu_id: 17

ADDITIONAL INFORMATION
slcra 3EC
??
ssexhd 498
??

Symptom Data
REPORTABLE
1
INTERNAL ERROR
0
SYMPTOM CODE
PCSS/SPI2 FLDS/oracle SIG/6 FLDS/slcra VALU/3ec
---------------------------------------------------------------------------
LABEL:       J2_FS_FULL
IDENTIFIER:    F7FA22C9

Date/Time:    Mon Feb 18 16:26:14 GMT+08:00 2013
Sequence Number: 883
Machine Id:    00F8099D4C00
Node Id:       nxyxdb1
Class:          O
Type:          INFO
WPAR:          Global
Resource Name: SYSJ2

Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM

Probable Causes
FILE SYSTEM FULL

      Recommended Actions
      INCREASE THE SIZE OF THE ASSOCIATED FILE SYSTEM
      REMOVE UNNECESSARY DATA FROM FILE SYSTEM
      USE FUSER UTILITY TO LOCATE UNLINKED FILES STILL REFERENCED

Detail Data
JFS2 MAJOR/MINOR DEVICE NUMBER
000A 000F
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/oraclelv, /oracle
---------------------------------------------------------------------------
LABEL:       CORE_DUMP
IDENTIFIER:    A924A5FC

Date/Time:    Sun Feb 17 15:10:50 GMT+08:00 2013
Sequence Number: 882
Machine Id:    00F8099D4C00
Node Id:       nxyxdb1
Class:          S
Type:          PERM
WPAR:          Global
Resource Name: SYSPROC

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

      Recommended Actions
      CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

      Recommended Actions
      RERUN THE APPLICATION PROGRAM
      IF PROBLEM PERSISTS THEN DO THE FOLLOWING
      CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
         6
USER'S PROCESS ID:
            31916290
FILE SYSTEM SERIAL NUMBER
      14
INODE NUMBER
         0    229381
CORE FILE NAME
/oracle/admin/pmdb/cdump/core_31916290/core
PROGRAM NAME
oracle
STACK EXECUTION DISABLED
         0
COME FROM ADDRESS REGISTER
ldata_all 6C

PROCESSOR ID
  hw_fru_id: 1
  hw_cpu_id: 13

ADDITIONAL INFORMATION
slcra 3EC
??

Symptom Data
REPORTABLE
1
INTERNAL ERROR
0
SYMPTOM CODE
PCSS/SPI2 FLDS/oracle SIG/6 FLDS/slcra VALU/3ec

11^#

发表于 2013-2-19 23:18:09

考虑 relink一下 ioracle，并把AIX 补丁打到最新

12^#

发表于 2013-2-20 16:39:05

好的谢谢回复