- 最后登录
- 2015-7-23
- 在线时间
- 39 小时
- 威望
- 158
- 金钱
- 677
- 注册时间
- 2012-4-1
- 阅读权限
- 50
- 帖子
- 115
- 精华
- 0
- 积分
- 158
- UID
- 324
|
1#
发表于 2012-4-12 11:03:32
|
查看: 7843 |
回复: 7
环境:9i RAC
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /u01/app/oracle/product/9.2.0
System name: AIX
Node name: kmolcom02
Release: 3
Version: 5
Machine: 000C1042D600
Instance name: kmsop2
描述:今早来查看alert日志,发现一号节点出现很多cdump
节点一:
Thu Apr 12 05:25:53 2012
Trace dumping is performing id=[cdmp_20120412052553]
Thu Apr 12 05:26:55 2012
Trace dumping is performing id=[cdmp_20120412052654]
Thu Apr 12 05:27:55 2012
Trace dumping is performing id=[cdmp_20120412052755]
Thu Apr 12 05:28:39 2012
Trace dumping is performing id=[cdmp_20120412052839]
Thu Apr 12 05:29:56 2012
Trace dumping is performing id=[cdmp_20120412052956]
Thu Apr 12 05:30:39 2012
Trace dumping is performing id=[cdmp_20120412053039]
Thu Apr 12 05:31:58 2012
Trace dumping is performing id=[cdmp_20120412053158]
Thu Apr 12 05:33:58 2012
Trace dumping is performing id=[cdmp_20120412053358]
Thu Apr 12 05:35:58 2012
Trace dumping is performing id=[cdmp_20120412053558]
Thu Apr 12 05:37:58 2012
Trace dumping is performing id=[cdmp_20120412053758]
Thu Apr 12 05:38:41 2012
Trace dumping is performing id=[cdmp_20120412053841]
Thu Apr 12 05:40:20 2012
Trace dumping is performing id=[cdmp_20120412054020]
Thu Apr 12 05:41:58 2012
Trace dumping is performing id=[cdmp_20120412054158]
Thu Apr 12 05:43:01 2012
Trace dumping is performing id=[cdmp_20120412054301]
Thu Apr 12 05:43:59 2012
Trace dumping is performing id=[cdmp_20120412054358]
Thu Apr 12 09:08:58 2012
--5点25分开始,5点43分结束,所幸,没有发生宕机。
查看节点2:
Thu Apr 12 05:25:52 2012
Errors in file /u01/app/oracle/admin/sopdb/udump/kmsop2_ora_876696.trc:
ORA-27506: IPC error connecting to a port
ORA-27300: OS system dependent operation:connect failed with status: 4
ORA-27301: OS failure message: Interrupted system call
ORA-27302: failure occurred at: skgxpdoaconr
ORA-27303: additional information: remote process is out of memory
Thu Apr 12 05:25:52 2012
Errors in file /u01/app/oracle/admin/sopdb/udump/kmsop2_ora_876696.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
ORA-27300: OS system dependent operation:connect failed with status: 4
ORA-27301: OS failure message: Interrupted system call
ORA-27302: failure occurred at: skgxpdoaconr
ORA-27303: additional information: remote process is out of memory
Thu Apr 12 05:25:53 2012
Trace dumping is performing id=[cdmp_20120412052553]
。。。
以下一直重复和上面一样的错误~~
---查看第一个报错信息:
Thu Apr 12 05:25:52 2012
Errors in file /u01/app/oracle/admin/sopdb/udump/kmsop2_ora_876696.trc:
ORA-27506: IPC error connecting to a port
ORA-27300: OS system dependent operation:connect failed with status: 4
ORA-27301: OS failure message: Interrupted system call
ORA-27302: failure occurred at: skgxpdoaconr
ORA-27303: additional information: remote process is out of memory
Thu Apr 12 05:25:52 2012
Errors in file /u01/app/oracle/admin/sopdb/udump/kmsop2_ora_876696.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
ORA-27300: OS system dependent operation:connect failed with status: 4
ORA-27301: OS failure message: Interrupted system call
ORA-27302: failure occurred at: skgxpdoaconr
ORA-27303: additional information: remote process is out of memory
Thu Apr 12 05:25:53 2012
Trace dumping is performing id=[cdmp_20120412052553]---和节点一的cdump一样的时间点,一样的序号!
-------/u01/app/oracle/admin/sopdb/udump/kmsop2_ora_876696.trc
kmolcom02:/home/oracle#more /u01/app/oracle/admin/sopdb/udump/kmsop2_ora_876696.trc
/u01/app/oracle/admin/sopdb/udump/kmsop2_ora_876696.trc
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /u01/app/oracle/product/9.2.0
System name: AIX
Node name: kmolcom02
Release: 3
Version: 5
Machine: 000C1042D600
Instance name: kmsop2
Redo thread mounted by this instance: 2
Oracle process number: 192
Unix process pid: 876696, image: oracle@kmolcom02 (TNS V1-V3)
*** 2012-04-10 18:07:07.925
*** SESSION ID:(141.4742) 2012-04-10 18:07:07.903
skgxpdocon: warning outstanding accept handle count has reached new high water mark 1000
*** 2012-04-10 21:21:53.288
skgxpdocon: warning outstanding accept handle count has reached new high water mark 2000
*** 2012-04-11 00:36:35.674
skgxpdocon: warning outstanding accept handle count has reached new high water mark 3000
*** 2012-04-11 03:51:39.216
skgxpdocon: warning outstanding accept handle count has reached new high water mark 4000
*** 2012-04-11 07:07:28.269
skgxpdocon: warning outstanding accept handle count has reached new high water mark 5000
*** 2012-04-11 10:22:30.875
skgxpdocon: warning outstanding accept handle count has reached new high water mark 6000
*** 2012-04-11 13:37:37.553
skgxpdocon: warning outstanding accept handle count has reached new high water mark 7000
*** 2012-04-11 16:52:21.419
skgxpdocon: warning outstanding accept handle count has reached new high water mark 8000
*** 2012-04-11 20:07:40.614
skgxpdocon: warning outstanding accept handle count has reached new high water mark 9000
*** 2012-04-11 23:23:34.957
skgxpdocon: warning outstanding accept handle count has reached new high water mark 10000
*** 2012-04-12 02:40:37.181
skgxpdocon: warning outstanding accept handle count has reached new high water mark 11000
ORA-27506: IPC error connecting to a port
ORA-27300: OS system dependent operation:connect failed with status: 4
ORA-27301: OS failure message: Interrupted system call
ORA-27302: failure occurred at: skgxpdoaconr
ORA-27303: additional information: remote process is out of memory
*** 2012-04-12 05:25:52.239
SKGXPCNH: 0x1035e8b0 SKGXPCON_CONN_SENT (3) sconno 459243771 accono 0 admno 1616406595
Remote admin port
SSKGXPT 0x1035e8d4 flags SSKGXPT_WRITE active network 0
info for network 0
socket no 8 IP 66.66.66.1 UDP 61718
HACMP network_id 0 sflags SSKGXPT_UP
Remote data port
SSKGXPT 0x1035e970 flags active network 0
info for network 0
socket no 0 IP 0.0.0.0 UDP 0
HACMP network_id 0 sflags
ERROR connect requestion should be on done q
next seqno 32763 credits 8 ertt 64 resends on con 0
*** 2012-04-12 05:25:52.239
---trace文件里面找到相关sql
*** 2012-04-12 05:25:52.275
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [] [] [] [] [] []
ORA-27300: OS system dependent operation:connect failed with status: 4
ORA-27301: OS failure message: Interrupted system call
ORA-27302: failure occurred at: skgxpdoaconr
ORA-27303: additional information: remote process is out of memory
Current SQL statement for this session:
select 'RACMembership :' || inst_name ||
':' || inst_number dbMembers
from sys.v_$active_instances a,gv$instance b
where a.inst_number=b.instance_number
----- Call Stack Trace -----
返回:查看1节点有trace文件,在dump目录下面cdump_***目录
主要看看这些trace文件内容
文件比较多,差不多查看了一遍。里面的内容都大部分相似:
2551691C:009A4DFE 250 0 10429 6 Free MB: buf 7000000c9c66ab0 (SO queue 0) pool 70000000000e6b0, size 488
2551691E:009A4DFF 250 0 10429 10 IPC SODC: Exiting cleanup for 7000000c8883e48, rc: 1
25516920:009A4E00 250 0 10427 8 Disconn : Disconnect from inst 1, receiver 0
25516921:009A4E01 250 0 10401 2 KSXPMSGCNCL: client 2 tid inst 2 ptid 1 flags 0x500
25516924:009A4E02 250 0 10401 46 KSXPMSGCNCL: could not map tid(2, 1, 0xfe0e1cda) to cnh
25516925:009A4E03 250 0 10427 8 Disconn : Disconnect from inst 1, receiver 1
25516926:009A4E04 250 0 10401 2 KSXPMSGCNCL: client 2 tid inst 2 ptid 2 flags 0x500
2551692A:009A4E05 250 0 10427 8 Disconn : Disconnect from inst 1, receiver 2
2551692B:009A4E06 250 0 10401 2 KSXPMSGCNCL: client 2 tid inst 2 ptid 3 flags 0x500
C864682E:009A9BBD 250 0 10280 1 0x00000000000000FA
C86468D9:009A9BBE 250 0 10401 29 KSXPUNMAP: client 1
C86468DB:009A9BBF 250 0 10401 28 KSXPMAP: client 1 base 0x7000000000b7000 size 0xf4f49000
C86A729A:009A9BC7 250 357 10429 7 MB SO Al: Allocated MBSO 7000000c8873f48
C86A729F:009A9BC8 250 357 10427 7 Connect : Connect to inst 1, receiver 0
C86A72A0:009A9BC9 250 357 10427 7 Connect : Connect to inst 1, receiver 1
C86A72A2:009A9BCA 250 357 10427 7 Connect : Connect to inst 1, receiver 2
------------------------------------------------------------------------------------------------------------------------
网络资源:
http://www.itpub.net/thread-918844-1-1.html
http://www.itpub.net/thread-766945-1-1.html |
|