Oracle数据库数据恢复、性能优化

找回密码
注册
搜索
热搜: 活动 交友 discuz
Oracle数据库数据恢复、性能优化»论坛 › Oracle › Oracle数据库管理 › Rac 节点down 掉
发新帖
yobyin

125

积分

0

好友

4

主题

[SQL调优] Rac 节点down 掉

1#
发表于 2012-12-7 09:43:52 | 查看: 7527| 回复: 6
4节点rac
os:suse 10 sp2
dbms:11.1.0.7

现停服务是在2012-11-08 18:00 --- 19:15 左右第一个节点down 掉
alter 日志
IPC Send timeout detected.Sender: ospid 11279
Receiver: inst 2 binc 1718214658 ospid 29484
Thu Nov 08 18:26:38 2012
Errors in file /opt/oracle/diag/rdbms/mos5200/mos52001/trace/mos52001_dia0_11294.trc:
ORA-27508: IPC error sending a message
Thu Nov 08 18:29:39 2012
ERROR: LMS0 (ospid: 11309) detects an idle connection to instance 2
Thu Nov 08 18:29:39 2012
Communications reconfiguration: instance_number 2
Thu Nov 08 18:31:19 2012
Evicting instance 2 from cluster
Waiting for instances to leave:

crs 日志
2012-11-08 17:57:57.554: [  CRSRES][1510197568] startRunnable: setting CLI values
2012-11-08 17:57:57.683: [    CRSD][1510197568] SM:dE2Ec: all E2E cmds done. 0
2012-11-08 18:31:38.878: [  CRSAPP][1510197568] CheckResource error for ora.mos5200.rac_load.cs error code = 1
2012-11-08 18:31:38.882: [  CRSRES][1510197568] In stateChanged, ora.mos5200.rac_load.cs target is ONLINE
2012-11-08 18:31:38.882: [  CRSRES][1510197568] ora.mos5200.rac_load.cs on bndmos52dbsv01 went OFFLINE unexpectedly
2012-11-08 18:31:38.882: [  CRSRES][1510197568] StopResource: setting CLI values
2012-11-08 18:31:38.912: [  CRSRES][1510197568] Attempting to stop `ora.mos5200.rac_load.cs` on member `bndmos52dbsv01`
2012-11-08 18:31:39.027: [  CRSRES][1510197568] Stop of `ora.mos5200.rac_load.cs` on member `bndmos52dbsv01` succeeded.
2012-11-08 18:31:39.027: [  CRSRES][1510197568] ora.mos5200.rac_load.cs RESTART_COUNT=0 RESTART_ATTEMPTS=0
2012-11-08 18:31:39.040: [  CRSRES][1510197568] ora.mos5200.rac_load.cs exceeded it's failure threshold.  Stopping it and its dependents!
2012-11-08 18:31:39.044: [  CRSRES][1510197568] Resource recovery not purged:ora.mos5200.rac_load.cs
2012-11-08 18:31:39.060: [  CRSRES][1510197568] ora.mos5200.rac_load.cs target set to OFFLINE before stop action
2012-11-08 18:31:39.060: [  CRSRES][1510197568] StopResource: setting CLI values
2012-11-08 18:31:39.067: [  CRSRES][1510197568] Target set to OFFLINE for `ora.mos5200.rac_load.cs`
2012-11-08 19:14:49.888: [    CRSD][1411549504] SM: rE2Ec: 4
2012-11-08 19:14:49.906: [  CRSRES][1510197568] startRunnable: setting CLI values

racle@BNDMOS52DBSV01:/opt/oracrs/log/bndmos52dbsv01/crsd> crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    bndm...sv01
ora....01.lsnr application    ONLINE    OFFLINE               
ora....v01.gsd application    ONLINE    ONLINE    bndm...sv01
ora....v01.ons application    ONLINE    ONLINE    bndm...sv01
ora....v01.vip application    ONLINE    ONLINE    bndm...sv02
ora....SM2.asm application    ONLINE    ONLINE    bndm...sv02
ora....02.lsnr application    ONLINE    ONLINE    bndm...sv02
ora....v02.gsd application    ONLINE    ONLINE    bndm...sv02
ora....v02.ons application    ONLINE    ONLINE    bndm...sv02
ora....v02.vip application    ONLINE    ONLINE    bndm...sv02
ora....SM3.asm application    ONLINE    ONLINE    bndm...sv03
ora....03.lsnr application    ONLINE    ONLINE    bndm...sv03
ora....v03.gsd application    ONLINE    ONLINE    bndm...sv03
ora....v03.ons application    ONLINE    ONLINE    bndm...sv03
ora....v03.vip application    ONLINE    ONLINE    bndm...sv03
ora....SM4.asm application    ONLINE    ONLINE    bndm...sv04
ora....04.lsnr application    ONLINE    ONLINE    bndm...sv04
ora....v04.gsd application    ONLINE    ONLINE    bndm...sv04
ora....v04.ons application    ONLINE    ONLINE    bndm...sv04
ora....v04.vip application    ONLINE    ONLINE    bndm...sv04
ora.mos5200.db application    ONLINE    ONLINE    bndm...sv01
ora....01.inst application    ONLINE    ONLINE    bndm...sv01
ora....02.inst application    ONLINE    ONLINE    bndm...sv02
ora....03.inst application    ONLINE    ONLINE    bndm...sv03
ora....04.inst application    ONLINE    ONLINE    bndm...sv04
ora....awei.cs application    ONLINE    ONLINE    bndm...sv04
ora....004.srv application    ONLINE    ONLINE    bndm...sv04
ora....load.cs application    ONLINE    ONLINE    bndm...sv01
ora....001.srv application    ONLINE    OFFLINE               
ora....002.srv application    ONLINE    ONLINE    bndm...sv02
ora...._web.cs application    ONLINE    ONLINE    bndm...sv03
ora....003.srv application    ONLINE    ONLINE    bndm...sv03



个人分析像是监测不到heartbeat,被清出cluster.还请分析

在metalink 查到类同的bug 只在windows 平台出现,并没有在linux 平台出现

Bug 6782276  Win: ORA-27508 from RAC IPC
This note gives a brief overview of bug 6782276.
The content was last updated on: 14-OCT-2011
Click here for details of each of the sections below.
Affects:
Product (Component)        Oracle Server (Rdbms)
Range of versions believed to be affected        Versions BELOW 11.2
Versions confirmed as being affected        •        11.1.0.6
•        10.2.0.4

Platforms affected        •        Windows/NT/XP
Fixed:
This issue is fixed in        •        11.2.0.1 (Base Release)
•        11.1.0.7 (Server Patch Set)
•        10.2.0.5 (Server Patch Set)
•        10.2.0.4 Patch 9 on Windows Platforms

Symptoms:        Related To:               
•        ORA-27508 / ORA-27300         •        RAC (Real Application Clusters) / OPS
       
                               
Description
RAC instance on Windows may get failures such as:
ORA-27508: IPC error sending a message
ORA-27300: OS system dependent operation:IPCSOCK_Send failed with status: 10055
ORA-27301: OS failure message: An operation on a socket could not be performed ...
ORA-27302: failure occurred at: send_3

Please note: The above is a summary description only. Actual symptoms can vary. Matching to any symptoms here does not confirm that you are encountering this problem. For questions about this bug please consult Oracle Support.
References


附件为相关日志

RAC1 LOG.rar

165.99 KB, 下载次数: 913
7#
发表于 2012-12-12 20:00:32
谢谢。下次一定注意用格式。
结贴。

回复 只看该作者 道具 举报

6#
发表于 2012-12-12 14:51:20
贴日志请用code 模式！！！！！
  1. ov  8 18:26:34 BNDMOS52DBSV01 ifprobe:     bond0     changed config file: ifcfg-bond0 --> restart interface!
  2. Nov  8 18:26:34 BNDMOS52DBSV01 ifdown:     bond0     is still used from interfaces   eth0 eth1
  3. Nov  8 18:26:34 BNDMOS52DBSV01 ifdown:     eth0      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
  4. Nov  8 18:26:34 BNDMOS52DBSV01 ifdown:     eth0      configuration: eth-id-34:40:b5:b0:c1:3c
  5. Nov  8 18:26:34 BNDMOS52DBSV01 kernel: bonding: bond0: Removing slave eth0
  6. Nov  8 18:26:34 BNDMOS52DBSV01 kernel: bonding: bond0: Warning: the permanent HWaddr of eth0 - 34:40:b5:b0:c1:3c - is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts.
  7. Nov  8 18:26:34 BNDMOS52DBSV01 kernel: bonding: bond0: releasing backup interface eth0
  8. Nov  8 18:26:34 BNDMOS52DBSV01 ifdown:     eth1      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
  9. Nov  8 18:26:34 BNDMOS52DBSV01 ifdown:     eth1      configuration: eth-id-34:40:b5:b0:c1:3c
  10. Nov  8 18:26:35 BNDMOS52DBSV01 kernel: bonding: bond0: Removing slave eth1
  11. Nov  8 18:26:35 BNDMOS52DBSV01 kernel: bonding: bond0: releasing active interface eth1
  12. Nov  8 18:26:35 BNDMOS52DBSV01 ifdown:     bond0     now going down itself
  13. Nov  8 18:26:35 BNDMOS52DBSV01 ifdown:     bond0     
  14. Nov  8 18:26:35 BNDMOS52DBSV01 kernel: bonding: bond0 is being deleted...
  15. Nov  8 18:26:36 BNDMOS52DBSV01 ifprobe:     eth0      still using old config:  --> restart interface!
  16. Nov  8 18:26:36 BNDMOS52DBSV01 ifdown:     eth0      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
  17. Nov  8 18:26:36 BNDMOS52DBSV01 ifdown:     eth0      configuration: eth-id-34:40:b5:b0:c1:3c
  18. Nov  8 18:26:37 BNDMOS52DBSV01 ifprobe:     eth1      still using old config:  --> restart interface!
  19. Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth1      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
  20. Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth1      configuration: eth-id-34:40:b5:b0:c1:3e
  21. Nov  8 18:26:37 BNDMOS52DBSV01 ifprobe:     eth2      changed config file: ifcfg-eth-id-90:e2:ba:0d:ca:d2 --> restart interface!
  22. Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth2      device: Intel Corporation Device 1516 (rev 01)
  23. Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth2      configuration: eth-id-90:e2:ba:0d:ca:d2
  24. Nov  8 18:26:37 BNDMOS52DBSV01 ifprobe:     eth3      changed config file: ifcfg-eth-id-90:e2:ba:0d:ca:d3 --> restart interface!
  25. Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth3      device: Intel Corporation Device 1516 (rev 01)
  26. Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth3      configuration: eth-id-90:e2:ba:0d:ca:d3
  27. Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth0      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
  28. Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth0      configuration: eth-id-34:40:b5:b0:c1:3c
  29. Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth0      Startmode is 'off'
  30. Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth1      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
  31. Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth1      configuration: eth-id-34:40:b5:b0:c1:3e
  32. Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth1      Startmode is 'off'
  33. Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth2      device: Intel Corporation Device 1516 (rev 01)
  34. Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth2      configuration: eth-id-90:e2:ba:0d:ca:d2
  35. Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth2      
复制代码
就日志来看 eth0 eth1 bond0都有down过

回复 只看该作者 道具 举报

5#
发表于 2012-12-12 09:51:46
截取了相应时点内的os log
Nov  8 18:20:01 BNDMOS52DBSV01 /usr/sbin/cron[31494]: (root) CMD (/usr/sbin/ntpdate 10.24.134.234)
Nov  8 18:26:32 BNDMOS52DBSV01 kernel: bonding: bond0: link status definitely down for interface eth0, disabling it
Nov  8 18:26:32 BNDMOS52DBSV01 kernel: bonding: bond0: making interface eth1 the new active one.
Nov  8 18:26:34 BNDMOS52DBSV01 ifprobe:     bond0     changed config file: ifcfg-bond0 --> restart interface!
Nov  8 18:26:34 BNDMOS52DBSV01 ifdown:     bond0     is still used from interfaces   eth0 eth1
Nov  8 18:26:34 BNDMOS52DBSV01 ifdown:     eth0      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Nov  8 18:26:34 BNDMOS52DBSV01 ifdown:     eth0      configuration: eth-id-34:40:b5:b0:c1:3c
Nov  8 18:26:34 BNDMOS52DBSV01 kernel: bonding: bond0: Removing slave eth0
Nov  8 18:26:34 BNDMOS52DBSV01 kernel: bonding: bond0: Warning: the permanent HWaddr of eth0 - 34:40:b5:b0:c1:3c - is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts.
Nov  8 18:26:34 BNDMOS52DBSV01 kernel: bonding: bond0: releasing backup interface eth0
Nov  8 18:26:34 BNDMOS52DBSV01 ifdown:     eth1      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Nov  8 18:26:34 BNDMOS52DBSV01 ifdown:     eth1      configuration: eth-id-34:40:b5:b0:c1:3c
Nov  8 18:26:35 BNDMOS52DBSV01 kernel: bonding: bond0: Removing slave eth1
Nov  8 18:26:35 BNDMOS52DBSV01 kernel: bonding: bond0: releasing active interface eth1
Nov  8 18:26:35 BNDMOS52DBSV01 ifdown:     bond0     now going down itself
Nov  8 18:26:35 BNDMOS52DBSV01 ifdown:     bond0     
Nov  8 18:26:35 BNDMOS52DBSV01 kernel: bonding: bond0 is being deleted...
Nov  8 18:26:36 BNDMOS52DBSV01 ifprobe:     eth0      still using old config:  --> restart interface!
Nov  8 18:26:36 BNDMOS52DBSV01 ifdown:     eth0      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Nov  8 18:26:36 BNDMOS52DBSV01 ifdown:     eth0      configuration: eth-id-34:40:b5:b0:c1:3c
Nov  8 18:26:37 BNDMOS52DBSV01 ifprobe:     eth1      still using old config:  --> restart interface!
Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth1      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth1      configuration: eth-id-34:40:b5:b0:c1:3e
Nov  8 18:26:37 BNDMOS52DBSV01 ifprobe:     eth2      changed config file: ifcfg-eth-id-90:e2:ba:0d:ca:d2 --> restart interface!
Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth2      device: Intel Corporation Device 1516 (rev 01)
Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth2      configuration: eth-id-90:e2:ba:0d:ca:d2
Nov  8 18:26:37 BNDMOS52DBSV01 ifprobe:     eth3      changed config file: ifcfg-eth-id-90:e2:ba:0d:ca:d3 --> restart interface!
Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth3      device: Intel Corporation Device 1516 (rev 01)
Nov  8 18:26:37 BNDMOS52DBSV01 ifdown:     eth3      configuration: eth-id-90:e2:ba:0d:ca:d3
Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth0      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth0      configuration: eth-id-34:40:b5:b0:c1:3c
Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth0      Startmode is 'off'
Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth1      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth1      configuration: eth-id-34:40:b5:b0:c1:3e
Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth1      Startmode is 'off'
Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth2      device: Intel Corporation Device 1516 (rev 01)
Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth2      configuration: eth-id-90:e2:ba:0d:ca:d2
Nov  8 18:26:38 BNDMOS52DBSV01 ifup:     eth2      
Nov  8 18:26:38 BNDMOS52DBSV01 ifup: IP address: 10.10.10.11/24  
Nov  8 18:26:38 BNDMOS52DBSV01 ifup:  
Nov  8 18:26:38 BNDMOS52DBSV01 ifup-route: Warning: Could not set up default route via interface
Nov  8 18:26:38 BNDMOS52DBSV01 ifup-route:  Command ip route replace to default via 172.30.134.1 returned:
Nov  8 18:26:38 BNDMOS52DBSV01 ifup-route:  . RTNETLINK answers: Network is unreachable
Nov  8 18:26:38 BNDMOS52DBSV01 ifup-route:  Configuration line: default 172.30.134.1 - -
Nov  8 18:26:38 BNDMOS52DBSV01 ifup-route:  This needs NOT to be AN ERROR if you set up multiple interfaces.
Nov  8 18:26:38 BNDMOS52DBSV01 ifup-route:  See man 5 routes how to avoid this warning.
Nov  8 18:26:38 BNDMOS52DBSV01 ifup-route:
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4152]: ntpd 4.2.4p3@1.1502-o Sat Sep  5 11:15:41 UTC 2009 (1)
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4153]: precision = 1.000 usec
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4153]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4153]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4153]: Listening on interface #1 lo, 127.0.0.1#123 Enabled
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4153]: Listening on interface #2 lo, 127.0.0.2#123 Enabled
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4153]: Listening on interface #3 eth2, 10.10.10.11#123 Enabled
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4153]: kernel time sync status 0040
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4153]: frequency initialized 0.000 PPM from /var/lib/ntp/drift/ntp.drift
Nov  8 18:26:39 BNDMOS52DBSV01 ifup:     eth3      device: Intel Corporation Device 1516 (rev 01)
Nov  8 18:26:39 BNDMOS52DBSV01 ifup:     eth3      configuration: eth-id-90:e2:ba:0d:ca:d3
Nov  8 18:26:39 BNDMOS52DBSV01 ifup:     eth3      
Nov  8 18:26:39 BNDMOS52DBSV01 ifup: IP address: 192.168.1.31/24  
Nov  8 18:26:39 BNDMOS52DBSV01 ifup:  
Nov  8 18:26:39 BNDMOS52DBSV01 ifup-route: Warning: Could not set up default route via interface
Nov  8 18:26:39 BNDMOS52DBSV01 ifup-route:  Command ip route replace to default via 172.30.134.1 returned:
Nov  8 18:26:39 BNDMOS52DBSV01 ifup-route:  . RTNETLINK answers: Network is unreachable
Nov  8 18:26:39 BNDMOS52DBSV01 ifup-route:  Configuration line: default 172.30.134.1 - -
Nov  8 18:26:39 BNDMOS52DBSV01 ifup-route:  This needs NOT to be AN ERROR if you set up multiple interfaces.
Nov  8 18:26:39 BNDMOS52DBSV01 ifup-route:  See man 5 routes how to avoid this warning.
Nov  8 18:26:39 BNDMOS52DBSV01 ifup-route:
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4286]: ntpd 4.2.4p3@1.1502-o Sat Sep  5 11:15:41 UTC 2009 (1)
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4287]: precision = 1.000 usec
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4287]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4287]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4287]: Listening on interface #1 lo, 127.0.0.1#123 Enabled
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4287]: Listening on interface #2 lo, 127.0.0.2#123 Enabled
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4287]: Listening on interface #3 eth2, 10.10.10.11#123 Enabled
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4287]: Listening on interface #4 eth3, 192.168.1.31#123 Enabled
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4287]: kernel time sync status 0040
Nov  8 18:26:39 BNDMOS52DBSV01 ntpd[4287]: frequency initialized 0.000 PPM from /var/lib/ntp/drift/ntp.drift
Nov  8 18:26:39 BNDMOS52DBSV01 ifup:     bond0     
Nov  8 18:26:39 BNDMOS52DBSV01 kernel: bonding: bond0 is being created...
Nov  8 18:26:40 BNDMOS52DBSV01 kernel: bonding: bond0: setting mode to active-backup (1).
Nov  8 18:26:40 BNDMOS52DBSV01 kernel: bonding: bond0: Setting MII monitoring interval to 100.
Nov  8 18:26:40 BNDMOS52DBSV01 ifup:     bond0     enslaving interfaces:  eth0 eth1
Nov  8 18:26:41 BNDMOS52DBSV01 kernel: bnx2: eth0: using MSI
Nov  8 18:26:41 BNDMOS52DBSV01 kernel: bonding: bond0: enslaving eth0 as a backup interface with a down link.
Nov  8 18:26:41 BNDMOS52DBSV01 kernel: bnx2: eth1: using MSI
Nov  8 18:26:41 BNDMOS52DBSV01 kernel: bonding: bond0: enslaving eth1 as a backup interface with a down link.
Nov  8 18:26:41 BNDMOS52DBSV01 kernel: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Nov  8 18:26:41 BNDMOS52DBSV01 kernel: igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Nov  8 18:26:42 BNDMOS52DBSV01 ifup:     bond0     
Nov  8 18:26:42 BNDMOS52DBSV01 ifup: IP address: 172.30.134.31/24  
Nov  8 18:26:42 BNDMOS52DBSV01 ifup:  as bonding master
Nov  8 18:26:42 BNDMOS52DBSV01 ifup:  
Nov  8 18:26:42 BNDMOS52DBSV01 ntpd[4460]: ntpd 4.2.4p3@1.1502-o Sat Sep  5 11:15:41 UTC 2009 (1)
Nov  8 18:26:42 BNDMOS52DBSV01 ntpd[4461]: precision = 1.000 usec
Nov  8 18:26:42 BNDMOS52DBSV01 ntpd[4461]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Nov  8 18:26:42 BNDMOS52DBSV01 ntpd[4461]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
Nov  8 18:26:42 BNDMOS52DBSV01 ntpd[4461]: Listening on interface #1 lo, 127.0.0.1#123 Enabled
Nov  8 18:26:42 BNDMOS52DBSV01 ntpd[4461]: Listening on interface #2 lo, 127.0.0.2#123 Enabled
Nov  8 18:26:42 BNDMOS52DBSV01 ntpd[4461]: Listening on interface #3 eth2, 10.10.10.11#123 Enabled
Nov  8 18:26:42 BNDMOS52DBSV01 ntpd[4461]: Listening on interface #4 eth3, 192.168.1.31#123 Enabled
Nov  8 18:26:42 BNDMOS52DBSV01 ntpd[4461]: Listening on interface #5 bond0, 172.30.134.31#123

回复 只看该作者 道具 举报

4#
发表于 2012-12-7 21:34:39
11.1.0.7.0  +　LINUX + RAC
  1. Receiver: inst 3 binc 6 ospid 28385
  2. Errors in file /opt/oracle/diag/rdbms/mos5200/mos52001/trace/mos52001_lck0_11515.trc:
  3. ORA-27508: IPC error sending a message
  4. IPC Send timeout detected.Sender: ospid 11515
  5. Receiver: inst 4 binc 4 ospid 22262
  6. Errors in file /opt/oracle/diag/rdbms/mos5200/mos52001/trace/mos52001_lck0_11515.trc:
  7. ORA-27508: IPC error sending a message
  8. Thu Nov 08 19:16:59 2012
  9. Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
  10. Thu Nov 08 20:02:07 2012


  13. *** 2012-11-08 19:14:42.423
  14. ksxpsrvdt: Completed KST Dump on Send timeout

  16. *** 2012-11-08 19:14:42.424
  17. IPC Send timeout detected.Sender: ospid 11515
  18. Receiver: inst 4 binc 4 ospid 22262
  19. SKGXP: SKGXPCTX: 0x0x2adcaa242bc0 ctx
  20. SKGXP:
  21. SKGXP: WAIT HISTORY
  22. SKGXP: Time(msec)         Wait Type         Return Code
  23. SKGXP: ----------         ---------         ------------
  24. SKGXP: 1232                 NORMAL                 SUCC
  25. SKGXP: 1248                 NORMAL                 SUCC
  26. SKGXP: 0                 NORMAL                 TIMEDOUT
  27. SKGXP: 126                 NORMAL                 TIMEDOUT
  28. SKGXP: 24                 NORMAL                 TIMEDOUT
  29. SKGXP: 0                 NORMAL                 TIMEDOUT
  30. SKGXP: 0                 NORMAL                 TIMEDOUT
  31. SKGXP: 1551                 NORMAL                 SUCC
  32. SKGXP: 1236                 NORMAL                 SUCC
  33. SKGXP: 0                 NORMAL                 TIMEDOUT
  34. SKGXP: 158                 NORMAL                 TIMEDOUT
  35. SKGXP: 0                 NORMAL                 TIMEDOUT
  36. SKGXP: 24                 NORMAL                 TIMEDOUT
  37. SKGXP: 0                 NORMAL                 SUCC
  38. SKGXP: 0                 NORMAL                 POSTED
  39. SKGXP: 0                 NORMAL                 TIMEDOUT
  40. SKGXP: wait delta 0 sec (0 msec) ctx ts 0x4996772 last ts 0x4996772
  41. SKGXP: user cpu time since last wait 0 sec 0 ticks
  42. SKGXP: system cpu time since last wait 0 sec 0 ticks
  43. SKGXP: locked 1
  44. SKGXP: blocked 15403
  45. SKGXP: timed wait receives 5
  46. SKGXP: admno 0x1f4ae8e4 admport:
  47. SKGXP: SSKGXPT 0xaa243b04 flags  sockno 10 IP 10.10.10.11 UDP 64353
  48. SKGXP: context timestamp 0x4996772
  49. SKGXP: buffers queued on port 0x2adcaa47e078
  50. SKGXP:
  51. SKGXP:    Dumping Connection Handle Table
  52. SKGXP:     sconno     accono   ertt  state   seq#   RcvPid   TotCreditsSKGXP:   sent rtrans   acks
  53. SKGXP:     CNH Table Bucket: 1
  54. SKGXP: 0x678fde3f 0x51741532 47124381171728 47124381171716 47124381206655 47124381191647 47124381171720SKGXP:   2180d      0d  34943d
  55. SKGXP:     CNH Table Bucket: 2
  56. SKGXP: 0x678fde40 0x56cb6acd     16 47124381171716 47124381204901 47124381191870 47124381171720SKGXP:    426d      0d  33189d
  57. SKGXP:     CNH Table Bucket: 94
  58. SKGXP: 0x678fde37 0x6d2fb523     16 47124381171716 47124381188850 47124381193763 47124381171720SKGXP: 115445d      0d  17138d
  59. SKGXP:     CNH Table Bucket: 96
  60. SKGXP: 0x678fde39 0x6ff9bf96     16 47124381171716 47124381199111 47124381199886 47124381171720SKGXP: 125706d      0d  27399d
  61. SKGXP:
  62. SKGXP:    Dumping Accept Handle Table
  63. SKGXP:        ach     accono     sconno      admno  state     SndPid   seq#    rcv rtrans   acks    credits
  64. SKGXP:     ACH Table Bucket: 1566
  65. SKGXP: 0x2adcaa259968 0x4cbc48a7 0x2adc6fe134aa 0x2adc5efd8d43 47124381171752  22262 47124381210173  71252     20  70775      8
  66. SKGXP:     ACH Table Bucket: 1568
  67. SKGXP: 0x2adcaa25d080 0x4cbc48a9 0x2adc74de1fee 0x2adc4d77ce00 47124381171752  28385 47124381210020  71090     12  70596      8
  68. SKGXP:     ACH Table Bucket: 1570
  69. SKGXP: 0x2adcaa260798 0x4cbc48ab 0x2adc374edd8a 0x2adc4b3d800f 47124381171752  29726 47124381207760  68837     18  68369      8
  70. SKGXP:     ACH Table Bucket: 1572
  71. SKGXP: 0x2adcaa263eb0 0x4cbc48ad 0x2adc1c0c66f1 0x2adc5ac9e465 47124381171752   9933 47124381207128   2653      0   2638      8
  72. SKGXP:     ACH Table Bucket: 1574
  73. SKGXP: 0x2adcaa2675c8 0x4cbc48af 0x2adc2606dba5 0x2adc3a19c1b6 47124381171752  20158 47124381205730   1255      0   1255      8
  74. Dumping KSXP Connection Handle: cnh: 0x2adcaa7c8f28
  75. Link in Component Context : 0x2adcaa8083e8 <- cnh -> 0x2adcaa471020
  76. Link in Connection Cache  : 0x2adcaa7c8a10 <- cnh -> 0x2adcaa7c6fc0
  77. Pending Async Sends       : First: 0x2adcaa7552d0 Last 0x2adcaa740e30
  78. Name: LCK0
  79. State: 4 ==> CLOSEWAIT
  80. Flags: 0x1  ==> KSXPCNH_NAME_CACHED |
  81. Connect request handle: (nil)
  82. Miners Canary         : 0x2adcaa7552d0
  83. OSD Connection handle : 0x2adcaa7d5a40
  84. Stats pointer         : 0x2bffe0c10
  85. System incarnation when connection was made: 4
  86. Dumping Target id for cnh
  87. vers: 1 type: (1 ==> KSXP_TRANSPORT_ID_SKGXPPID) inst: 4 ptid: 257 ukey: 4
  88. Protocol Info: vendor: 1 proto: 2 maj: 3 min: 0
  89. pnum: 106 ospid: 22262
  90. p1_pad1: 0 p2_pad1: (nil)
  91. pad2: 0 pad3: 0
  92. Dumping SKGXP Port ID: 0x2adcaa7c8fb0
  93. SKGXP: SKGXPID 0x2adcaa7c8fb0
  94. SKGXP: skgxp port number 0x3d184b85 process id 22262
  95. SKGXP: admin port id
  96. SKGXP: SKGXPGPID 0x7fff520bf248
  97. SKGXP:         Internet address 10.10.10.14
  98. SKGXP:         UDP port number 2478
  99. Dumping connection stats 0x2bffe0c10
  100. Dumping SKGXP Connection Handle: osd cnh: 0x2adcaa7d5a40
  101. SKGXP: SKGXPCNH: 0x2adcaa7d5a40 SKGXPCON_CLOSED (1) sconno 678fde36 accono 570d699e admno 5efd8d43
  102. SKGXP:         Remote admin port
  103. SKGXP: SSKGXPT 0xaa7d5a7c flags SSKGXPT_WRITE sockno 10 IP 10.10.10.14 UDP 2478
  104. SKGXP:         Remote data port
  105. SKGXP: SSKGXPT 0xaa7d5b10 flags SSKGXPT_WRITE sockno 10 IP 10.10.10.14 UDP 2482
  106. SKGXP:         no disconnect notification request handle
  107. SKGXP:          next seqno 14317 credits 8 ertt 16 resends on con 202

  109. *** 2012-11-08 19:14:42.426
  110. Submitting synchronized dump request [268435460]
  111. ORA-27508: IPC error sending a message

  113. *** 2012-11-08 19:14:43.137
  114. ksxpsrvdt: Invoking KST Dump on Send timeout
  115. -------------------------------------------------------------------------------
  116. Trace Bucket Dump Begin: default bucket for process 106 (osid: 11515, LCK0)
  117. TIME(*=approx):SEQ:COMPONENT:FILE@LINE:FUNCTION:SECT/DUMP: [EVENT#:PID:SID] DATA

复制代码
11月8日发生的问题 现在才来问？


请给出 2个节点的cssd.log 和 /var/log/messages 完整日志

回复 只看该作者 道具 举报

3#
发表于 2012-12-7 14:53:50
要看操作系统的日志，RAC挂掉首先要看操作系统。

回复 只看该作者 道具 举报

2#
发表于 2012-12-7 13:15:15
为什么大神不来了，自先顶。

在一些特定os kernel下也会出现问题,但我的是
SUSE Linux Enterprise Server 10 SP3 (x86_64) - Kernel 2.6.16.60-0.66.1-smp (1).

In this Document
  Symptoms
  Cause
  Solution
  References
________________________________________
This document is being delivered to you via Oracle Support's Rapid Visibility (RaV) process, and therefore has not been subject to an independent technical review.
Applies to:
Linux Kernel - Version: 2.6.9-67.0.0.1 to 2.6.9-67.0.4.1
Linux x86-64
Symptoms
In a RAC environment, the instance is down and the following errors could be found in trace file:
ORA-27508: IPC error sending a message
ORA-27300: OS system dependent operation:sendmsg failed with status: 14
ORA-27301: OS failure message: Bad address
ORA-27302: failure occurred at: sskgxpsnd1

and there is no present error in system log /var/log/messages.
Cause
In kernel 2.6.9-67.0.4.0.1.EL, there has a defect that may cause EFAULT (14) very randomly for system call sendmsg or sendto. Setting MTU to 1500 can avoid this bug. However, when using jumbo frames, this bug may be triggered in a certain payload length. The bug has been fixed on OEL4U7.
Solution
Upgrading the kernel to EL4U7 can resolve this issue. To do so, please run:
#up2date -u kernel
Reboot the machine with the new EL4U7 kernel and the problem should be gone.

回复 只看该作者 道具 举报

返回列表
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 注册

QQ|手机版|Archiver|Oracle数据库数据恢复、性能优化

GMT+8, 2024-11-16 04:39 , Processed in 0.055339 second(s), 24 queries .

Powered by Discuz! X2.5

© 2001-2012 Comsenz Inc.

回顶部
TEL/電話+86 13764045638
Email service@parnassusdata.com
QQ 47079569