rac 踢出节点失败

lhl1212

一套RAC数据库于2012/07/28 16:39 出现宕机的情形，具体表现为：数据库当时无法用SQLPLUS登录，节点2 SSH远程登录和控制台均无响应。
节点1 alert_log,一直在等待踢出节点2不成功,而节点2在故障时段未发现产生相关日志。在Jul 28 17:20:05 强制重启节点bf02后CLUSTER恢复正常。请帮忙分析具体原因及解决措施。

环境：
[root@bf01 oswvmstat]# lsb_release -d
Description:    Red Hat Enterprise Linux AS release 4 (Nahant Update 8)

SQL> select * from v$version;
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi
PL/SQL Release 10.2.0.4.0 - Production
CORE    10.2.0.4.0      Production
TNS for Linux: Version 10.2.0.4.0 - Production
NLSRTL Version 10.2.0.4.0 - Production

alter_bf01.log :
Sat Jul 28 16:40:42 2012
IPC Send timeout detected.Sender: ospid 26231
Receiver: inst 2 binc 429416626 ospid 19964
Sat Jul 28 16:40:58 2012
Trace dumping is performing id=[cdmp_20120728164027]
Sat Jul 28 16:41:08 2012
IPC Send timeout to 1.0 inc 4 for msg type 12 from opid 95
Sat Jul 28 16:41:39 2012
Trace dumping is performing id=[cdmp_20120728164027]
Sat Jul 28 16:42:17 2012
Evicting instance 2 from cluster
Sat Jul 28 16:42:52 2012
Waiting for instances to leave:
2
......

相关日志见附件。

lhl1212

diagcollection和故障时点DUMP、AWR信息
请大神帮忙看看。