集群服务无法启动的问题，急求解答

zl293

刘大，我这生产服务器的其中一个RAC节点无法启动，只有另一个能够启动，请帮忙看看
T4-1小型机与st2540存储
ORACLE VERSION 11.2.0.3
OS VERSION     5.10 Generic_147440-25 sun4v sparc sun4v

rac2是正常的
-bash-3.2$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    racdb2      
ora....ER.lsnr ora....er.type ONLINE    ONLINE    racdb2      
ora....N1.lsnr ora....er.type ONLINE    ONLINE    racdb2      
ora.asm        ora.asm.type   ONLINE    ONLINE    racdb2      
ora.cvu        ora.cvu.type   ONLINE    ONLINE    racdb2      
ora.gbiaps.db  ora....se.type ONLINE    ONLINE    racdb2      
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    racdb2      
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    racdb2      
ora.ons        ora.ons.type   ONLINE    ONLINE    racdb2      
ora.racdb1.vip ora....t1.type ONLINE    ONLINE    racdb2      
ora....SM2.asm application    ONLINE    ONLINE    racdb2      
ora....B2.lsnr application    ONLINE    ONLINE    racdb2      
ora.racdb2.gsd application    OFFLINE   OFFLINE               
ora.racdb2.ons application    ONLINE    ONLINE    racdb2      
ora.racdb2.vip ora....t1.type ONLINE    ONLINE    racdb2      
ora....ry.acfs ora....fs.type ONLINE    OFFLINE               
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    racdb2  

Maclean Liu(刘相兵

需要
/oracle/app/grid_home/log/racdb1/agent/ohasd/oraagent_grid/oraagent_grid.log

ASM 的alert.log

ohasd.log 和ocssd.log

zl293

日志文件已上传

Maclean Liu(刘相兵

Error 29701: unexpected return code 6 from the Cluster Synchronization Service
Errors in file /oracle/app/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_5743.trc:
ORA-29701: unable to connect to Cluster Synchronization Service
LMON (ospid: 5743): terminating the instance due to error 29701
System state dump requested by (instance=1, osid=5743 (LMON)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_diag_5737.trc
Dumping diagnostic data in directory=[cdmp_20130828223634], requested by (instance=1, osid=5743 (LMON)), summary=[abnormal instance termination].
2013-08-28 22:36:35.197000 +08:00
Instance terminated by LMON, pid = 5743

zl293

Maclean Liu(刘相兵 发表于 2013-8-29 14:03
Error 29701: unexpected return code 6 from the Cluster Synchronization Service
Errors in file /oracl ...

ASM只有昨晚的日志，昨晚因为维护，在十点半把机器给关了。今早这个实例就起不来了。

zl293

ASM trc文件

Maclean Liu(刘相兵

2013-08-28 22:36:32.720: [    CSSD][20](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 1601904734 ms for voting file /dev/rdsk/c10t60080E50002F61F8000003B750BD2A4Cd0s6)
2013-08-28 22:36:32.721: [    CSSD][20]clssnmvDiskAvailabilityChange: voting file /dev/rdsk/c10t60080E50002F61F8000003B750BD2A4Cd0s6 now offline
2013-08-28 22:36:32.721: [    CSSD][20](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2013-08-28 22:36:32.721: [    CSSD][20]clssscExit: abort already set 1
2013-08-28 22:36:32.721: [   SKGFD][18]Lib :UFS:: closing handle 1016c8990 for disk :/dev/rdsk/c10t60080E50002F61F8000003B750BD2A4Cd0s6:

2013-08-28 22:36:32.721: [   SKGFD][19]Lib :UFS:: closing handle 101956dd0 for disk :/dev/rdsk/c10t60080E50002F61F8000003B750BD2A4Cd0s6:

2013-08-28 22:36:32.721: [   SKGFD][17]Lib :UFS:: closing handle 101747210 for disk :/dev/rdsk/c10t60080E50002F61F8000003B750BD2A4Cd0s6:

2013-08-28 22:36:32.728: [    CSSD][5]clssgmUnregisterShared: Same group share client 2 (1018d2590), grp DB+ASM, member 0



只有一个vote disk?

/dev/rdsk/c10t60080E50002F61F8000003B750BD2A4Cd0s6

zl293

c10t60080E50002F61F8000003B750BD2A4Cd0s0
c10t60080E50002F61F8000003B750BD2A4Cd0s1
c10t60080E50002F61F8000003B750BD2A4Cd0s2
c10t60080E50002F61F8000003B750BD2A4Cd0s3
c10t60080E50002F61F8000003B750BD2A4Cd0s4
c10t60080E50002F61F8000003B750BD2A4Cd0s5
c10t60080E50002F61F8000003B750BD2A4Cd0s6
c10t60080E50002F61F8000003B750BD2A4Cd0s7
/dev/rdsk/下有这些