hpux 11.23 oracle10201 rac 突然不能启动crs了

chinadm123

$ crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
$
-------------------------------------------------------------------------------
# ls
admin           alertnode1.log  client          crsd            cssd            evmd            racg
# pwd
/u01/app/oracle/product/10.2.0/crs_1/log/node1
# tail -50 al*
2011-12-02 04:49:41.273
[cssd(3466)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 .
2011-12-02 04:49:42.064
[crsd(1863)]CRS-1204:Recovering CRS resources for node node2.
2011-12-02 04:54:44.294
[cssd(3466)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 node2 .
2011-12-02 05:15:59.822
[cssd(3466)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 .
2011-12-02 05:15:59.878
[crsd(1863)]CRS-1204:Recovering CRS resources for node node2.
2011-12-02 05:16:17.194
[cssd(3466)]CRS-1603:CSSD on node node1 shutdown by user.
2011-12-04 20:42:05.178
[cssd(2289)]CRS-1605:CSSD voting file is online: /dev/vgvote/rmyvote. Details in /u01/app/oracle/product/10.2.0/crs_1/log/node1/cssd/ocssd.log.
2011-12-04 20:42:09.782
[cssd(2289)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 node2 .
2011-12-04 20:42:10.450
[crsd(1904)]CRS-1012:The OCR service started on node node1.
2011-12-04 20:42:10.472
[evmd(2136)]CRS-1401:EVMD started on node node1.
2011-12-04 20:42:12.066
[crsd(1904)]CRS-1201:CRSD started on node node1.
2011-12-04 21:14:38.847
[cssd(2291)]CRS-1605:CSSD voting file is online: /dev/vgvote/rmyvote. Details in /u01/app/oracle/product/10.2.0/crs_1/log/node1/cssd/ocssd.log.
2011-12-04 21:14:43.315
[cssd(2291)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 node2 .
2011-12-04 21:14:43.984
[crsd(1905)]CRS-1012:The OCR service started on node node1.
2011-12-04 21:14:44.005
[evmd(2135)]CRS-1401:EVMD started on node node1.
2011-12-04 21:14:45.688
[crsd(1905)]CRS-1201:CRSD started on node node1.
2011-12-04 21:41:18.393
[cssd(2291)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 .
2011-12-04 21:41:35.251
[cssd(2291)]CRS-1603:CSSD on node node1 shutdown by user.
2011-12-04 21:47:05.646
[cssd(2224)]CRS-1605:CSSD voting file is online: /dev/vgvote/rmyvote. Details in /u01/app/oracle/product/10.2.0/crs_1/log/node1/cssd/ocssd.log.
2011-12-04 21:48:25.442
[cssd(2224)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 .
2011-12-04 21:48:26.231
[crsd(1837)]CRS-1012:The OCR service started on node node1.
2011-12-04 21:48:26.253
[evmd(2075)]CRS-1401:EVMD started on node node1.
2011-12-04 21:48:27.794
[crsd(1837)]CRS-1201:CRSD started on node node1.
2011-12-05 19:53:32.992
[client(5014)]CRS-1006:The OCR location /dev/vgvote/rmyvote is inaccessible. Details in /u01/app/oracle/product/10.2.0/crs_1/log/node1/client/ocrcheck_5014.log.
2011-12-05 19:53:45.159
[client(5017)]CRS-1006:The OCR location /dev/vgvote/rmyvote is inaccessible. Details in /u01/app/oracle/product/10.2.0/crs_1/log/node1/client/ocrcheck_5017.log.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
# more /u01/app/oracle/product/10.2.0/crs_1/log/node1/client/ocrcheck_5014.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle.  All rights reserved.
2011-12-05 19:53:32.982: [OCRCHECK][1]ocrcheck starts...
2011-12-05 19:53:32.985: [  OCRRAW][1]propriogid:1: INVALID FORMAT
2011-12-05 19:53:32.997: [  OCRRAW][1]ibctx:1:ERROR: INVALID FORMAT
2011-12-05 19:53:32.997: [  OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22

2011-12-05 19:53:32.997: [ default][1]a_init:7!: Backend init unsuccessful : [22]
2011-12-05 19:53:32.998: [OCRCHECK][1]Failed to initialize OCR context: [PROC-22: The OCR backend has an invalid f
ormat]
2011-12-05 19:53:32.998: [OCRCHECK][1]Failed to initialize ocrchek2
2011-12-05 19:53:32.998: [OCRCHECK][1]Exiting [status=failed]...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
# more /u01/app/oracle/product/10.2.0/crs_1/log/node1/client/ocrcheck_5017.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle.  All rights reserved.
2011-12-05 19:53:45.155: [OCRCHECK][1]ocrcheck starts...
2011-12-05 19:53:45.158: [  OCRRAW][1]propriogid:1: INVALID FORMAT
2011-12-05 19:53:45.168: [  OCRRAW][1]ibctx:1:ERROR: INVALID FORMAT
2011-12-05 19:53:45.169: [  OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22

2011-12-05 19:53:45.169: [ default][1]a_init:7!: Backend init unsuccessful : [22]
2011-12-05 19:53:45.169: [OCRCHECK][1]Failed to initialize OCR context: [PROC-22: The OCR backend has an invalid f
ormat]
2011-12-05 19:53:45.169: [OCRCHECK][1]Failed to initialize ocrchek2
2011-12-05 19:53:45.169: [OCRCHECK][1]Exiting [status=failed]...
#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
# ./ocrconfig -showbackup

node1     2011/12/05 01:48:27     /u01/app/oracle/product/10.2.0/crs_1/cdata/crs

node1     2011/12/02 00:33:11     /u01/app/oracle/product/10.2.0/crs_1/cdata/crs

node1     2011/12/01 00:54:40     /u01/app/oracle/product/10.2.0/crs_1/cdata/crs

node1     2011/12/05 01:48:27     /u01/app/oracle/product/10.2.0/crs_1/cdata/crs

node1     2011/11/30 20:54:39     /u01/app/oracle/product/10.2.0/crs_1/cdata/crs
#
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
# lvdisplay /dev/vgcrs/mycrs
--- Logical volumes ---
LV Name                     /dev/vgcrs/mycrs
VG Name                     /dev/vgcrs
LV Permission               read/write   
LV Status                   available/syncd           
Mirror copies               0            
Consistency Recovery        MWC                 
Schedule                    parallel     
LV Size (Mbytes)            5000            
Current LE                  1250      
Allocated PE                1250        
Stripes                     0      
Stripe Size (Kbytes)        0                  
Bad block                   on           
Allocation                  strict                    
IO Timeout (Seconds)        default            

# lvdisplay /dev/vgvote/myvote
--- Logical volumes ---
LV Name                     /dev/vgvote/myvote
VG Name                     /dev/vgvote
LV Permission               read/write   
LV Status                   available/syncd           
Mirror copies               0            
Consistency Recovery        MWC                 
Schedule                    parallel     
LV Size (Mbytes)            500            
Current LE                  125      
Allocated PE                125         
Stripes                     0      
Stripe Size (Kbytes)        0                  
Bad block                   on           
Allocation                  strict                    
IO Timeout (Seconds)        default            
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
我也尝试过重启crs：
# ./crsctl stop crs
Stopping resources.
Error while stopping resources. Possible cause: CRSD is down.
Stopping CSSD.
Unable to communicate with the CSS daemon.
# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
# ./crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
我的ocr和vote disk都是使用的裸卷，分别是/dev/vgcrs/mycrs、/dev/vgvote/myvote
现在我感觉是我的ocr disk出了问题？但我不确定。最终应该怎么做，希望大家给点意见!多谢！

[ 本帖最后由 chinadm123 于 2011-12-6 13:10 编辑 ]

Honcho

You should provide more information like crsd.log ,ocssd log  ,vip log and so on;

Maclean Liu(刘相兵

你好， 抱歉回复的有点迟

就日志来看 最主要的症结在于 OCR无法访问

[client(5014)]CRS-1006:The OCR location /dev/vgvote/rmyvote is inaccessible. Details in /u01/app/oracle/product/10.2.0/crs_1/log/node1/client/ocrcheck_5014.log.


2011-12-05 19:53:32.982: [OCRCHECK][1]ocrcheck starts...
2011-12-05 19:53:32.985: [  OCRRAW][1]propriogid:1: INVALID FORMAT
2011-12-05 19:53:32.997: [  OCRRAW][1]ibctx:1:ERROR: INVALID FORMAT
2011-12-05 19:53:32.997: [  OCRRAW][1]proprinit:problem reading the bootblock or superbloc 22


因为你的问题是 突然无法访问 ， 可以从之前主机或存储上做过的操作入手来查该问题。

恢复建议：

1. 使用备份的OCR 还原
2. 重新安装CRS软件， 建议至少升级到10.2.0.5