rac 一个实例无法启动

albert

环境
rhel5.7 64bit
两节点RAC，oracle 11.2.0.2 64bit
inst1 运行正常，inst2 crs进程无法启动，crs日志为：
2013-06-26 19:13:22.698: [  OCRASM][710406736]proprasmo: The ASM instance is down
2013-06-26 19:13:22.699: [  OCRRAW][710406736]proprioo: Failed to open [+OCRDG]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2013-06-26 19:13:22.699: [  OCRRAW][710406736]proprioo: No OCR/OLR devices are usable
2013-06-26 19:13:22.699: [  OCRASM][710406736]proprasmcl: asmhandle is NULL
2013-06-26 19:13:22.699: [    GIPC][710406736] gipcCheckInitialization: possible incompatible non-threaded init from [prom.c : 687], original from [clsss.c : 5325]
2013-06-26 19:13:22.701: [ default][710406736]clsvactversion:4: Retrieving Active Version from local storage.
2013-06-26 19:13:22.703: [  OCRRAW][710406736]proprrepauto: The local OCR configuration matches with the configuration published by OCR Cache Writer. No repair required.
2013-06-26 19:13:22.706: [  OCRRAW][710406736]proprinit: Could not open raw device
2013-06-26 19:13:22.706: [  OCRASM][710406736]proprasmcl: asmhandle is NULL
2013-06-26 19:13:22.708: [  OCRAPI][710406736]a_init:16!: Backend init unsuccessful : [26]
2013-06-26 19:13:22.708: [  CRSOCR][710406736] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage
ORA-15077: could not locate ASM instance serving a required diskgroup

2013-06-26 19:13:22.708: [ CRSMAIN][710406736] Created alert : (:CRSD00111:) :  Could not init OCR, error: PROC-26: Error while accessing the physical storage
ORA-15077: could not locate ASM instance serving a required diskgroup

2013-06-26 19:13:22.708: [    CRSD][710406736][PANIC] CRSD exiting: Could not init OCR, code: 26
2013-06-26 19:13:22.708: [    CRSD][710406736] Done.

尝试启动ASM实例，后台日志为：
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
Using parameter settings in server-side spfile +OCRDG/omsrpt-cluster/asmparameterfile/registry.253.816787259
System parameters with non-default values:
  large_pool_size          = 12M
  instance_type            = "asm"
  remote_login_passwordfile= "EXCLUSIVE"
  asm_diskstring           = "/dev/mapper"
  asm_diskgroups           = "DATA_DM01"
  asm_diskgroups           = "RECO_DM01"
  asm_diskgroups           = "DATA_DM01_0013"
  asm_power_limit          = 1
  diagnostic_dest          = "/oracle/app/grid"
Cluster communication is configured to use the following interface(s) for this instance
  169.254.191.23
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=25556
PSP0 started with pid=3, OS id=25560
2013-06-26 19:48:42.647000 +08:00
VKTM started with pid=4, OS id=25564 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
GEN0 started with pid=5, OS id=25570
DIAG started with pid=6, OS id=25574
PING started with pid=7, OS id=25578
DIA0 started with pid=8, OS id=25582
LMON started with pid=9, OS id=25586
LMD0 started with pid=10, OS id=25590
* Load Monitor used for high load check
* New Low - High Load Threshold Range = [23040 - 30720]
LMS0 started with pid=11, OS id=25594 at elevated priority
LMHB started with pid=12, OS id=25600
MMAN started with pid=13, OS id=25604
DBW0 started with pid=14, OS id=25608
LGWR started with pid=15, OS id=25612
CKPT started with pid=16, OS id=25616
SMON started with pid=17, OS id=25620
RBAL started with pid=18, OS id=25624
GMON started with pid=19, OS id=25628
MMON started with pid=20, OS id=25632
MMNL started with pid=21, OS id=25636
lmon registered with NM - instance number 2 (internal mem no 1)
2013-06-26 19:48:43.729000 +08:00
Reconfiguration started (old inc 0, new inc 16)
ASM instance
List of instances:
1 2 (myinst: 2)
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid = 1 according to instance 1
* allocate domain 1, invalid = TRUE
* domain 1 valid = 1 according to instance 1
* allocate domain 2, invalid = TRUE
* domain 2 valid = 1 according to instance 1
* allocate domain 4, invalid = TRUE
* domain 4 valid = 1 according to instance 1
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=22, OS id=25640
2013-06-26 19:48:46.022000 +08:00
ORACLE_BASE from environment = /oracle/app/grid
2013-06-26 19:50:56.700000 +08:00
SQL> alter diskgroup OCRDG mount
NOTE: cache registered group OCRDG number=2 incarn=0x5590c1d8
NOTE: cache began mount (not first) of group OCRDG number=2 incarn=0x5590c1d8
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Assigning number (2,2) to disk (/dev/mapper/disk10c)
NOTE: Assigning number (2,1) to disk (/dev/mapper/disk10b)
NOTE: Assigning number (2,0) to disk (/dev/mapper/disk10a)
GMON querying group 2 at 2 for pid 23, osid 25647
NOTE: cache opening disk 0 of grp 2: OCRDG_0000 path:/dev/mapper/disk10a
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache opening disk 1 of grp 2: OCRDG_0001 path:/dev/mapper/disk10b
NOTE: F1X0 found on disk 1 au 2 fcn 0.0
NOTE: cache opening disk 2 of grp 2: OCRDG_0002 path:/dev/mapper/disk10c
NOTE: F1X0 found on disk 2 au 2 fcn 0.0
NOTE: cache mounting (not first) normal redundancy group 2/0x5590C1D8 (OCRDG)
kjbdomatt send to inst 1
NOTE: attached to recovery domain 2
NOTE: redo buffer size is 256 blocks (1053184 bytes)
NOTE: LGWR attempting to mount thread 2 for diskgroup 2 (OCRDG)
NOTE: LGWR found thread 2 closed at ABA 7.5
NOTE: LGWR mounted thread 2 for diskgroup 2 (OCRDG)
NOTE: LGWR opening thread 2 at fcn 0.743 ABA 8.6
NOTE: cache mounting group 2/0x5590C1D8 (OCRDG) succeeded
NOTE: cache ending mount (success) of group OCRDG number=2 incarn=0x5590c1d8
GMON querying group 2 at 3 for pid 18, osid 25624
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2
2013-06-26 19:50:58.100000 +08:00
NOTE: Attempting voting file refresh on diskgroup OCRDG
在mount OCRDG时长时间没有反应，故取消。挂载其他dg时现象一样
没有思路了

albert

在正常运行的实例上面查看v$asm_diskgroup，程序一直没有反应