rac升级 asm权限问题

qhd2004

环境：
Red Hat Enterprise Linux Server release 5.4 (Tikanga)
10205 rac升级到11204 rac
已经建好磁盘组TEST用于ocr和votedisk

但ocr和votedisk从裸设备迁移到asm磁盘组时报错：
1， ./ocrconfig -add +TEST执行后，TEST磁盘组就认不到了（dismount了）
[oracle@BJ-CP-VM-107-71 ~]$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576     20479    11346                0           11346              0             N  DATA01/
MOUNTED  EXTERN  N         512   4096  1048576      5120     3515                0            3515              0             N  TEST/
上面为执行./ocrconfig -add +TEST之前的查询

ASMCMD> lsdg        
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576     20479    11346                0           11346              0             N  DATA01/
ASMCMD>
上面为执行./ocrconfig -add +TEST之后的查询

2，alert日志
tail -n 100 alert_+ASM1.log |more
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
ORA-15066: offlining disk "TEST_0000" in group "TEST" may result in a data loss
Errors in file /apps/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_20281.trc:
ORA-15025: could not open disk "/dev/asm/ocr_voting"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
ORA-15066: offlining disk "TEST_0000" in group "TEST" may result in a data loss
NOTE: starting check of diskgroup TEST
Errors in file /apps/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_20281.trc:
ORA-15025: could not open disk "/dev/asm/ocr_voting"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
ORA-15066: offlining disk "TEST_0000" in group "TEST" may result in a data loss
Errors in file /apps/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_20281.trc:
ORA-15025: could not open disk "/dev/asm/ocr_voting"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
ORA-15066: offlining disk "TEST_0000" in group "TEST" may result in a data loss
WARNING: cache failed reading from group=2(TEST) fn=1 blk=0 count=1 from disk= 0 (TEST_0000) kfkist=0x20 status=0x02 osderr=0x0 file=kfc.c line=11596
Errors in file /apps/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_20281.trc:
ORA-15025: could not open disk "/dev/asm/ocr_voting"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
ORA-15080: synchronous I/O operation to a disk failed
ORA-15066: offlining disk "TEST_0000" in group "TEST" may result in a data loss
ERROR: cache failed to read group=2(TEST) fn=1 blk=0 from disk(s): 0(TEST_0000)
ORA-15080: synchronous I/O operation to a disk failed
ORA-15066: offlining disk "TEST_0000" in group "TEST" may result in a data loss
NOTE: cache initiating offline of disk 0 group TEST
NOTE: process _user20281_+asm1 (20281) initiating offline of disk 0.3915948627 (TEST_0000) with mask 0x7e in group 2
NOTE: initiating PST update: grp = 2, dsk = 0/0xe968a253, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 19 for pid 31, osid 20281
ERROR: no read quorum in group: required 1, found 0 disks
Fri Apr 24 21:10:47 2015
NOTE: cache dismounting (not clean) group 2/0x2CA852DE (TEST)
WARNING: Offline for disk TEST_0000 in mode 0x7f failed.
Fri Apr 24 21:10:47 2015
NOTE: halting all I/Os to diskgroup 2 (TEST)
NOTE: messaging CKPT to quiesce pins Unix process pid: 20287, image: oracle@BJ-CP-VM-107-71 (B000)
ORA-15032: not all alterations performed
ORA-15130: diskgroup "TEST" is being dismounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "TEST"
ORA-15080: synchronous I/O operation to a disk failed
ORA-15066: offlining disk "TEST_0000" in group "TEST" may result in a data loss
ERROR: alter diskgroup TEST check /* proxy */
Fri Apr 24 21:10:47 2015
NOTE: No asm libraries found in the system
ASM Health Checker found 1 new failures
Fri Apr 24 21:10:48 2015
NOTE: LGWR doing non-clean dismount of group 2 (TEST)
NOTE: LGWR sync ABA=4.109 last written ABA 4.109
Fri Apr 24 21:10:48 2015
kjbdomdet send to inst 2
detach from dom 2, sending detach message to inst 2
Fri Apr 24 21:10:48 2015
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE
2 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Fri Apr 24 21:10:48 2015
WARNING: dirty detached from domain 2
NOTE: cache dismounted group 2/0x2CA852DE (TEST)
SQL> alter diskgroup TEST dismount force /* ASM SERVER:749228766 */
Fri Apr 24 21:10:48 2015
NOTE: cache deleting context for group TEST 2/0x2ca852de
GMON dismounting group 2 at 20 for pid 32, osid 20287
NOTE: Disk TEST_0000 in mode 0x7f marked for de-assignment
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 2
Received dirty detach msg from inst 2 for dom 2
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 2, cluster inc 4)
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE
2 GCS resources traversed, 0 cancelled
freeing rdom 2
Dirty Detach Reconfiguration complete
Fri Apr 24 21:11:02 2015
* allocate domain 2, invalid = TRUE
Fri Apr 24 21:11:12 2015
SQL> ALTER DISKGROUP TEST MOUNT  /* asm agent *//* {0:7:10} */
WARNING: Disk Group TEST containing configured OCR is not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "TEST" cannot be mounted
ORA-15013: diskgroup "TEST" is already mounted
ERROR: ALTER DISKGROUP TEST MOUNT  /* asm agent *//* {0:7:10} */
Fri Apr 24 21:11:18 2015
SUCCESS: diskgroup TEST was dismounted
SUCCESS: alter diskgroup TEST dismount force /* ASM SERVER:749228766 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group TEST
Fri Apr 24 21:11:18 2015
NOTE: diskgroup resource ora.TEST.dg is offline

3，权限
[oracle@BJ-CP-VM-107-71 trace]$ ll /dev/asm/*
brw-rw---- 1 oracle oinstall 8, 65 04-24 21:26 /dev/asm/data01
brwxrwxrwx 1 oracle asmadmin 8, 80 04-24 21:10 /dev/asm/ocr_voting

各位，如果是权限问题，那应该给什么权限呢？/dev/asm/ocr_voting的权限给过grid:asmadmin也给过grid:oinstall都不管用。
如果还要其他信息，请提出，我上传。

Maclean Liu(刘相兵

设备权限绑定了吗？ 事实认定似乎不太清楚

qhd2004

[oracle@BJ-CP-VM-107-71 trace]$ more /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="14f504e46494c4500785635375a542d6b614b462d4d707661", NAME="asm/data01", OWNER="oracle", GROUP
="oinstall", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="14f504e46494c45003869596262322d5954626e2d39524138", NAME="asm/ocr_voting", OWNER="oracle", G
ROUP="oinstall", MODE="0660"

这个地方是grid?

qhd2004

[root@BJ-CP-VM-107-71 bin]# ./ocrconfig -add +TEST
PROT-29: The Oracle Cluster Registry location is already configured
[root@BJ-CP-VM-107-71 bin]# ./ocrcheck
Errors in file :
ORA-27091: unable to queue I/O
ORA-15081: failed to submit an I/O operation to a disk
ORA-06512: at line 4
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :    1534396
         Used space (kbytes)      :       6528
         Available space (kbytes) :    1527868
         ID                       :  819046077
         Device/File Name         : /dev/raw/raw1
                                    Device/File integrity check succeeded
         Device/File Name         :      +TEST
                                    Device/File needs to be synchronized with the other device

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded