RAC升级后的问题

紫恒

RAC版本：    10.2.0.5
数据库版本： 10.2.0.5
RAC升级前的版本是10.2.0.1,在升级完集群和数据库之后，重新启动集群的时候asm实例只能启动一边[root@node1 bin]# ./crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....B1.inst application    ONLINE    OFFLINE               
ora....B2.inst application    ONLINE    OFFLINE               
ora.RACDB.db   application    ONLINE    OFFLINE               
ora....SM1.asm application    ONLINE    ONLINE    node1      
ora....E1.lsnr application    ONLINE    ONLINE    node1      
ora.node1.gsd  application    ONLINE    ONLINE    node1      
ora.node1.ons  application    ONLINE    ONLINE    node1      
ora.node1.vip  application    ONLINE    ONLINE    node1      
ora....SM2.asm application    ONLINE    OFFLINE              
ora....E2.lsnr application    ONLINE    ONLINE    node2      
ora.node2.gsd  application    ONLINE    ONLINE    node2      
ora.node2.ons  application    ONLINE    ONLINE    node2      
ora.node2.vip  application    ONLINE    ONLINE    node2      
手动用命令启动ASM实例，出现报错
PRKS-1009 : 无法启动节点 "node2", [PRKS-1009 : 无法启动节点 "node2", [CRS-0215: Could not start resource 'ora.node2.ASM2.asm'.] 上的 ASM 实例 "+ASM2"] 上的 ASM 实例 "+ASM2"
  [PRKS-1009 : 无法启动节点 "node2", [CRS-0215: Could not start resource 'ora.node2.ASM2.asm'.] 上的 ASM 实例 "+ASM2"]
日志在附件里，如果还缺少哪个日志，请告知。。。谢谢

seedog123

贴出你集群的升级过程

紫恒

我是按照补丁里面的readme.html的步骤进行升级的，关闭em,监听，数据库实例，集群，然后升级集群软件，执行root102.sh成功，无报错信息，然后再次关闭rac的服务，升级数据库，完成后，执行ROOT.SH；手工启动集群，ASM时，ASM报错。。就这个咯，然后没继续了

紫恒

需不需要运行这个语句？
SQL> DECLARE
              CURSOR stmt IS
                       SELECT u.name, o.name, r.pname
                           FROM user$ u, obj$ o, rls$ r
                       WHERE u.user# = o.owner#
                           AND r.obj# = o.obj#
                           AND bitand(r.stmt_type,65536) > 0;

                       object_schema VARCHAR2(32) := NULL;
                       object_name VARCHAR2(32) := NULL;
                       policy_name VARCHAR2(32) := NULL;

                       BEGIN
                        OPEN stmt;
                          LOOP
                               FETCH stmt INTO object_schema,
                                               object_name,
                                               policy_name;
                               EXIT WHEN stmt%NOTFOUND;
                               DBMS_RLS.DROP_POLICY(
                               '"'||object_schema||'"',
                               '"'||object_name||'"',
                               '"'||policy_name||'"');
                          END LOOP;
                        CLOSE stmt;
                       END;

Maclean Liu(刘相兵

ODM DATA:

2012-06-14 18:03:17.997: [  CLSVER][2448160]0Active Version from OCR:10.2.0.5.0

2012-06-14 18:13:37.347: [  CRSRES][2632965008]0node1 : CRS-1019: Resource ora.node2.ASM2.asm (application) cannot run on node1

2012-06-14 18:13:37.379: [  CRSAPP][2687413136]0StartResource error for ora.RACDB.db error code = 1
2012-06-14 18:13:37.457: [  CRSRES][2687413136]0Start of `ora.RACDB.db` on member `node2` failed.
2012-06-14 18:13:37.542: [  CRSRES][2687413136]0Attempting to start `ora.RACDB.db` on member `node1`
2012-06-14 18:13:37.692: [  CRSRES][2643454864]0startRunnable: setting CLI values
2012-06-14 18:15:43.357: [  CRSAPP][2643454864]0StartResource error for ora.node2.ASM2.asm error code = 1
2012-06-14 18:15:45.070: [  CRSRES][2687413136]0Start of `ora.RACDB.db` on member `node1` failed.
2012-06-14 18:19:56.958: [  CRSRES][2687413136]0startRunnable: setting CLI values
2012-06-14 18:19:57.062: [  CRSEVT][2687413136]0CAAMonitorHandler :: 0:Action Script for resource 'ora.node2.ASM2.asm' stdout redirection failed for `/u01/app/oracle/product/10.2.0/crs_1/crs/log/startEwdrST.stdout` : No such file or directory
2012-06-14 18:22:02.582: [  CRSAPP][2687413136]0StartResource error for ora.node2.ASM2.asm error code = 1
2012-06-14 18:22:59.205: [  CRSRES][2687413136]0startRunnable: setting CLI values
2012-06-14 18:22:59.224: [  CRSEVT][2687413136]0CAAMonitorHandler :: 0:Action Script for resource 'ora.node2.ASM2.asm' stdout redirection failed for `/u01/app/oracle/product/10.2.0/crs_1/crs/log/startgduYtr.stdout` : No such file or directory
2012-06-14 18:25:04.805: [  CRSAPP][2687413136]0StartResource error for ora.node2.ASM2.asm error code = 1
2012-06-14 18:25:19.151: [  CRSRES][2687413136]0startRunnable: setting CLI values
2012-06-14 18:25:19.172: [  CRSEVT][2687413136]0CAAMonitorHandler :: 0:Action Script for resource 'ora.node2.ASM2.asm' stdout redirection failed for `/u01/app/oracle/product/10.2.0/crs_1/crs/log/startKHFEvf.stdout` : No such file or directory



上传 2号节点  ASM实例的alert.log

紫恒

大神。。会不会是共享磁盘出错了导致的？

Maclean Liu(刘相兵

10.2.0.5.0.

Thu Jun 14 18:23:00 CST 2012
LMS 0: 0 GCS shadows traversed, 0 replayed
Thu Jun 14 18:23:00 CST 2012
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=15, OS id=15399
Thu Jun 14 18:25:01 CST 2012
USER: terminating instance due to error 15150
Instance terminated by USER, pid = 15364





[oracle@vrh8 ~]$ oerr ora 16766
16766, 00000, "Redo Apply unexpectedly offline"
// *Cause:  Redo Apply was offline when it should be online.
// *Action: If necessary, start Redo Apply.
[oracle@vrh8 ~]$ oerr ora 15150
15150, 00000, "instance lock mode '%s' conflicts with other ASM instance(s)"
// *Cause: Some other ASM instance used the lock name space in a conflicting
//         mode.
// *Action: Shut down the other instance or start up in compatible mode.
//          Alternatively, set the DB_UNIQUE_NAME initialization parameter
//          to avoid the conflict.
//



Thu Jun 14 16:20:46 2012
NOTE: detached from domain 2
Thu Jun 14 16:20:46 2012
SUCCESS: diskgroup RAC_VOLGRP1 was dismounted
Thu Jun 14 17:41:31 CST 2012
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 19.168.100.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 172.24.65.0 configured from OCR for use as  a public interface
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/10.2.0/db_1/dbs/arch
Autotune of undo retention is turned off.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.5.0.
System parameters with non-default values:
  large_pool_size          = 12582912
  instance_type            = asm
  cluster_database         = TRUE
  instance_number          = 2
  remote_login_passwordfile= EXCLUSIVE
  background_dump_dest     = /u01/app/oracle/admin/+ASM/bdump
  user_dump_dest           = /u01/app/oracle/admin/+ASM/udump
  core_dump_dest           = /u01/app/oracle/admin/+ASM/cdump
  asm_diskgroups           = RAC_VOLGRP1, FLASH_RECOVERY_AREA
Cluster communication is configured to use the following interface(s) for this instance
  19.168.100.102
Thu Jun 14 17:41:32 CST 2012
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=23682
DIAG started with pid=3, OS id=23684
PSP0 started with pid=4, OS id=23686
LMON started with pid=5, OS id=23688
LMD0 started with pid=6, OS id=23690
LMS0 started with pid=7, OS id=23692
MMAN started with pid=8, OS id=23696
LGWR started with pid=10, OS id=23700
CKPT started with pid=11, OS id=23702
SMON started with pid=12, OS id=23704
RBAL started with pid=13, OS id=23706
DBW0 started with pid=9, OS id=23698
GMON started with pid=14, OS id=23708
Thu Jun 14 17:41:33 CST 2012
lmon registered with NM - instance id 2 (internal mem no 1)
Thu Jun 14 17:41:33 CST 2012
Reconfiguration started (old inc 0, new inc 2)
ASM instance
List of nodes:
0 1
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu Jun 14 17:41:34 CST 2012
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Thu Jun 14 17:41:35 CST 2012
LMS 0: 0 GCS shadows traversed, 0 replayed
Thu Jun 14 17:41:35 CST 2012
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=15, OS id=23747
Thu Jun 14 17:43:35 CST 2012
USER: terminating instance due to error 15150



第一次出现 ORA-15150 之前+ASM2 实例应当没有 干净地关闭 ， 没有 shutdown的日志


建议你 重启 2号节点

紫恒

ASM都没挂载上。。能重启么。。实例都开不了吧。。。难道是先让+ASM2先ONLINE。然后进入数据库关闭么？

紫恒

我两边重启后，先挂上NODE2的ASM,然后再挂NODE1的ASM，也提示出错
[root@node1 bin]# ./srvctl start asm -n node1
PRKS-1009 : 无法启动节点 "node1", [PRKS-1009 : 无法启动节点 "node1", [node1:ora.node1.ASM1.asm:
node1:ora.node1.ASM1.asm:SQL*Plus: Release 10.2.0.5.0 - Production on Thu Jun 14 23:34:06 2012
node1:ora.node1.ASM1.asm:
node1:ora.node1.ASM1.asm:Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.
node1:ora.node1.ASM1.asm:
node1:ora.node1.ASM1.asm:Enter user-name: Connected to an idle instance.
node1:ora.node1.ASM1.asm:
node1:ora.node1.ASM1.asm:SQL> ORA-15150: instance lock mode 'EXCLUSIVE' conflicts with other ASM instance(s)
node1:ora.node1.ASM1.asm:SQL> Disconnected
node1:ora.node1.ASM1.asm:
CRS-0215: Could not start resource 'ora.node1.ASM1.asm'.] 上的 ASM 实例 "+ASM1"] 上的 ASM 实例 "+ASM1"
  [PRKS-1009 : 无法启动节点 "node1", [node1:ora.node1.ASM1.asm:
node1:ora.node1.ASM1.asm:SQL*Plus: Release 10.2.0.5.0 - Production on Thu Jun 14 23:34:06 2012
node1:ora.node1.ASM1.asm:
node1:ora.node1.ASM1.asm:Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.
node1:ora.node1.ASM1.asm:
node1:ora.node1.ASM1.asm:Enter user-name: Connected to an idle instance.
node1:ora.node1.ASM1.asm:
node1:ora.node1.ASM1.asm:SQL> ORA-15150: instance lock mode 'EXCLUSIVE' conflicts with other ASM instance(s)
node1:ora.node1.ASM1.asm:SQL> Disconnected
node1:ora.node1.ASM1.asm:

Maclean Liu(刘相兵

ORA-15150: instance lock mode 'EXCLUSIVE' conflicts with other ASM instance(s)


怀疑你的 asm1的参数有问题 导致 实例冲突

贴出ASM1 的alert.log