怎么取消cluster rolling upgrade incomplete状态

riverchild

升级GI，从11.2.0.1 到11.2.0.3,
因为没打patch，出现错误，怎么取消升级？
现在ORA-15154:cluster rolling upgrade incomplete

上海-maximus

Ensure that all the instances in the cluster are upgraded before retrying the command.

Maclean Liu(刘相兵

[oracle@nas ~]$ oerr ora 15154
15154, 00000, "cluster rolling upgrade incomplete"
// *Cause:  The cluster was still performing rolling upgrade.
// *Action: Ensure that all the instances in the cluster are upgraded before
//          retrying the command.
//

ODM FINDING:

Hdr: 13620680 11.2.0.3 PCW 11.2.0.3 PRODID-5 PORTID-226 9706490
Abstract: ROOTUPGRADE.SH FAILING WITH ORA-15154: CLUSTER ROLLING UPGRADE INCOMPLETE

PROBLEM:
--------
2-node RAC cluster being upgraded from 11.2.0.1 to 11.2.0.3.   rootupgrade.sh
is failing in the first node with "ORA-15154: cluster rolling upgrade
incomplete" error.

DIAGNOSTIC ANALYSIS:
--------------------
Failure happen just after asmca is started and 'crstl stop crs -f' is
executed.   

>  ORA-15154: cluster rolling upgrade incomplete
>  ORA-15154: cluster rolling upgrade incomplete
>  ORA-15154: cluster rolling upgrade incomplete
>  ORA-15154: cluster rolling upgrade incomplete
>  ORA-15154: cluster rolling upgrade incomplete
>  ORA-15154: cluster rolling upgrade incomplete
>  ORA-15154: cluster rolling upgrade incomplete

WORKAROUND:
-----------
none.   Tried downgrade procedure and is also failing:

/root : /orahome/11.2.0.3/grid/crs/install/rootcrs.pl -downgrade -force
-lastnode -oldcrshome /orahome/11.2.0/grid -version 11.2.0.1.0
Using configuration parameter file:
/orahome/11.2.0.3/grid/crs/install/crsconfig_params
Could not find OLR backup
Failed to write the checkpoint:'' with status:FAIL.Error code is 256
OLR backup check failed at /orahome/11.2.0.3/grid/crs/install/crsdelete.pm
line 1330.

RELATED BUGS:
-------------
9413827

REPRODUCIBILITY:
----------------
Same problem is also occuring in node 2 where rootupgrade.sh haven't been
executed yet.

TEST CASE:
----------
complete loss of service.   

*** If you are upgrading to 11.2.0.3, Check the section "Upgrade to 11.2.0.3" at the bottom of the note. ***
Upgrade from 11.2.0.1 to 11.2.0.2
If you are upgrading from 11.2.0.1 to 11.2.0.2, following the next section to decide what patch to apply.

There has been some confusion as what patches need to be applied for 11.2.0.1 ASM rolling upgrade to 11.2.0.2 to be successful. Documentation regarding this is not very clear (at the time of writing) and a documentation bug has been filed and documentation will be updated in the future.

There are two bugs related to 11.2.0.1 ASM rolling upgrade to 11.2.0.2:

Unpublished bug 9413827: 11201 TO 11202 ASM ROLLING UPGRADE - OLD CRS STACK FAILS TO STOP

Unpublished bug 9706490: LNX64-11202-UD 11201 -> 11202, DG OFFLINE AFTER RESTART CRS STACK DURING UPGRADE

Some of the symptoms include error message when running rootupgrade.sh:


ORA-15154: cluster rolling upgrade incomplete (from bug: 9413827)


or

Diskgroup status is shown offline after the upgrade, crsd.log may have:


2010-05-12 03:45:49.029: [ AGFW][1506556224] Agfw Proxy Server sending the
last reply to PE for message:RESOURCE_START[ora.MYDG1.dg rwsdcvm44 1] ID 4098:1526
TextMessage[CRS-2674: Start of 'ora.MYDG1.dg' on 'rwsdcvm44' failed]
TextMessage[ora.MYDG1.dg rwsdcvm44 1]
ora.MYDG1.dg rwsdcvm44 1:



To overcome this issue, there are two actions you need to take:

a). apply proper patch.
b). change crsconfig_lib.pm

Applying Patch:

1). If $GI_HOME is on version 11.2.0.1.2 (i.e GI PSU2 is applied):

Action: You can apply Patch:9706490 for version 11.2.0.1.2.


Unpublished bug 9413827 is fixed in 11.2.0.1.2 GI PSU2. Patch:9706490 for version 11.2.0.1.2 is built on top of 11.2.0.1.2 GI PSU2 (i.e. includes the 11.2.0.1.2 GI PSU2, hence includes the fix for 9413827). Applying Patch:9706490 includes both fixes. opatch will recognize 9706490 is superset of 11.2.0.1.2 GI PSU2 (Patch: 9655006) and rollback patch 9655006 before applying Patch: 9706490).

2). If $GI_HOME is on version 11.2.0.1.0 (i.e. no GI PSU applied).

Action: You can apply Patch:9706490 for version 11.2.0.1.2. This would make sure you have applied 11.2.0.1.2 GI PSU2 plus both 9706490 and 9413827 (which is included in GI PSU2).

For platforms that do not have 11.2.0.1.2 GI PSU, then you can apply patch 9413827 on 11.2.0.1.0.




3). If $GI_HOME is on version 11.2.0.1.1 (GI PSU1) (this is rare since GI PSU1 was only released for Linux platforms and was quite old).

Action: You can rollback GI PSU1 then apply Patch:9706490 on version 11.2.0.1.2 if your platform has 11.2.0.1.2 GI PSU. If your platform does not have 11.2.0.1.2GI PSU, then apply patch 9413827.

Modify crsconfig_lib.pm

After patch is applied, modify $11.2.0.2_GI_HOME/crs/install/crsconfig_lib.pm:

Before the change:
# grep for bugs 9655006 or 9413827
cmdout = grep(/(9655006|9413827)/, @output);

After the change:
# grep for bugs 9655006 or 9413827 or 9706490
cmdout = grep(/(9655006|9413827|9706490)/, @output);

This would prevent rootupgrade.sh from failing when it validates the pre-requsite patches.

Upgrade to 11.2.0.3
If you are upgrading from previous version to 11.2.0.3, notice that upgrade process to 11.2.0.3 has been improved that CVU is used to verify the Oracle prerequisite patches before starting the upgrade. The source home patches are checked during the installer OUI. The target home patches are checked during rootupgrade.sh. cluvfy -upgrade command line can also be used to check the upgrade pre-requsite check.

For example, if cluvfy says it's missing patch for bug 9413827, then you need to apply patch 9413827 (instead of patch 9706490), because cluvfy on some platform only check for patch 9413827.

Maclean Liu(刘相兵

Upgrade to 11.2.0.3
If you are upgrading from previous version to 11.2.0.3, notice that upgrade process to 11.2.0.3 has been improved that CVU is used to verify the Oracle prerequisite patches before starting the upgrade. The source home patches are checked during the installer OUI. The target home patches are checked during rootupgrade.sh. cluvfy -upgrade command line can also be used to check the upgrade pre-requsite check.

For example, if cluvfy says it's missing patch for bug 9413827, then you need to apply patch 9413827 (instead of patch 9706490), because cluvfy on some platform only check for patch 9413827.


请参考
http://www.oracledatabase12g.com ... 2-0-2-in-linux.html

先在11.2.0.1 GI HOME的基础上 apply patch 9413827

若遇到问题 可以尝试 ：

$CRS_HOME/bin/crsctl stop crs -f
$CRS_HOME/bin/crsctl start crs -excl

独占重启CRS

riverchild

crsctl stop crs -f也不用不了！
[root@rac2 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac2'
CRS-2679: Attempting to clean 'ora.VOTE.dg' on 'rac2'
ORA-15154: cluster rolling upgrade incomplete
ORA-15154: cluster rolling upgrade incomplete
CRS-2680: Clean of 'ora.VOTE.dg' on 'rac2' failed
CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'rac2' has failed
CRS-2675: Stop of 'ora.crsd' on 'rac2' failed
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has failed
CRS-4687: Shutdown command has completed with error(s).
CRS-4000: Command Stop failed, or completed with errors.
[root@rac2 bin]#