apply 11.2.0.3.2 PSU p13696251 RAC grid infrastructure

Maclean Liu(刘相兵

apply 11.2.0.3.2 PSU p13696251 RAC grid infrastructure


1. 下载相关的PSU补丁包 和11.2上 最新版本的OPATCH  如X86-64 Linux上的

p6880880_112000_Linux-x86-64.zip                   ==》最新的OPATCH
p13696251_112030_Linux-x86-64.zip                 ==》11.2.0.3.2 PSU


2. 解压 PSU 和 Opatch压缩包 ，并将最新的OPatch 覆盖到$CRS_HOME/OPatch (FOR GI) 和 $ORACLE_HOME/OPatch (FOR GI )

su - grid
[grid@vrh1 ~]$ unzip p6880880_112000_Linux-x86-64.zip

[grid@vrh1 ~]$ mv $CRS_HOME/OPatch $CRS_HOME/OPatch_old
[grid@vrh1 ~]$ cp -r  OPatch $CRS_HOME


su - oracle

[oracle@vrh1 tmp]$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_old
[oracle@vrh1 tmp]$ cp -r OPatch $ORACLE_HOME


unzip p13696251_112030_Linux-x86-64.zip



3. GI补丁可以rolling upgrade滚动升级，在一个节点上执行升级 时其他节点DB Instance可以online

关闭 当前打PSU节点的所有数据库实例：

su - oracle
$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>



4. 切换到root用户 执行<GI_HOME>/crs/install/rootcrs.pl -unlock 脚本

su - root

[root@vrh1 ~]# $CRS_HOME/crs/install/rootcrs.pl -unlock
Using configuration parameter file: /g01/11.2.0/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'vrh1'
CRS-2673: Attempting to stop 'ora.crsd' on 'vrh1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'vrh1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'vrh1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'vrh1'
CRS-2673: Attempting to stop 'ora.SYSTEMDG.dg' on 'vrh1'
CRS-2673: Attempting to stop 'ora.BACKUPDG.dg' on 'vrh1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'vrh1'
CRS-2673: Attempting to stop 'ora.LSN_MACLEAN.lsnr' on 'vrh1'
CRS-2673: Attempting to stop 'ora.vrh2.vip' on 'vrh1'
CRS-2677: Stop of 'ora.vrh2.vip' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.LSN_MACLEAN.lsnr' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.vrh1.vip' on 'vrh1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'vrh1'
CRS-2677: Stop of 'ora.vrh1.vip' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.BACKUPDG.dg' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.SYSTEMDG.dg' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'vrh1'
CRS-2677: Stop of 'ora.asm' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'vrh1'
CRS-2677: Stop of 'ora.ons' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'vrh1'
CRS-2677: Stop of 'ora.net1.network' on 'vrh1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'vrh1' has completed
CRS-2677: Stop of 'ora.crsd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'vrh1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'vrh1'
CRS-2673: Attempting to stop 'ora.evmd' on 'vrh1'
CRS-2673: Attempting to stop 'ora.asm' on 'vrh1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'vrh1'
CRS-2677: Stop of 'ora.crf' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'vrh1' succeeded
CRS-2677: Stop of 'ora.asm' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'vrh1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'vrh1'
CRS-2677: Stop of 'ora.cssd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'vrh1'
CRS-2677: Stop of 'ora.gipcd' on 'vrh1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'vrh1'
CRS-2677: Stop of 'ora.gpnpd' on 'vrh1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'vrh1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /g01/11.2.0/grid



执行完成以上脚本后 $CRS_HOME目录的属性 会被修改 为 GI home Owner所拥有 ， 才可以继续使用GI home owner执行PSU 升级


[root@vrh1 ~]# ls -ld $CRS_HOME
drwxr-xr-x 66 grid oinstall 4096 Nov  7 19:46 /g01/11.2.0/grid

[root@vrh1 ~]# ls -ld $CRS_HOME/install
drwxr-xr-x 8 grid oinstall 4096 Nov  7 20:02 /g01/11.2.0/grid/install

[root@vrh1 ~]# ls -ld $CRS_HOME/.patch_storage
ls: /g01/11.2.0/grid/.patch_storage: No such file or directory
[root@vrh1 ~]#



5. 使用GI home owner用户实际执行PSU升级，包括 2次opatch apply

opatch napply -oh $CRS_HOME  -local <UNZIP_LOCATION>/13696251
opatch napply -oh $CRS_HOME  -local <UNZIP_LOCATION>/13696216





[grid@vrh1 tmp]$ opatch napply -oh $CRS_HOME  -local <UNZIP_LOCATION>/13696251
Oracle Interim Patch Installer version 11.2.0.3.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /g01/11.2.0/grid
Central Inventory : /g01/oraInventory
   from           : /g01/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.0
OUI version       : 11.2.0.3.0
Log file location : /g01/11.2.0/grid/cfgtoollogs/opatch/opatch2012-05-05_08-22-48AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   13696251  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/g01/11.2.0/grid')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '13696251' to OH '/g01/11.2.0/grid'

Patching component oracle.crs, 11.2.0.3.0...
Patching component oracle.usm, 11.2.0.3.0...

Verifying the update...

OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
ins_srvm.mk:68: warning: overriding commands for target `libsrvm11.so'
ins_srvm.mk:31: warning: ignoring old commands for target `libsrvm11.so'
ins_srvm.mk:71: warning: overriding commands for target `libsrvmocr11.so'
ins_srvm.mk:34: warning: ignoring old commands for target `libsrvmocr11.so'
ins_srvm.mk:74: warning: overriding commands for target `libsrvmhas11.so'
ins_srvm.mk:37: warning: ignoring old commands for target `libsrvmhas11.so'


Patch 13696251 successfully applied.
OPatch Session completed with warnings.
Log file location: /g01/11.2.0/grid/cfgtoollogs/opatch/opatch2012-05-05_08-22-48AM_1.log

OPatch completed with warnings.



[grid@vrh1 tmp]$ opatch napply -oh $CRS_HOME  -local 13696216
Oracle Interim Patch Installer version 11.2.0.3.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /g01/11.2.0/grid
Central Inventory : /g01/oraInventory
   from           : /g01/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.0
OUI version       : 11.2.0.3.0
Log file location : /g01/11.2.0/grid/cfgtoollogs/opatch/opatch2012-05-05_08-30-33AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   13343438  13696216  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/g01/11.2.0/grid')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '13343438' to OH '/g01/11.2.0/grid'

Patching component oracle.rdbms.rsf, 11.2.0.3.0...

Patching component oracle.rdbms, 11.2.0.3.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.3.0...

Verifying the update...
Applying sub-patch '13696216' to OH '/g01/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.3.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.4 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.3.0...

Patching component oracle.rdbms, 11.2.0.3.0...

Patching component oracle.sdo.locator, 11.2.0.3.0...

Verifying the update...
Composite patch 13696216 successfully applied.
Log file location: /g01/11.2.0/grid/cfgtoollogs/opatch/opatch2012-05-05_08-30-33AM_1.log

OPatch succeeded.


6. 切换到RDBMS home owner ， 并执行DB部分的升级任务

<UNZIP_LOCATION>/13696251/custom/server/13696251/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME
/tmp/13696251/custom/server/13696251/custom/scripts/prepatch.sh completed successfully.



opatch napply -oh $ORACLE_HOME -local  <UNZIP_LOCATION>/13696251/custom/server/13696251


Oracle Interim Patch Installer version 11.2.0.3.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /s01/orabase/product/11.2.0/dbhome_1
Central Inventory : /g01/oraInventory
   from           : /s01/orabase/product/11.2.0/dbhome_1/oraInst.loc
OPatch version    : 11.2.0.3.0
OUI version       : 11.2.0.3.0
Log file location : /s01/orabase/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2012-05-05_08-48-04AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   13696251  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/s01/orabase/product/11.2.0/dbhome_1')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '13696251' to OH '/s01/orabase/product/11.2.0/dbhome_1'

Patching component oracle.rdbms, 11.2.0.3.0...

Verifying the update...

OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
ins_srvm.mk:68: warning: overriding commands for target `libsrvm11.so'
ins_srvm.mk:31: warning: ignoring old commands for target `libsrvm11.so'
ins_srvm.mk:71: warning: overriding commands for target `libsrvmocr11.so'
ins_srvm.mk:34: warning: ignoring old commands for target `libsrvmocr11.so'
ins_srvm.mk:74: warning: overriding commands for target `libsrvmhas11.so'
ins_srvm.mk:37: warning: ignoring old commands for target `libsrvmhas11.so'


Patch 13696251 successfully applied.
OPatch Session completed with warnings.
Log file location: /s01/orabase/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2012-05-05_08-48-04AM_1.log

OPatch completed with warnings.



[oracle@vrh1 tmp]$ opatch apply -oh $ORACLE_HOME -local   <UNZIP_LOCATION>/13696216


Oracle Interim Patch Installer version 11.2.0.3.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /s01/orabase/product/11.2.0/dbhome_1
Central Inventory : /g01/oraInventory
   from           : /s01/orabase/product/11.2.0/dbhome_1/oraInst.loc
OPatch version    : 11.2.0.3.0
OUI version       : 11.2.0.3.0
Log file location : /s01/orabase/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2012-05-05_08-49-24AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   13343438  13696216  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/s01/orabase/product/11.2.0/dbhome_1')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '13343438' to OH '/s01/orabase/product/11.2.0/dbhome_1'

Patching component oracle.rdbms.rsf, 11.2.0.3.0...

Patching component oracle.rdbms, 11.2.0.3.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.3.0...

Verifying the update...
Applying sub-patch '13696216' to OH '/s01/orabase/product/11.2.0/dbhome_1'

Patching component oracle.rdbms.rsf, 11.2.0.3.0...

Patching component oracle.rdbms, 11.2.0.3.0...

Patching component oracle.sdo.locator, 11.2.0.3.0...

Patching component oracle.sysman.console.db, 11.2.0.3.0...

Patching component oracle.sysman.oms.core, 10.2.0.4.4...

Verifying the update...
Composite patch 13696216 successfully applied.
Log file location: /s01/orabase/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2012-05-05_08-49-24AM_1.log

OPatch succeeded.





[oracle@vrh1 tmp]$ <UNZIP_LOCATION>/13696251/custom/server/13696251/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME
Reading /s01/orabase/product/11.2.0/dbhome_1/install/params.ora..
Reading /s01/orabase/product/11.2.0/dbhome_1/install/params.ora..
Parsing file /s01/orabase/product/11.2.0/dbhome_1/bin/racgwrap
Parsing file /s01/orabase/product/11.2.0/dbhome_1/bin/srvctl
Parsing file /s01/orabase/product/11.2.0/dbhome_1/bin/srvconfig
Parsing file /s01/orabase/product/11.2.0/dbhome_1/bin/cluvfy
Verifying file /s01/orabase/product/11.2.0/dbhome_1/bin/racgwrap
Verifying file /s01/orabase/product/11.2.0/dbhome_1/bin/srvctl
Verifying file /s01/orabase/product/11.2.0/dbhome_1/bin/srvconfig
Verifying file /s01/orabase/product/11.2.0/dbhome_1/bin/cluvfy
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/bin/racgwrap
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/bin/srvctl
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/bin/srvconfig
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/bin/cluvfy
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/bin/diskmon.bin
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/bin/lsnodes
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/bin/osdbagrp
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/bin/rawutl
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/srvm/admin/ractrans
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/srvm/admin/getcrshome
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/bin/gnsd
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/bin/crsdiag.pl
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/libhasgen11.so
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/libclsra11.so
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/libdbcfg11.so
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/libocr11.so
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/libocrb11.so
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/libocrutl11.so
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/libuini11.so
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/librdjni11.so
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/libgns11.so
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/libgnsjni11.so
Reapplying file permissions on /s01/orabase/product/11.2.0/dbhome_1/lib/libagfw11.so


7. 切换到root 用户 执行相关脚本：

[root@vrh1 ~]# $CRS_HOME/rdbms/install/rootadd_rdbms.sh



[root@vrh1 ~]# $CRS_HOME/crs/install/rootcrs.pl -patch
Using configuration parameter file: /g01/11.2.0/grid/crs/install/crsconfig_params
CRS-4123: Oracle High Availability Services has been started.


[root@vrh1 ~]# crsctl query  crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]


[grid@vrh1 tmp]$ ls -ld $CRS_HOME
drwxr-xr-x 70 root oinstall 4096 May  5 08:31 /g01/11.2.0/grid

GI HOME 的拥有者在rootcrs.pl -patch 运行后变回root



8. 执行完成以上任务后 该节点就已经完成 11.2.0.3. 2 GI PSU的安装了 ，可以通过以下命令确认：


[grid@vrh1 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /g01/11.2.0/grid
Central Inventory : /g01/oraInventory
   from           : /g01/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.0
OUI version       : 11.2.0.3.0
Log file location : /g01/11.2.0/grid/cfgtoollogs/opatch/opatch2012-05-05_08-57-18AM_1.log

Lsinventory Output file location : /g01/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2012-05-05_08-57-18AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Grid Infrastructure                                           11.2.0.3.0
There are 1 products installed in this Oracle Home.


Interim patches (2) :

Patch  13696216     : applied on Sat May 05 08:31:09 EDT 2012
Unique Patch ID:  14600705
Patch description:  "Database Patch Set Update : 11.2.0.3.2 (13696216)"
   Created on 3 Apr 2012, 22:02:51 hrs PST8PDT
Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
   Bugs fixed:
     13070939, 13035804, 10350832, 13632717, 13041324, 12919564, 13420224
     13742437, 12861463, 12834027, 13742438, 13332439, 13036331, 13499128
     12998795, 12829021, 13492735, 9873405, 13742436, 13503598, 12960925
     12718090, 13742433, 12662040, 9703627, 12905058, 12938841, 13742434
     12849688, 12950644, 13362079, 13742435, 12620823, 12917230, 12845115
     12656535, 12764337, 13354082, 12588744, 11877623, 12612118, 12847466
     13742464, 13528551, 12894807, 13343438, 12582664, 12780983, 12748240
     12797765, 12780098, 13696216, 12923168, 13466801, 13772618, 11063191, 13554409

Patch  13696251     : applied on Sat May 05 08:27:06 EDT 2012
Unique Patch ID:  14600705
Patch description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.2 (13696251)"
   Created on 5 Apr 2012, 07:21:52 hrs PST8PDT
   Bugs fixed:
     13696251, 13348650, 12659561, 13039908, 13036424, 12794268, 13011520
     13569812, 12758736, 13077654, 13001901, 13430715, 12538907, 13066371
     12594616, 12897651, 12897902, 12896850, 12726222, 12829429, 12728585
     13079948, 12876314, 13090686, 12925041, 12995950, 13251796, 12398492
     12848480, 13652088, 12990582, 12975811, 12917897, 13082238, 12947871
     13037709, 13371153, 12878750, 11772838, 13058611, 13001955, 11836951
     12965049, 13440962, 12765467, 13425727, 12885323, 12784559, 13332363
     13074261, 12971251, 12857064, 13396284, 12899169, 13111013, 12867511
     12639013, 13085732, 12829917, 12934171, 12849377, 12349553, 12914824
     12730342, 13334158, 12950823, 13355963, 13531373, 13002015, 13024624
     12791719, 13886023, 13019958, 13255295, 12810890, 12782756, 13502441
     12873909, 13243172, 12820045, 12842804, 13045518, 12765868, 12772345
     12823838, 13345868, 12823042, 12932852, 12825835, 12695029, 13146560
     13038806, 13263435, 13025879, 13410987, 13396356, 12827493, 13637590
     13247273, 13258062, 12834777, 13068077




Rac system comprising of multiple nodes
  Local node = vrh1
  Remote node = vrh2

--------------------------------------------------------------------------------

OPatch succeeded.



7.  启动打完PSU节点上的DB instance:
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name>


8. 在其他节点上重复以上步骤

ricky

你的安装步骤不错，细致~参看你的我成功安装了，readme太复杂，我下来好好看看readme。学习了

ricky

我在readme里面看到还有：
The following steps load modified SQL files into the database. For an Oracle RAC or Oracle RAC One Node environment, perform these steps on only one node.

For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script as follows:

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT
The catbundle.sql execution is reflected in the dba_registry_history view by a row associated with bundle series PSU.

For information about the catbundle.sql script, see My Oracle Support Note 605795.1 Introduction to Oracle Database catbundle.sql.

Check the following log files in $ORACLE_BASE/cfgtoollogs/catbundle for any errors:

catbundle_PSU_<database SID>_APPLY_<TIMESTAMP>.log
catbundle_PSU_<database SID>_GENERATE_<TIMESTAMP>.log
where TIMESTAMP is of the form YYYYMMMDD_HH_MM_SS. If there are errors, refer to Section 3, "Known Issues".