Oracle数据库数据恢复、性能优化»论坛 › Oracle › Oracle数据库管理 › AIX，11G R2 RAC的PSU升级CRS,CSS服务不能启动 ...

KFC

0 积分	0 好友	5 主题

发消息

[ORA-600/ORA-7445] AIX，11G R2 RAC的PSU升级CRS,CSS服务不能启动

1^#

发表于 2013-2-28 22:19:32 | 查看: 9851| 回复: 13

本帖最后由 KFC 于 2013-2-28 22:24 编辑

1.系统因为需要进行升级由原来的11.2.0.2升到11.2.0.2.3
2.平台：AIX6 ,11G RAC 11.2.0.2(2个节点）
3.升级按作MANUAL方式，按步骤操作进行，期间除了有警告外，其它一切顺利，警告类似如下
ld: 0711-773 WARNING: Object /mboss/app/grid/lib//libordsdo11.a[mdrt.o], imported symbol kcbstdbz_
      Symbol was expected to be local. Extra instructions
      are being generated to reference the symbol.
ld: 0711-773 WARNING: Object /mboss/app/grid/lib//libordsdo11.a[mdrcnc.o], imported symbol mdrcnccmt_
      Symbol was expected to be local. Extra instructions
ld: 0711-783 WARNING: TOC overflow. TOC size: 68160    Maximum size: 65536
      Extra instructions are being generated for each reference to a TOC
      symbol if the symbol is in the TOC overflow area.
4.在执行到最后一步时比较有这个提示
rootcrs.pl -patch
Using configuration parameter file: /mboss/app/grid/crs/install/crsconfig_params
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
/mboss/app/grid/bin/clsecho[272]: /mboss/app/grid/bin/clsecho.bin: 0403-006 Execute permission denied.
/mboss/app/grid/bin/crsctl[272]: /mboss/app/grid/bin/crsctl.bin: 0403-006 Execute permission denied.
Timed out waiting for the CRS stack to start.
5.对相关文件赋给777权限后再次执行，有个节点执行2次（：）应该是没影响吧），
rootcrs.pl -patch
Using configuration parameter file: /mboss/app/grid/crs/install/crsconfig_params
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9359: The AIX ODM entry for 'acfsctl' was successfully added.
ACFS-9359: The AIX ODM entry for 'advmctl' was successfully added.
ACFS-9359: The AIX ODM entry for 'advmvol' was successfully added.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
Timed out waiting for the CRS stack to start.
6.执行一些命令提示
crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
crsctl check cluster -all
**************************************************************

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
**************************************************************
7.CSS,CRS的日志没有最后更新，也就是比如我20点后运行的报错都没有记录，但OHASD.LOG有最新记录
2013-02-28 21:34:44.960: [UiServer][7711] Closed: remote end failed/disc.
2013-02-28 21:36:07.198: [UiServer][7711] CS(111838990)set Properties ( root,111837f10)
2013-02-28 21:36:07.198: [UiServer][7711] SS(11455a390)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI
_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-02-28 21:36:07.208: [UiServer][7454] {0:0:7} processMessage called
2013-02-28 21:36:07.208: [UiServer][7454] {0:0:7} Container [ Name: SHUTDOWNAPPS
      LOCALE:
      TextMessage[AMERICAN_AMERICA.US7ASCII]
]
2013-02-28 21:36:07.209: [UiServer][7454] {0:0:7} Sending message to PE. ctx= 111838dd0
2013-02-28 21:36:07.209: [UiServer][7454] {0:0:7} Sending command to PE: 4
2013-02-28 21:36:07.210: [ CRSPE][7197] {0:0:7} Processing PE command id=106. Description: [Server Shutdown {} : pass=0 : 113c00490]
2013-02-28 21:36:07.210: [ CRSPE][7197] {0:0:7} Prepared shutdown cmd for: gdgis1
2013-02-28 21:36:07.210: [ CRSPE][7197] {0:0:7} Cannot start shutdown: CRS-2796: The command may not proceed when Cluster Ready Serv
ices is not running

2013-02-28 21:36:07.211: [ CRSPE][7197] {0:0:7} Server [gdgis1] has changed state from [ONLINE] to [LEAVING]
2013-02-28 21:36:07.212: [UiServer][7454] {0:0:7} Container [ Name: ORDER
      MESSAGE:
      TextMessage[CRS-2796: The command may not proceed when Cluster Ready Services is not running]
      MSGTYPE:
      TextMessage[1]
      OBJID:
      TextMessage[gdgis1]
      WAIT:
      TextMessage[0]
]
2013-02-28 21:36:07.212: [  CRSOCR][6940] {0:0:7} Multi Write Batch processing...
2013-02-28 21:36:07.213: [ CRSPE][7197] {0:0:7} PE Command [ Server Shutdown {gdgis1} : pass=1 : 113c00490 ] has completed
2013-02-28 21:36:07.213: [ CRSPE][7197] {0:0:7} Shutdown cmd failed: Server Shutdown {gdgis1} : pass=1 : 113c00490
2013-02-28 21:36:07.213: [ CRSPE][7197] {0:0:7} Server [gdgis1] has changed state from [LEAVING] to [ONLINE]
2013-02-28 21:36:07.215: [ CRSPE][7197] {0:0:7} UI Command [Server Shutdown {gdgis1} : pass=1 : 113c00490] is replying to sender.
2013-02-28 21:36:07.215: [UiServer][7454] {0:0:7} Container [ Name: UI_DATA
      RESULT:
      TextMessage[200]
]
2013-02-28 21:36:07.215: [UiServer][7454] {0:0:7} Done for ctx=111838dd0
2013-02-28 21:36:07.218: [UiServer][7711] Closed: remote end failed/disc.
2013-02-28 21:36:07.373: [  CRSOCR][6940] {0:0:7} Multi Write Batch done.
2013-02-28 21:36:07.373: [  CRSOCR][6940] {0:0:7} Multi Write Batch processing...
2013-02-28 21:36:07.481: [  CRSOCR][6940] {0:0:7} Multi Write Batch done.
2013-02-28 21:39:42.398: [UiServer][7711] CS(113c005d0)set Properties ( root,113c02a30)
2013-02-28 21:39:42.398: [UiServer][7711] SS(11455a390)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI
_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-02-28 21:39:42.409: [UiServer][7454] {0:0:8} processMessage called
2013-02-28 21:39:42.409: [UiServer][7454] {0:0:8} Sending message to PE. ctx= 111837e70
2013-02-28 21:39:42.410: [UiServer][7454] {0:0:8} Sending command to PE: 5
2013-02-28 21:39:42.410: [ CRSPE][7197] {0:0:8} Processing PE command id=107. Description: [Stat Resource : 113c00f70]
2013-02-28 21:39:42.411: [ CRSPE][7197] {0:0:8} Expression Filter : (((NAME == ora.crsd) OR (NAME == ora.cssd)) OR (NAME == ora.evmd
))
2013-02-28 21:39:42.418: [UiServer][7454] {0:0:8} Done for ctx=111837e70
2013-02-28 21:39:42.419: [UiServer][7711] Closed: remote end failed/disc.
2013-02-28 21:40:33.330: [UiServer][7711] CS(113c005d0)set Properties ( root,113c02a30)
2013-02-28 21:40:33.330: [UiServer][7711] SS(11455a390)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI
_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-02-28 21:40:33.340: [UiServer][7454] {0:0:9} processMessage called
2013-02-28 21:40:33.341: [UiServer][7454] {0:0:9} Sending message to PE. ctx= 111837e70
2013-02-28 21:40:33.341: [UiServer][7454] {0:0:9} Sending command to PE: 6
2013-02-28 21:40:33.342: [ CRSPE][7197] {0:0:9} Processing PE command id=108. Description: [Stat Resource : 113c00f70]
2013-02-28 21:40:33.346: [UiServer][7454] {0:0:9} Done for ctx=111837e70
2013-02-28 21:40:33.350: [UiServer][7711] Closed: remote end failed/disc.
2013-02-28 21:41:49.947: [UiServer][7711] CS(113c005d0)set Properties ( root,113c02a30)
2013-02-28 21:41:49.947: [UiServer][7711] SS(11455a390)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI
_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-02-28 21:41:49.957: [UiServer][7454] {0:0:10} processMessage called
2013-02-28 21:41:49.957: [UiServer][7454] {0:0:10} Container [ Name: SHUTDOWNAPPS
      LOCALE:
      TextMessage[AMERICAN_AMERICA.US7ASCII]

分享0

收藏0 回复只看该作者道具举报

KFC

2^#

发表于 2013-2-28 22:19:54

本帖最后由 KFC 于 2013-2-28 22:26 编辑

]
2013-02-28 21:41:49.958: [UiServer][7454] {0:0:10} Sending message to PE. ctx= 1118379d0
2013-02-28 21:41:49.958: [UiServer][7454] {0:0:10} Sending command to PE: 7
2013-02-28 21:41:49.959: [ CRSPE][7197] {0:0:10} Processing PE command id=109. Description: [Server Shutdown {} : pass=0 : 113c00f70
]
2013-02-28 21:41:49.959: [ CRSPE][7197] {0:0:10} Prepared shutdown cmd for: gdgis1
2013-02-28 21:41:49.959: [ CRSPE][7197] {0:0:10} Cannot start shutdown: CRS-2796: The command may not proceed when Cluster Ready Ser
vices is not running

2013-02-28 21:41:49.960: [ CRSPE][7197] {0:0:10} Server [gdgis1] has changed state from [ONLINE] to [LEAVING]
2013-02-28 21:41:49.961: [UiServer][7454] {0:0:10} Container [ Name: ORDER
      MESSAGE:
      TextMessage[CRS-2796: The command may not proceed when Cluster Ready Services is not running]
      MSGTYPE:
      TextMessage[1]
      OBJID:
      TextMessage[gdgis1]
      WAIT:
      TextMessage[0]
]
2013-02-28 21:41:49.961: [ CRSPE][7197] {0:0:10} PE Command [ Server Shutdown {gdgis1} : pass=1 : 113c00f70 ] has completed
2013-02-28 21:41:49.962: [ CRSPE][7197] {0:0:10} Shutdown cmd failed: Server Shutdown {gdgis1} : pass=1 : 113c00f70
2013-02-28 21:41:49.962: [ CRSPE][7197] {0:0:10} Server [gdgis1] has changed state from [LEAVING] to [ONLINE]
2013-02-28 21:41:49.962: [  CRSOCR][6940] {0:0:10} Multi Write Batch processing...
2013-02-28 21:41:49.963: [ CRSPE][7197] {0:0:10} UI Command [Server Shutdown {gdgis1} : pass=1 : 113c00f70] is replying to sender.
2013-02-28 21:41:49.964: [UiServer][7454] {0:0:10} Container [ Name: UI_DATA
8.比较不幸的事，METALINK帐号也过期，没有办法，在此救助！谢谢

回复只看该作者道具举报

Maclean Liu(刘相兵

3^#

发表于 2013-2-28 22:27:32

不行就做 deconfigure 后reconfigure

FYI

C. GI Cluster Deconfigure and Reconfigure

Identify cause of root.sh failure by reviewing logs in $GRID_HOME/cfgtoollogs/crsconfig and $GRID_HOME/log, once cause is identified and problem is fixed, deconfigure and reconfigure with steps below - keep in mind that you will need wait till each step finishes successfully before move to next one:

Step 0: For 11.2.0.2 and above, root.sh is restartable.

Once cause is identified and the problem is fixed, root.sh can be executed again on the failed node. If it succeeds, continue with your planned installation procedure; otherwise as root sequentially execute "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force" and $GRID_HOME/root.sh on local node, if it succeeds, continue with your planned installation procedure, otherwise proceed to next step (Step 1) of the note.

Step 1: As root, run "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force" on all nodes, except the last one.

Step 2: As root, run "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode" on last node. This command will zero out OCR, Voting Disk and the ASM diskgroup for OCR and Voting Disk

Note:

a. Step1 and 2 can be skipped on node(s) where root.sh haven't been executed this time.

b. Step1 and 2 should remove checkpoint file. To verify:

      ls -l $ORACLE_BASE/Clusterware/ckptGridHA_.xml

If it's still there, please remove it manually with "rm" command on all nodes

c. If GPNP profile is different between nodes/setup, clean it up on all nodes as grid user

      $ rm -rf $GRID_HOME/gpnp/*
      $ mkdir -p $GRID_HOME/gpnp/profiles/peer $GRID_HOME/gpnp/wallets/peer $GRID_HOME/gpnp/wallets/prdr $GRID_HOME/gpnp/wallets/pa $GRID_HOME/gpnp/wallets/root

The profile needs to be cleaned up:

      c1. If root.sh is executed concurrently - one should not execute root.sh on any other nodes before it finishes on first node.

      c2. If network info, location of OCR or Voting Disk etc changed after Grid is installed - rare

Step 3: As root, run $GRID_HOME/root.sh on first node

Step 4: As root, run $GRID_HOME/root.sh on all other node(s), except last one.

Step 5: As root, run $GRID_HOME/root.sh on last node.

回复只看该作者道具举报

KFC

4^#

发表于 2013-3-6 11:23:21

本帖最后由 KFC 于 2013-3-6 11:32 编辑

节点1执行：/mboss/app/grid/crs/install/rootcrs.pl -verbose -deconfig -force
[root@gdgis1:/]#/mboss/app/grid/crs/install/rootcrs.pl -verbose -deconfig -force
Using configuration parameter file: /mboss/app/grid/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

ACFS-9200: Supported
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'gdgis1'
CRS-2679: Attempting to clean 'ora.drivers.acfs' on 'gdgis1'
CRS-2679: Attempting to clean 'ora.cssdmonitor' on 'gdgis1'
CRS-2679: Attempting to clean 'ora.gpnpd' on 'gdgis1'
CRS-2679: Attempting to clean 'ora.mdnsd' on 'gdgis1'
CRS-2680: Clean of 'ora.drivers.acfs' on 'gdgis1' failed
CRS-2680: Clean of 'ora.cssdmonitor' on 'gdgis1' failed
CRS-2680: Clean of 'ora.gpnpd' on 'gdgis1' failed
CRS-2680: Clean of 'ora.mdnsd' on 'gdgis1' failedCRS-2799: Failed to shut down resource 'ora.cssdmonitor' on 'gdgis1'
CRS-2799: Failed to shut down resource 'ora.drivers.acfs' on 'gdgis1'
CRS-2799: Failed to shut down resource 'ora.gpnpd' on 'gdgis1'
CRS-2799: Failed to shut down resource 'ora.mdnsd' on 'gdgis1'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'gdgis1' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
You must kill crs processes or reboot the system to properly
cleanup the processes started by Oracle clusterware
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
Successfully deconfigured Oracle clusterware stack on this node

回复只看该作者道具举报

KFC

5^#

发表于 2013-3-6 11:23:58

本帖最后由 KFC 于 2013-3-6 11:28 编辑

2节点(最后节点）[root@gdgis2:/]#/mboss/app/grid/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode
Using configuration parameter file: /mboss/app/grid/crs/install/crsconfig_params
CRS-2679: Attempting to clean 'ora.mdnsd' on 'gdgis2'
CRS-2680: Clean of 'ora.mdnsd' on 'gdgis2' failed
CRS-2679: Attempting to clean 'ora.gpnpd' on 'gdgis2'
CRS-2680: Clean of 'ora.gpnpd' on 'gdgis2' failed
CRS-2679: Attempting to clean 'ora.cssdmonitor' on 'gdgis2'
CRS-2680: Clean of 'ora.cssdmonitor' on 'gdgis2' failed
CRS-5802: Unable to start the agent process
CRS-4000: Command Start failed, or completed with errors.
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

ACFS-9200: Supported
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-4689: Oracle Clusterware is already stopped
CRS-4000: Command Stop failed, or completed with errors.
CRS-2679: Attempting to clean 'ora.mdnsd' on 'gdgis2'
CRS-2680: Clean of 'ora.mdnsd' on 'gdgis2' failed
CRS-2679: Attempting to clean 'ora.gpnpd' on 'gdgis2'
CRS-2680: Clean of 'ora.gpnpd' on 'gdgis2' failed
CRS-2679: Attempting to clean 'ora.cssdmonitor' on 'gdgis2'
CRS-2680: Clean of 'ora.cssdmonitor' on 'gdgis2' failed
CRS-5802: Unable to start the agent process
CRS-4000: Command Start failed, or completed with errors.
Unable to communicate with the Cluster Synchronization Services daemon.
CRS-4000: Command Delete failed, or completed with errors.
crsctl delete for vds in DATA ... failed
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'gdgis2'
CRS-2679: Attempting to clean 'ora.cssdmonitor' on 'gdgis2'
CRS-2679: Attempting to clean 'ora.gpnpd' on 'gdgis2'
CRS-2679: Attempting to clean 'ora.mdnsd' on 'gdgis2'
CRS-2680: Clean of 'ora.cssdmonitor' on 'gdgis2' failed
CRS-2680: Clean of 'ora.gpnpd' on 'gdgis2' failed
CRS-2680: Clean of 'ora.mdnsd' on 'gdgis2' failed
CRS-2799: Failed to shut down resource 'ora.cssdmonitor' on 'gdgis2'
CRS-2799: Failed to shut down resource 'ora.gpnpd' on 'gdgis2'
CRS-2799: Failed to shut down resource 'ora.mdnsd' on 'gdgis2'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'gdgis2' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
Successfully deconfigured Oracle clusterware stack on this node

回复只看该作者道具举报

KFC

6^#

发表于 2013-3-6 11:24:32

本帖最后由 KFC 于 2013-3-6 11:29 编辑

--------------

回复只看该作者道具举报

KFC

7^#

发表于 2013-3-6 11:24:59

本帖最后由 KFC 于 2013-3-6 11:27 编辑

1节点root.sh[root@gdgis1:/]#/mboss/app/grid/root.sh
Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /mboss/app/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /mboss/app/grid/crs/install/crsconfig_params
User grid has the required capabilities to run CSSD in realtime mode
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9359: The AIX ODM entry for 'acfsctl' was successfully added.
ACFS-9359: The AIX ODM entry for 'advmctl' was successfully added.
ACFS-9359: The AIX ODM entry for 'advmvol' was successfully added.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-2672: Attempting to start 'ora.mdnsd' on 'gdgis1'
CRS-2674: Start of 'ora.mdnsd' on 'gdgis1' failed
CRS-2679: Attempting to clean 'ora.mdnsd' on 'gdgis1'
CRS-2678: 'ora.mdnsd' on 'gdgis1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-2672: Attempting to start 'ora.gpnpd' on 'gdgis1'
CRS-2674: Start of 'ora.gpnpd' on 'gdgis1' failed
CRS-2679: Attempting to clean 'ora.gpnpd' on 'gdgis1'
CRS-2678: 'ora.gpnpd' on 'gdgis1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'gdgis1'
CRS-2674: Start of 'ora.cssdmonitor' on 'gdgis1' failed
CRS-2679: Attempting to clean 'ora.cssdmonitor' on 'gdgis1'
CRS-2678: 'ora.cssdmonitor' on 'gdgis1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-5802: Unable to start the agent process
CRS-4000: Command Start failed, or completed with errors.
The exlusive mode cluster start failed, see Clusterware alert log for more information
Initial cluster configuration failed. See /mboss/app/grid/cfgtoollogs/crsconfig/rootcrs_gdgis1.log for details

回复只看该作者道具举报

KFC

8^#

发表于 2013-3-6 11:26:26

本帖最后由 KFC 于 2013-3-6 11:31 编辑

2节点root.sh[root@gdgis2:/]#/mboss/app/grid/root.shRunning Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /mboss/app/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /mboss/app/grid/crs/install/crsconfig_params
User grid has the required capabilities to run CSSD in realtime mode
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9359: The AIX ODM entry for 'acfsctl' was successfully added.
ACFS-9359: The AIX ODM entry for 'advmctl' was successfully added.
ACFS-9359: The AIX ODM entry for 'advmvol' was successfully added.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-2672: Attempting to start 'ora.mdnsd' on 'gdgis2'
CRS-2674: Start of 'ora.mdnsd' on 'gdgis2' failedCRS-2679: Attempting to clean 'ora.mdnsd' on 'gdgis2'
CRS-2678: 'ora.mdnsd' on 'gdgis2' has experienced an unrecoverable failureCRS-0267: Human intervention required to resume its availability.
CRS-2672: Attempting to start 'ora.gpnpd' on 'gdgis2'
CRS-2674: Start of 'ora.gpnpd' on 'gdgis2' failed
CRS-2679: Attempting to clean 'ora.gpnpd' on 'gdgis2'
CRS-2678: 'ora.gpnpd' on 'gdgis2' has experienced an unrecoverable failureCRS-0267: Human intervention required to resume its availability.
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'gdgis2'
CRS-2674: Start of 'ora.cssdmonitor' on 'gdgis2' failed
CRS-2679: Attempting to clean 'ora.cssdmonitor' on 'gdgis2'
CRS-2678: 'ora.cssdmonitor' on 'gdgis2' has experienced an unrecoverable failureCRS-0267: Human intervention required to resume its availability.
CRS-5802: Unable to start the agent processCRS-4000: Command Start failed, or completed with errors.
The exlusive mode cluster start failed, see Clusterware alert log for more information
Initial cluster configuration failed. See /mboss/app/grid/cfgtoollogs/crsconfig/rootcrs_gdgis2.log for details

回复只看该作者道具举报

KFC

9^#

发表于 2013-3-6 11:34:11

[root@gdgis1:/]#crsctl stat resource -t -init
--------------------------------------------------------------------------------
NAME          TARGET  STATE       SERVER                STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
   1       OFFLINE OFFLINE
ora.cluster_interconnect.haip
   1       OFFLINE OFFLINE
ora.crsd
   1       OFFLINE OFFLINE
ora.cssd
   1       ONLINE  OFFLINE
ora.cssdmonitor
   1       ONLINE  UNKNOWN    gdgis1
ora.ctssd
   1       OFFLINE OFFLINE
ora.diskmon
   1       ONLINE  OFFLINE
ora.drivers.acfs
   1       OFFLINE OFFLINE
ora.evmd
   1       OFFLINE OFFLINE
ora.gipcd
   1       OFFLINE OFFLINE
ora.gpnpd
   1       OFFLINE UNKNOWN    gdgis1
ora.mdnsd
   1       OFFLINE UNKNOWN    gdgis1

-----------------------------------------------------------------------------------------------
[root@gdgis2:/]#crsctl stat resource -t -init
--------------------------------------------------------------------------------
NAME          TARGET  STATE       SERVER                STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
   1       OFFLINE OFFLINE
ora.cluster_interconnect.haip
   1       OFFLINE OFFLINE
ora.crsd
   1       OFFLINE OFFLINE
ora.cssd
   1       ONLINE  OFFLINE
ora.cssdmonitor
   1       ONLINE  UNKNOWN    gdgis2
ora.ctssd
   1       OFFLINE OFFLINE
ora.diskmon
   1       ONLINE  OFFLINE
ora.drivers.acfs
   1       OFFLINE OFFLINE
ora.evmd
   1       OFFLINE OFFLINE
ora.gipcd
   1       OFFLINE OFFLINE
ora.gpnpd
   1       OFFLINE UNKNOWN    gdgis2
ora.mdnsd
   1       OFFLINE UNKNOWN    gdgis2

回复只看该作者道具举报

KFC

10^#

发表于 2013-3-6 11:38:12

小结下
[root@gdgis1:/]#ps -ef |grep init.ohasd
root 12845236 1 0 11:18:12 - 0:00 /bin/sh /etc/init.ohasd run
1.OHASD进程在运行，下边衍生出来的代理进程启动不了,其它后续进程也就无法启动
2.之前在操作deconfig和root.sh后做了补丁程序的回滚，但还是报现在类似的错
3.现在想确认下的是能否通过deinstall集群和DB软件再重新安装回去来解决问题？这个方案

回复只看该作者道具举报

KFC

11^#

发表于 2013-3-6 11:40:33

ohasd日志
gdgis1-> tail -300 ohasd.log
      TextMessage[1]
      LOCALE:
      TextMessage[AMERICAN_AMERICA.US7ASCII]
      QUEUE_TAG:
      TextMessage[1]
      RESOURCE:
      TextMessage[ora.gpnpd]
]
2013-03-06 11:19:30.638: [UiServer][7454] {0:0:35} Sending message to PE. ctx= 1147fac90
2013-03-06 11:19:30.638: [UiServer][7454] {0:0:35} Sending command to PE: 33
2013-03-06 11:19:30.639: [ CRSPE][7197] {0:0:35} Cmd : 1148b52d0 : flags: FORCE_TAG | QUEUE_TAG
2013-03-06 11:19:30.639: [ CRSPE][7197] {0:0:35} Processing PE command id=134. Description: [Stop Resource : 1148b52d0]
2013-03-06 11:19:30.642: [ CRSPE][7197] {0:0:35} RI [ora.gpnpd 1 1] new target state: [OFFLINE] old value: [ONLINE]
2013-03-06 11:19:30.642: [  CRSOCR][6940] {0:0:35} Multi Write Batch processing...
2013-03-06 11:19:30.643: [ CRSPE][7197] {0:0:35} RI [ora.gpnpd 1 1] new internal state: [CLEANING] old value: [STABLE]
2013-03-06 11:19:30.644: [ CRSPE][7197] {0:0:35} Sending message to agfw: id = 149
2013-03-06 11:19:30.644: [ AGFW][5912] {0:0:35} Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.gpnpd 1 1] ID 4100:149
2013-03-06 11:19:30.644: [ CRSPE][7197] {0:0:35} CRS-2679: Attempting to clean 'ora.gpnpd' on 'gdgis1'

2013-03-06 11:19:30.644: [ AGFW][5912] {0:0:35} Starting the agent: /mboss/app/grid/bin/oraagent with user id: grid and incarnation:7
2013-03-06 11:19:30.671: [ AGFW][5912] {0:0:35} Created alert : (:CRSAGF00123:) :  Failed to start the agent process: /mboss/app/grid/bin/oraagent Category: -2 Operation: waitpid Loc: usrgetgrp12 OS error: 0 Other : child returned 232
2013-03-06 11:19:30.671: [ AGFW][5912] {0:0:35} Created alert : (:CRSAGF00126:) :  Agent start failed
2013-03-06 11:19:30.671: [ AGFW][5912] {0:0:35} Could not forward message [RESOURCE_CLEAN[ora.gpnpd 1 1] ID 4100:149] to agent. /mboss/app/grid/bin/oraagent_grid is not running
2013-03-06 11:19:30.672: [ AGFW][5912] {0:0:35} Starting the agent: /mboss/app/grid/bin/oraagent with user id: grid and incarnation:8
2013-03-06 11:19:30.696: [ AGFW][5912] {0:0:35} Created alert : (:CRSAGF00123:) :  Failed to start the agent process: /mboss/app/grid/bin/oraagent Category: -2 Operation: waitpid Loc: usrgetgrp12 OS error: 0 Other : child returned 232
2013-03-06 11:19:30.696: [ AGFW][5912] {0:0:35} Created alert : (:CRSAGF00126:) :  Agent start failed
2013-03-06 11:19:30.696: [ AGFW][5912] {0:0:35} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_CLEAN[ora.gpnpd 1 1] ID 4100:149
2013-03-06 11:19:30.699: [  CRSOCR][6940] {0:0:35} Multi Write Batch done.
2013-03-06 11:19:30.846: [UiServer][7454] {0:0:35} Container [ Name: ORDER
      MESSAGE:
      TextMessage[CRS-2679: Attempting to clean 'ora.gpnpd' on 'gdgis1']
      MSGTYPE:
      TextMessage[3]
      OBJID:
      TextMessage[ora.gpnpd]
      WAIT:
      TextMessage[0]
]
2013-03-06 11:19:30.846: [ CRSPE][7197] {0:0:35} Received reply to action [Clean] message ID: 149
2013-03-06 11:19:30.846: [ CRSPE][7197] {0:0:35} RI [ora.gpnpd 1 1] new internal state: [STABLE] old value: [CLEANING]
2013-03-06 11:19:30.847: [ CRSPE][7197] {0:0:35} Fatal Error from AGFW Proxy: Unable to start the agent process
2013-03-06 11:19:30.847: [ CRSPE][7197] {0:0:35} CRS-2680: Clean of 'ora.gpnpd' on 'gdgis1' failed

2013-03-06 11:19:30.848: [ CRSPE][7197] {0:0:35} Sequencer for [ora.gpnpd 1 1] has completed with error: CRS-5802: Unable to start the agent process

2013-03-06 11:19:30.850: [ CRSPE][7197] {0:0:35} PE Command [ Stop Resource : 1148b52d0 ] has completed
2013-03-06 11:19:30.851: [UiServer][7454] {0:0:35} Container [ Name: ORDER
      MESSAGE:
      TextMessage[CRS-2680: Clean of 'ora.gpnpd' on 'gdgis1' failed]
      MSGTYPE:
      TextMessage[1]
      OBJID:
      TextMessage[ora.gpnpd]
      WAIT:
      TextMessage[0]
]
2013-03-06 11:19:30.851: [ AGFW][5912] {0:0:35} Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:150
2013-03-06 11:19:30.851: [ AGFW][5912] {0:0:35} Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:150
2013-03-06 11:19:30.852: [UiServer][7454] {0:0:35} Container [ Name: ORDER
      MESSAGE:
      TextMessage[CRS-5802: Unable to start the agent process]
      MSGTYPE:
      TextMessage[1]
      OBJID:
      TextMessage[ora.gpnpd]
      WAIT:
      TextMessage[0]
]
2013-03-06 11:19:30.852: [ CRSPE][7197] {0:0:35} UI Command [Stop Resource : 1148b52d0] is replying to sender.
2013-03-06 11:19:30.853: [UiServer][7454] {0:0:35} Container [ Name: UI_DATA
      ora.gpnpd:
      TextMessage[216]
]
2013-03-06 11:19:30.853: [UiServer][7454] {0:0:35} Done for ctx=1147fac90
2013-03-06 11:19:30.861: [UiServer][7711] Closed: remote end failed/disc.
2013-03-06 11:19:31.277: [UiServer][7711] CS(113c00310)set Properties ( root,113bff7b0)
2013-03-06 11:19:31.277: [UiServer][7711] SS(11455a290)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-03-06 11:19:31.287: [UiServer][7454] {0:0:36} processMessage called
2013-03-06 11:19:31.288: [UiServer][7454] {0:0:36} Container [ Name: UI_CHECK
      CLIENT:
      TextMessage[]
      DELAY_SECS:
      TextMessage[0]
      LOCALE:
      TextMessage[AMERICAN_AMERICA.US7ASCII]
      RESOURCE:
      TextMessage[ora.gpnpd]
]
2013-03-06 11:19:31.288: [UiServer][7454] {0:0:36} Sending message to PE. ctx= 113c007d0
2013-03-06 11:19:31.288: [UiServer][7454] {0:0:36} Sending command to PE: 34
2013-03-06 11:19:31.289: [ CRSPE][7197] {0:0:36} Processing PE command id=135. Description: [Check Resource : 1148b52d0]
2013-03-06 11:19:31.290: [ AGFW][5912] {0:0:36} Agfw Proxy Server received the message: RESOURCE_PROBE[ora.gpnpd 1 1] ID 4097:153
2013-03-06 11:19:31.290: [ CRSPE][7197] {0:0:36} PE Command [ Check Resource : 1148b52d0 ] has completed
2013-03-06 11:19:31.290: [ AGFW][5912] {0:0:36} Starting the agent: /mboss/app/grid/bin/oraagent with user id: grid and incarnation:9
2013-03-06 11:19:31.291: [ CRSPE][7197] {0:0:36} UI Command [Check Resource : 1148b52d0] is replying to sender.
2013-03-06 11:19:31.316: [ AGFW][5912] {0:0:36} Created alert : (:CRSAGF00123:) :  Failed to start the agent process: /mboss/app/grid/bin/oraagent Category: -2 Operation: waitpid Loc: usrgetgrp12 OS error: 0 Other : child returned 232
2013-03-06 11:19:31.317: [ AGFW][5912] {0:0:36} Created alert : (:CRSAGF00126:) :  Agent start failed
2013-03-06 11:19:31.317: [ AGFW][5912] {0:0:36} Could not forward message [RESOURCE_PROBE[ora.gpnpd 1 1] ID 4097:153] to agent. /mboss/app/grid/bin/oraagent_grid is not running
2013-03-06 11:19:31.317: [ AGFW][5912] {0:0:36} Starting the agent: /mboss/app/grid/bin/oraagent with user id: grid and incarnation:10
2013-03-06 11:19:31.340: [ AGFW][5912] {0:0:36} Created alert : (:CRSAGF00123:) :  Failed to start the agent process: /mboss/app/grid/bin/oraagent Category: -2 Operation: waitpid Loc: usrgetgrp12 OS error: 0 Other : child returned 232
2013-03-06 11:19:31.341: [ AGFW][5912] {0:0:36} Created alert : (:CRSAGF00126:) :  Agent start failed
2013-03-06 11:19:31.341: [ AGFW][5912] {0:0:36} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_PROBE[ora.gpnpd 1 1] ID 4097:153
2013-03-06 11:19:31.491: [ CRSPE][7197] {0:0:36} No listener for operation [Resource Check: [ora.gpnpd 1 1]]. Skipped sending action failed event
2013-03-06 11:19:31.491: [UiServer][7454] {0:0:36} Container [ Name: UI_DATA
      ora.gpnpd:

回复只看该作者道具举报

KFC

12^#

发表于 2013-3-6 11:41:11

      TextMessage[0]
]
2013-03-06 11:19:31.492: [UiServer][7454] {0:0:36} Done for ctx=113c007d0
2013-03-06 11:19:31.493: [UiServer][7711] Closed: remote end failed/disc.
2013-03-06 11:19:31.919: [UiServer][7711] CS(113c00310)set Properties ( root,113bff7b0)
2013-03-06 11:19:31.919: [UiServer][7711] SS(11455a290)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-03-06 11:19:31.929: [UiServer][7454] {0:0:37} processMessage called
2013-03-06 11:19:31.930: [UiServer][7454] {0:0:37} Sending message to PE. ctx= 114803e90
2013-03-06 11:19:31.930: [UiServer][7454] {0:0:37} Sending command to PE: 35
2013-03-06 11:19:31.931: [ CRSPE][7197] {0:0:37} Processing PE command id=136. Description: [Stat Resource : 1148b52d0]
2013-03-06 11:19:31.936: [UiServer][7454] {0:0:37} Done for ctx=114803e90
2013-03-06 11:19:31.939: [UiServer][7711] Closed: remote end failed/disc.
2013-03-06 11:19:37.381: [UiServer][7711] CS(1148b52d0)set Properties ( root,113bff7b0)
2013-03-06 11:19:37.382: [UiServer][7711] SS(11455a290)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-03-06 11:19:37.392: [UiServer][7454] {0:0:38} processMessage called
2013-03-06 11:19:37.392: [UiServer][7454] {0:0:38} Container [ Name: UI_STOP
      CLIENT:
      TextMessage[]
      CLIENT_PRIMARY_GROUP:
      TextMessage[system]
      LOCALE:
      TextMessage[AMERICAN_AMERICA.US7ASCII]
      QUEUE_TAG:
      TextMessage[1]
      RESOURCE:
      TextMessage[ora.mdnsd]
]
2013-03-06 11:19:37.393: [UiServer][7454] {0:0:38} Sending message to PE. ctx= 113c00390
2013-03-06 11:19:37.393: [UiServer][7454] {0:0:38} Sending command to PE: 36
2013-03-06 11:19:37.393: [ CRSPE][7197] {0:0:38} Cmd : 1148c4b70 : flags: QUEUE_TAG
2013-03-06 11:19:37.394: [ CRSPE][7197] {0:0:38} Processing PE command id=137. Description: [Stop Resource : 1148c4b70]
2013-03-06 11:19:37.396: [ CRSPE][7197] {0:0:38} RI [ora.mdnsd 1 1] new target state: [OFFLINE] old value: [ONLINE]
2013-03-06 11:19:37.397: [  CRSOCR][6940] {0:0:38} Multi Write Batch processing...
2013-03-06 11:19:37.398: [ CRSPE][7197] {0:0:38} RI [ora.mdnsd 1 1] new internal state: [CLEANING] old value: [STABLE]
2013-03-06 11:19:37.398: [ CRSPE][7197] {0:0:38} Sending message to agfw: id = 159
2013-03-06 11:19:37.398: [ AGFW][5912] {0:0:38} Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.mdnsd 1 1] ID 4100:159
2013-03-06 11:19:37.399: [ CRSPE][7197] {0:0:38} CRS-2679: Attempting to clean 'ora.mdnsd' on 'gdgis1'

2013-03-06 11:19:37.399: [ AGFW][5912] {0:0:38} Starting the agent: /mboss/app/grid/bin/oraagent with user id: grid and incarnation:11
2013-03-06 11:19:37.425: [ AGFW][5912] {0:0:38} Created alert : (:CRSAGF00123:) :  Failed to start the agent process: /mboss/app/grid/bin/oraagent Category: -2 Operation: waitpid Loc: usrgetgrp12 OS error: 0 Other : child returned 232
2013-03-06 11:19:37.425: [ AGFW][5912] {0:0:38} Created alert : (:CRSAGF00126:) :  Agent start failed
2013-03-06 11:19:37.425: [ AGFW][5912] {0:0:38} Could not forward message [RESOURCE_CLEAN[ora.mdnsd 1 1] ID 4100:159] to agent. /mboss/app/grid/bin/oraagent_grid is not running
2013-03-06 11:19:37.425: [ AGFW][5912] {0:0:38} Starting the agent: /mboss/app/grid/bin/oraagent with user id: grid and incarnation:12
2013-03-06 11:19:37.449: [ AGFW][5912] {0:0:38} Created alert : (:CRSAGF00123:) :  Failed to start the agent process: /mboss/app/grid/bin/oraagent Category: -2 Operation: waitpid Loc: usrgetgrp12 OS error: 0 Other : child returned 232
2013-03-06 11:19:37.449: [ AGFW][5912] {0:0:38} Created alert : (:CRSAGF00126:) :  Agent start failed
2013-03-06 11:19:37.449: [ AGFW][5912] {0:0:38} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_CLEAN[ora.mdnsd 1 1] ID 4100:159
2013-03-06 11:19:37.455: [  CRSOCR][6940] {0:0:38} Multi Write Batch done.
2013-03-06 11:19:37.600: [UiServer][7454] {0:0:38} Container [ Name: ORDER
      MESSAGE:
      TextMessage[CRS-2679: Attempting to clean 'ora.mdnsd' on 'gdgis1']
      MSGTYPE:
      TextMessage[3]
      OBJID:
      TextMessage[ora.mdnsd]
      WAIT:
      TextMessage[0]
]
2013-03-06 11:19:37.601: [ CRSPE][7197] {0:0:38} Received reply to action [Clean] message ID: 159
2013-03-06 11:19:37.601: [ CRSPE][7197] {0:0:38} RI [ora.mdnsd 1 1] new internal state: [STABLE] old value: [CLEANING]
2013-03-06 11:19:37.601: [ CRSPE][7197] {0:0:38} Fatal Error from AGFW Proxy: Unable to start the agent process
2013-03-06 11:19:37.601: [ CRSPE][7197] {0:0:38} CRS-2680: Clean of 'ora.mdnsd' on 'gdgis1' failed

2013-03-06 11:19:37.602: [ CRSPE][7197] {0:0:38} Sequencer for [ora.mdnsd 1 1] has completed with error: CRS-5802: Unable to start the agent process

2013-03-06 11:19:37.604: [UiServer][7454] {0:0:38} Container [ Name: ORDER
      MESSAGE:
      TextMessage[CRS-2680: Clean of 'ora.mdnsd' on 'gdgis1' failed]
      MSGTYPE:
      TextMessage[1]
      OBJID:
      TextMessage[ora.mdnsd]
      WAIT:
      TextMessage[0]
]
2013-03-06 11:19:37.605: [ CRSPE][7197] {0:0:38} PE Command [ Stop Resource : 1148c4b70 ] has completed
2013-03-06 11:19:37.606: [UiServer][7454] {0:0:38} Container [ Name: ORDER
      MESSAGE:
      TextMessage[CRS-5802: Unable to start the agent process]
      MSGTYPE:
      TextMessage[1]
      OBJID:
      TextMessage[ora.mdnsd]
      WAIT:
      TextMessage[0]
]
2013-03-06 11:19:37.606: [ AGFW][5912] {0:0:38} Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:160
2013-03-06 11:19:37.606: [ AGFW][5912] {0:0:38} Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:160
2013-03-06 11:19:37.606: [ CRSPE][7197] {0:0:38} UI Command [Stop Resource : 1148c4b70] is replying to sender.
2013-03-06 11:19:37.607: [UiServer][7454] {0:0:38} Container [ Name: UI_DATA
      ora.mdnsd:
      TextMessage[216]
]
2013-03-06 11:19:37.607: [UiServer][7454] {0:0:38} Done for ctx=113c00390
2013-03-06 11:19:37.615: [UiServer][7711] Closed: remote end failed/disc.
2013-03-06 11:19:38.016: [UiServer][7711] CS(113c00310)set Properties ( root,113bff7b0)
2013-03-06 11:19:38.016: [UiServer][7711] SS(11455a290)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))

回复只看该作者道具举报

KFC

13^#

发表于 2013-3-6 11:41:37

2013-03-06 11:19:38.026: [UiServer][7454] {0:0:39} processMessage called
2013-03-06 11:19:38.027: [UiServer][7454] {0:0:39} Container [ Name: UI_CHECK
      CLIENT:
      TextMessage[]
      DELAY_SECS:
      TextMessage[0]
      LOCALE:
      TextMessage[AMERICAN_AMERICA.US7ASCII]
      RESOURCE:
      TextMessage[ora.mdnsd]
]
2013-03-06 11:19:38.027: [UiServer][7454] {0:0:39} Sending message to PE. ctx= 113c007d0
2013-03-06 11:19:38.027: [UiServer][7454] {0:0:39} Sending command to PE: 37
2013-03-06 11:19:38.028: [ CRSPE][7197] {0:0:39} Processing PE command id=138. Description: [Check Resource : 1148b52d0]
2013-03-06 11:19:38.029: [ AGFW][5912] {0:0:39} Agfw Proxy Server received the message: RESOURCE_PROBE[ora.mdnsd 1 1] ID 4097:163
2013-03-06 11:19:38.029: [ CRSPE][7197] {0:0:39} PE Command [ Check Resource : 1148b52d0 ] has completed
2013-03-06 11:19:38.029: [ AGFW][5912] {0:0:39} Starting the agent: /mboss/app/grid/bin/oraagent with user id: grid and incarnation:13
2013-03-06 11:19:38.030: [ CRSPE][7197] {0:0:39} UI Command [Check Resource : 1148b52d0] is replying to sender.
2013-03-06 11:19:38.055: [ AGFW][5912] {0:0:39} Created alert : (:CRSAGF00123:) :  Failed to start the agent process: /mboss/app/grid/bin/oraagent Category: -2 Operation: waitpid Loc: usrgetgrp12 OS error: 0 Other : child returned 232
2013-03-06 11:19:38.055: [ AGFW][5912] {0:0:39} Created alert : (:CRSAGF00126:) :  Agent start failed
2013-03-06 11:19:38.055: [ AGFW][5912] {0:0:39} Could not forward message [RESOURCE_PROBE[ora.mdnsd 1 1] ID 4097:163] to agent. /mboss/app/grid/bin/oraagent_grid is not running
2013-03-06 11:19:38.055: [ AGFW][5912] {0:0:39} Starting the agent: /mboss/app/grid/bin/oraagent with user id: grid and incarnation:14
2013-03-06 11:19:38.078: [ AGFW][5912] {0:0:39} Created alert : (:CRSAGF00123:) :  Failed to start the agent process: /mboss/app/grid/bin/oraagent Category: -2 Operation: waitpid Loc: usrgetgrp12 OS error: 0 Other : child returned 232
2013-03-06 11:19:38.079: [ AGFW][5912] {0:0:39} Created alert : (:CRSAGF00126:) :  Agent start failed
2013-03-06 11:19:38.079: [ AGFW][5912] {0:0:39} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_PROBE[ora.mdnsd 1 1] ID 4097:163
2013-03-06 11:19:38.230: [ CRSPE][7197] {0:0:39} No listener for operation [Resource Check: [ora.mdnsd 1 1]]. Skipped sending action failed event
2013-03-06 11:19:38.230: [UiServer][7454] {0:0:39} Container [ Name: UI_DATA
      ora.mdnsd:
      TextMessage[0]
]
2013-03-06 11:19:38.231: [UiServer][7454] {0:0:39} Done for ctx=113c007d0
2013-03-06 11:19:38.231: [UiServer][7711] Closed: remote end failed/disc.
2013-03-06 11:19:38.650: [UiServer][7711] CS(113c00310)set Properties ( root,113bff7b0)
2013-03-06 11:19:38.651: [UiServer][7711] SS(11455a290)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-03-06 11:19:38.661: [UiServer][7454] {0:0:40} processMessage called
2013-03-06 11:19:38.662: [UiServer][7454] {0:0:40} Sending message to PE. ctx= 114803e90
2013-03-06 11:19:38.662: [UiServer][7454] {0:0:40} Sending command to PE: 38
2013-03-06 11:19:38.663: [ CRSPE][7197] {0:0:40} Processing PE command id=139. Description: [Stat Resource : 1148b52d0]
2013-03-06 11:19:38.667: [UiServer][7454] {0:0:40} Done for ctx=114803e90
2013-03-06 11:19:38.671: [UiServer][7711] Closed: remote end failed/disc.
2013-03-06 11:19:44.077: [UiServer][7711] CS(113c00310)set Properties ( root,113bff7b0)
2013-03-06 11:19:44.077: [UiServer][7711] SS(11455a290)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-03-06 11:19:44.088: [UiServer][7454] {0:0:41} processMessage called
2013-03-06 11:19:44.088: [UiServer][7454] {0:0:41} Container [ Name: UI_CHECK
      CLIENT:
      TextMessage[]
      DELAY_SECS:
      TextMessage[0]
      LOCALE:
      TextMessage[AMERICAN_AMERICA.US7ASCII]
      RESOURCE:
      TextMessage[ora.diskmon]
]
2013-03-06 11:19:44.088: [UiServer][7454] {0:0:41} Sending message to PE. ctx= 113c007d0
2013-03-06 11:19:44.088: [UiServer][7454] {0:0:41} Sending command to PE: 39
2013-03-06 11:19:44.089: [ CRSPE][7197] {0:0:41} Processing PE command id=140. Description: [Check Resource : 1148b52d0]
2013-03-06 11:19:44.090: [ AGFW][5912] {0:0:41} Agfw Proxy Server received the message: RESOURCE_PROBE[ora.diskmon 1 1] ID 4097:168
2013-03-06 11:19:44.090: [ CRSPE][7197] {0:0:41} PE Command [ Check Resource : 1148b52d0 ] has completed
2013-03-06 11:19:44.091: [ CRSPE][7197] {0:0:41} UI Command [Check Resource : 1148b52d0] is replying to sender.
2013-03-06 11:19:44.091: [ AGFW][5912] {0:0:41} Creating the resource: ora.diskmon 1 1
2013-03-06 11:19:44.091: [ AGFW][5912] {0:0:41} Requesting resource[ora.diskmon 1 1 ] definition from PE
2013-03-06 11:19:44.091: [UiServer][7454] {0:0:41} Container [ Name: UI_DATA
      ora.diskmon:
      TextMessage[0]
]
2013-03-06 11:19:44.092: [ AGFW][5912] {0:0:41} Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:1|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:169:Ver:2]
2013-03-06 11:19:44.092: [UiServer][7454] {0:0:41} Done for ctx=113c007d0
2013-03-06 11:19:44.095: [ AGFW][5912] {0:0:41} Initializing the resource ora.diskmon 1 1 for type ora.diskmon.type
2013-03-06 11:19:44.095: [ AGFW][5912] {0:0:41} SR: acl = owner:root:rw-,pgrp:oinstall:rw-,other::r--,user:grid:r-x
2013-03-06 11:19:44.097: [ AGFW][5912] {0:0:41} Starting the agent: /mboss/app/grid/bin/orarootagent with user id: root and incarnation:1
2013-03-06 11:19:44.107: [UiServer][7711] Closed: remote end failed/disc.
2013-03-06 11:19:44.123: [ AGFW][5912] {0:0:41} Created alert : (:CRSAGF00123:) :  Failed to start the agent process: /mboss/app/grid/bin/orarootagent Category: -2 Operation: waitpid Loc: usrgetgrp12 OS error: 0 Other : child returned 232
2013-03-06 11:19:44.124: [ AGFW][5912] {0:0:41} Created alert : (:CRSAGF00126:) :  Agent start failed
2013-03-06 11:19:44.124: [ AGFW][5912] {0:0:41} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_PROBE[ora.diskmon 1 1] ID 4097:168
2013-03-06 11:19:44.124: [ CRSPE][7197] {0:0:41} No listener for operation [Resource Check: [ora.diskmon 1 1]]. Skipped sending action failed event
2013-03-06 11:19:44.502: [UiServer][7711] CS(113c00310)set Properties ( root,113bff7b0)
2013-03-06 11:19:44.503: [UiServer][7711] SS(11455a290)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-03-06 11:19:44.513: [UiServer][7454] {0:0:42} processMessage called
2013-03-06 11:19:44.514: [UiServer][7454] {0:0:42} Sending message to PE. ctx= 114862df0
2013-03-06 11:19:44.514: [UiServer][7454] {0:0:42} Sending command to PE: 40
2013-03-06 11:19:44.515: [ CRSPE][7197] {0:0:42} Processing PE command id=141. Description: [Stat Resource : 1148c7e30]
2013-03-06 11:19:44.520: [UiServer][7454] {0:0:42} Done for ctx=114862df0
2013-03-06 11:19:44.523: [UiServer][7711] Closed: remote end failed/disc.
2013-03-06 11:19:49.981: [UiServer][7711] CS(113c00310)set Properties ( root,113bff7b0)
2013-03-06 11:19:49.981: [UiServer][7711] SS(11455a290)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-03-06 11:19:49.991: [UiServer][7454] {0:0:43} processMessage called
2013-03-06 11:19:49.992: [UiServer][7454] {0:0:43} Sending message to PE. ctx= 114862df0
2013-03-06 11:19:49.992: [UiServer][7454] {0:0:43} Sending command to PE: 41
2013-03-06 11:19:49.993: [ CRSPE][7197] {0:0:43} Processing PE command id=142. Description: [Stat Resource : 1148c7e30]
2013-03-06 11:19:49.997: [UiServer][7454] {0:0:43} Done for ctx=114862df0
2013-03-06 11:19:50.001: [UiServer][7711] Closed: remote end failed/disc.
2013-03-06 11:33:22.574: [UiServer][7711] CS(113c00310)set Properties ( root,113bff7b0)
2013-03-06 11:33:22.574: [UiServer][7711] SS(11455a290)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=40)(KEY=OHASD_UI_SOCKET))daddr = (ADDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2013-03-06 11:33:22.584: [UiServer][7454] {0:0:44} processMessage called
2013-03-06 11:33:22.585: [UiServer][7454] {0:0:44} Sending message to PE. ctx= 114803f50
2013-03-06 11:33:22.585: [UiServer][7454] {0:0:44} Sending command to PE: 42
2013-03-06 11:33:22.586: [ CRSPE][7197] {0:0:44} Processing PE command id=143. Description: [Stat Resource : 1148c7e30]
2013-03-06 11:33:22.612: [UiServer][7454] {0:0:44} Done for ctx=114803f50
2013-03-06 11:33:22.616: [UiServer][7711] Closed: remote end failed/disc.

回复只看该作者道具举报

Maclean Liu(刘相兵

14^#

发表于 2013-3-6 11:53:57

怀疑还是和升级步骤可能有关系，我没有写过11.2.0.2 升级11.2.0.3的文档，可能里面有一些坑。

升级坏了确实麻烦，考虑你的方案3.

回复只看该作者道具举报

返回列表

		自动登录	找回密码
密码			注册