root.sh Failed to start Oracle Grid Infrastructure stack

anbob

环境
11203 rac   2 nodes
OEL 5.8
两节点运行正常，增加新节点3
[grid@znode1 deinstall]$ cluvfy stage -pre nodeadd -n znode3 -verbose
检查通过
[grid@znode1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={znode3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={znode3-vip}"
正常

1. Save inventory complete
   
2. WARNING:
   
3. The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
   
4. /u01/app/11.2.0/grid/root.sh #On nodes znode3
   
5. To execute the configuration scripts:
   
6.     1. Open a terminal window
   
7.     2. Log in as "root"
   
8.     3. Run the scripts in each cluster node
   
9.    
   
10. The Cluster Node Addition of /u01/app/11.2.0/grid was successful.
    
11. Please check '/soft/temp/silentInstall.log' for more details.
    
12. 
    
13. 
    
14. [root@znode3 oracle]# /u01/app/11.2.0/grid/root.sh
    
15. Performing root user operation for Oracle 11g
    
16. 
    
17. The following environment variables are set as:
    
18.     ORACLE_OWNER= grid
    
19.     ORACLE_HOME=  /u01/app/11.2.0/grid
    
20. 
    
21. Enter the full pathname of the local bin directory: [/usr/local/bin]:
    
22. The contents of "dbhome" have not changed. No need to overwrite.
    
23. The contents of "oraenv" have not changed. No need to overwrite.
    
24. The contents of "coraenv" have not changed. No need to overwrite.
    
25. 
    
26. Entries will be added to the /etc/oratab file as needed by
    
27. Database Configuration Assistant when a database is created
    
28. Finished running generic part of root script.
    
29. Now product-specific root actions will be performed.
    
30. perl: warning: Setting locale failed.
    
31. perl: warning: Please check that your locale settings:
    
32.         LANGUAGE = (unset),
    
33.         LC_ALL = (unset),
    
34.         LANG = "en_US.ZHS16GBK"
    
35.     are supported and installed on your system.
    
36. perl: warning: Falling back to the standard locale ("C").
    
37. Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
    
38. User ignored Prerequisites during installation
    
39. OLR initialization - successful
    
40. Adding Clusterware entries to inittab
    
41. CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node znode1, number 1, and is terminating
    
42. An active cluster was found during exclusive startup, restarting to join the cluster
    
43. Start of resource "ora.cssd" failed
    
44. CRS-2672: Attempting to start 'ora.cssdmonitor' on 'znode3'
    
45. CRS-2672: Attempting to start 'ora.gipcd' on 'znode3'
    
46. CRS-2676: Start of 'ora.cssdmonitor' on 'znode3' succeeded
    
47. CRS-2676: Start of 'ora.gipcd' on 'znode3' succeeded
    
48. CRS-2672: Attempting to start 'ora.cssd' on 'znode3'
    
49. CRS-2672: Attempting to start 'ora.diskmon' on 'znode3'
    
50. CRS-2676: Start of 'ora.diskmon' on 'znode3' succeeded
    
51. CRS-2674: Start of 'ora.cssd' on 'znode3' failed
    
52. CRS-2679: Attempting to clean 'ora.cssd' on 'znode3'
    
53. CRS-2681: Clean of 'ora.cssd' on 'znode3' succeeded
    
54. CRS-2673: Attempting to stop 'ora.gipcd' on 'znode3'
    
55. CRS-2677: Stop of 'ora.gipcd' on 'znode3' succeeded
    
56. CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'znode3'
    
57. CRS-2677: Stop of 'ora.cssdmonitor' on 'znode3' succeeded
    
58. CRS-5804: Communication error with agent process
    
59. CRS-4000: Command Start failed, or completed with errors.
    
60. Failed to start Oracle Grid Infrastructure stack
    
61. Failed to start Cluster Synchorinisation Service in clustered mode at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1211.
    
62. /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

复制代码

有执行/u01/app/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force
重复执行root.sh 错误同样

[ 本帖最后由 anbob 于 2012-7-24 16:21 编辑 ]

anbob

alertznode3.log
<code>
2012-07-24 15:55:26.908
[gpnpd(19645)]CRS-2328:GPNPD started on node znode3.
2012-07-24 15:55:29.721
[cssd(19715)]CRS-1713:CSSD daemon is started in exclusive mode
2012-07-24 15:55:31.536
[ohasd(19203)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2012-07-24 15:55:33.941
[cssd(19715)]CRS-1637:Unable to locate configured voting file with ID 7b48588b-96a44fc8-bf7c44c0-8d8ab516; details at (:CSSNM00020:) in /u01/app/11.2.0/grid/
log/znode3/cssd/ocssd.log
2012-07-24 15:55:33.941
[cssd(19715)]CRS-1637:Unable to locate configured voting file with ID 5ef1fd16-324c4fcd-bf4bd89b-da987740; details at (:CSSNM00020:) in /u01/app/11.2.0/grid/
log/znode3/cssd/ocssd.log
2012-07-24 15:55:39.609
[cssd(19715)]CRS-1707:Lease acquisition for node znode3 number 3 completed
2012-07-24 15:55:40.906
[cssd(19715)]CRS-1605:CSSD voting file is online: /dev/raw/raw1; details in /u01/app/11.2.0/grid/log/znode3/cssd/ocssd.log.
[cssd(19715)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node znode1 and is terminating; details at (:CSSNM00006:
) in /u01/app/11.2.0/grid/log/znode3/cssd/ocssd.log
2012-07-24 15:55:42.821
[ohasd(19203)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'znode3'.
2012-07-24 15:55:43.622
[gpnpd(19645)]CRS-2329:GPNPD on node znode3 shutdown.
2012-07-24 15:55:44.988
[mdnsd(19633)]CRS-5602:mDNS service stopping by request.
2012-07-24 15:56:07.779
[gpnpd(19902)]CRS-2328:GPNPD started on node znode3.
2012-07-24 15:56:10.969
[cssd(19981)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-24 15:56:12.801
[ohasd(19203)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2012-07-24 15:56:15.124
[cssd(19981)]CRS-1637:Unable to locate configured voting file with ID 7b48588b-96a44fc8-bf7c44c0-8d8ab516; details at (:CSSNM00020:) in /u01/app/11.2.0/grid/
log/znode3/cssd/ocssd.log
2012-07-24 15:56:15.124
[cssd(19981)]CRS-1637:Unable to locate configured voting file with ID 5ef1fd16-324c4fcd-bf4bd89b-da987740; details at (:CSSNM00020:) in /u01/app/11.2.0/grid/
log/znode3/cssd/ocssd.log
2012-07-24 15:56:15.124
[cssd(19981)]CRS-1705:Found 1 configured voting files but 2 voting files are required, terminating to ensure data integrity; details at (:CSSNM00021:) in /u0
1/app/11.2.0/grid/log/znode3/cssd/ocssd.log
2012-07-24 15:56:15.124
[cssd(19981)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/znode3/cssd/ocssd.log
2012-07-24 15:56:15.155
[cssd(19981)]CRS-1603:CSSD on node znode3 shutdown by user.
2012-07-24 15:56:15.163
[ohasd(19203)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'znode3'.
</code>

anbob

csstOUT.log

07/24/12 15:55:28: CSSD starting
07/24/12 15:56:09: CSSD starting


----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
clssscExit()+956     call     kgdsdst()            B3EDE094 ? 3 ?
clssnmCompleteVFDis  call     clssscExit()         9B0AF00 ? 2 ?
covery()+1288
clssnmCompleteInitV  call     clssnmCompleteVFDis  9B0AF00 ? 9C00A28 ? 9DA1B48 ?
FDiscovery()+1714             covery()             1 ?
clssnmvDDiscThread(  call     00000000             9B0AF00 ? 9C00A28 ? 9DA1B48 ?
)+2343                                             1 ?
clssscthrdmain()+23  call     00000000             9B0AF00 ? 99EE008 ?
8                                                  48FADFF4 ? 99EE778 ?
                                                   812E288 ? B3EDE4B8 ?
start_thread()+226   call     00000000             9B0AF00 ? B3EDE480 ?
                                                   B3EDE480 ? B3EDE480 ?
clone()+94           call     00000000             B3EDEB90 ? 0 ? 0 ? 0 ? 0 ?
                                                   0 ?

[ 本帖最后由 anbob 于 2012-7-24 16:21 编辑 ]

anbob

找到原因了，前几天搞asm启动，修改了node1的asm_diskstring='/dev/raw/raw*',后来重启node1启不来报一样的错误

解决在node2
SQL>  alter system set asm_diskstring='' scope=spfile sid='+ASM1';

System altered.
再启动znode1,znode3 root.sh都正常

Maclean Liu(刘相兵

Good!