请帮忙诊断ons异常

数据库版本oracle 10.2.0.4  rac  aix 6.1


当前crs各资源状态
[test1:/oracle10/app/product/crs/10.2.0/log/test1/racg]crs_stat-t
Name          Type          Target    State    Host        
------------------------------------------------------------         
ora....m1.inst application    ONLINE   ONLINE    test1   
ora....m2.inst application   ONLINE    ONLINE    test2   
ora....icom.db application   ONLINE    ONLINE    test2   
ora....om1.srv application   ONLINE    ONLINE    test2   
ora.....dtp.cs application   ONLINE    ONLINE    test1   
ora....B1.lsnr application   ONLINE    ONLINE    test1   
ora....db1.gsd application   ONLINE   OFFLINE              
ora....db1.ons application   ONLINE   OFFLINE              
ora....db1.vip application   ONLINE    ONLINE    test1   
ora....B2.lsnr application   ONLINE    ONLINE    test2   
ora....db2.gsd application   ONLINE   OFFLINE              
ora....db2.ons application   ONLINE    ONLINE    test2   
ora....db2.vip application   ONLINE    ONLINE    test2   

节点1上各种状态
1.
ons进程
[test1:/oracle10/app/product/crs/10.2.0/log/test1/racg]ps-ef|grep "ons -d"|grep -v grep
oracle10  5701830  5112098   0   Apr21      -  7:18/oracle10/app/product/crs/10.2.0/opmn/bin/ons -d
oracle10 20119606       1   0 09:02:27      -  0:00/oracle10/app/product/db/10.2.0/opmn/bin/ons -d
oracle10  5112098       1   0   Apr 21      -  0:00/oracle10/app/product/crs/10.2.0/opmn/bin/ons -d
oracle10 28836336 20119606   009:02:27      -  0:00 /oracle10/app/product/db/10.2.0/opmn/bin/ons–d

2.
端口情况
[test1:/oracle10/app/product/crs/10.2.0/opmn/conf]moreons.config
localport=6113
remoteport=6200
loglevel=3
useocr=on

3.
当前系统监听端口
[test1:/oracle10/app/product/crs/10.2.0/opmn/conf]netstat-Aan|grep 6200|grep -v grep
f1000e00014223b0 tcp4      0      0  134.100.9.12.1521 10.235.5.21.46200  ESTABLISHED
f1000e0000307bb0 tcp4      0      0 *.6200            *.*               LISTEN
f1000e0023216200 udp4      0      0  192.168.16.11.5187*.*              
f1000e00308d6200 udp4      0      0  192.168.16.11.5737*.*              
f1000e00081f6200 udp4      0      0  192.168.16.11.5738*.*              
f1000e00222e6200 udp4      0      0  192.168.16.11.3415*.*              
f1000e0022266200 udp4      0      0  192.168.16.11.6002*.*              
f1000e00231d6200 udp4      0      0  192.168.16.11.6185*.*              
f1000e00360b6200 udp4      0      0  192.168.16.11.3867*.*              
f1000e001acb6200 udp4      0      0  192.168.16.11.3910*.*              
f1000e0008216200 udp4      0      0  192.168.16.11.3963*.*              
f1000e0017436200
f1000e0009854008 dgram      0     0               0 f1000e000004c180               0 f1000e0017436200
[test1:/oracle10/app/product/crs/10.2.0/opmn/conf]rmsockf1000e0000307bb0 tcpcb
The socket 0x307808 is being held by proccess 5701830 (ons).
[test1:/oracle10/app/product/crs/10.2.0/opmn/conf]ps -ef|grep5701830|grep -v grep
oracle10  5701830  5112098   0   Apr21      -  7:18 /oracle10/app/product/crs/10.2.0/opmn/bin/ons–d

4.
onsctl ping
[test1:/oracle10/app/product/crs/10.2.0/opmn/conf]onsctl ping
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
   {node = test1, port = 6200}
Adding remote host test1:6200
onscfg[1]
   {node = test2, port = 6200}
Adding remote host test2:6200
/oracle10/app/product/db/10.2.0/opmn/logs/ons.log.test1:Permission denied
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files

ons is not running ...

Maclean Liu(刘相兵

有兴趣的话  对ons 做 trace 深入分析一下 ， follow 这个note:

1. How to Log or Trace ONS in RAC 11g Release 2 [ID 1270841.1]
   
2. 
   
3. Applies to:
   
4. JDBC - Version: 10.2.0.1 to 11.2.0.2.0 - Release: 10.2 to 11.2
   
5. Information in this document applies to any platform.
   
6. Goal
   
7. How to switch on ONS logging in RAC version 11.2.0.1 ?
   
8. The loglevel parameter, used in earlier versions of RAC, is deprecated in 11.2.0.1.
   
9. Solution
   
10. ONS version 11.2.0.1 has introduced a new method of logging or tracing ONS activities.
    
11. 
    
12. On each node, edit ons.config.<node name> and ons.config  files under /opmn/conf and add the following parameter:
    
13. 
    
14. debugcomp=ons[subcomponent]
    
15. 
    
16. 
    
17. Valid subcomponents for ons are:
    
18. 
    
19.       all - all messages
    
20.       local - ONS local information
    
21.       listener - ONS listener information
    
22.       discover - ONS discover (server or multicast) information
    
23.       servers - ONS remote servers currently up and connected to the cluster
    
24.       topology - ONS current cluster wide server connection topology
    
25.       server - ONS remote server connection information
    
26.       client - ONS client connection information
    
27.       connect - ONS generic connection information
    
28.       subscribe - ONS client subscription information
    
29.       message - ONS notification receiving and processing information
    
30.       deliver - ONS notification delivery information
    
31.       special - ONS special notification processing
    
32.       internal - ONS internal resource information
    
33.       secure - ONS SSL operation information
    
34.       workers - ONS worker threads
    
35. 
    
36. 
    
37. For example,  debugcomp=ons[all,!secure]  logs all information except ONS SSL operations.
    
38. 
    
39. All the log information is written to the file ons.dbg.<node name> under /opmn/logs.
    
40. 
    
41. 
    
42. Note: ONS has to be restarted after editing ons.config.<node name> file :
    
43. /bin/onsctl stop/start

复制代码

Maclean Liu(刘相兵

ODM data:

1. Hdr: 6429778 10.2.0.3 ONS 10.2.0.3 PRODID-1032 PORTID-212
   
2. Abstract: ONS KEEPS GROWING AND CONSUME HIGH MEMORY
   
3. 
   
4. *** 09/16/07 10:28 pm ***
   
5. TAR:
   
6. ----
   
7. 
   
8. PROBLEM:
   
9. --------
   
10. ONS keeps growing and consume high memory. ONS needs to be recycled after
    
11. every hour.
    
12. 
    
13. DIAGNOSTIC ANALYSIS:
    
14. --------------------
    
15. s24_ons.log
    
16. ===========
    
17. 07/09/14 10:49:36 [2] Passive connection 0,159.181.33.227,6203 invalid
    
18. connect server IP format
    
19. 2679448025,6203,6203,0
    
20. 
    
21. ONSinfo: !!2679448025!0!6203!0!6203
    
22. 
    
23. hostName: sa4dj025
    
24. 
    
25. clusterId: databaseClusterId
    
26. 
    
27. clusterName: databaseClusterName
    
28. 
    
29. instanceId: databaseInstanceId
    
30. 
    
31. instanceName: databaseInstanceName
    
32. 
    
33. 
    
34. s25_ons.log
    
35. ===========
    
36. 07/09/14 10:49:21 [2] Passive connection 0,159.181.33.217,6203 invalid
    
37. connect server IP format
    
38. 2679448035,6203,6203,0
    
39. 
    
40. ONSinfo: !!2679448035!0!6203!0!6203
    
41. 
    
42. hostName: sa4dj024
    
43. 
    
44. clusterId: databaseClusterId
    
45. 
    
46. clusterName: databaseClusterName
    
47. 
    
48. instanceId: databaseInstanceId
    
49. 
    
50. instanceName: databaseInstanceName
    
51. 
    
52. WORKAROUND:
    
53. -----------
    
54. Recycle ONS after some time.
    
55. 
    
56. RELATED BUGS:
    
57. -------------
    
58. 
    
59. REPRODUCIBILITY:
    
60. ----------------
    
61. continuously occurring in CT's environment
    
62. 
    
63. 
    
64. Hdr: 6883669 10.2.0.3 ONS 10.2.0.3 PRODID-1032 PORTID-212
    
65. Abstract: IN TWO NODE RAC, ONS.LOG REPORTS "INVALID CONNECT SERVER IP FORMAT" ERROR
    
66. 
    
67. *** 03/11/08 06:13 pm ***
    
68. TAR:
    
69. ----
    
70. 
    
71. PROBLEM:
    
72. --------
    
73. The ons.log grows continuously because of an error message that gets output
    
74. regularly.
    
75. 
    
76. ons.log has the following entries repeatedly:
    
77. 
    
78. 08/03/05 17:36:43 [2] Passive connection 0,172.16.1.101,6200 invalid connect
    
79. server IP format
    
80. 2886730086,6201,6114,0 ONSinfo: !!2886730086!0!6201!0!6114 hostName: orarac2
    
81. clusterId: databaseClusterId clusterName: databaseClusterName instanceId:
    
82. databaseInstanceId instanceName: databaseInstanceName
    
83. 08/03/05 17:36:56 [5] Listener thread 1286: 172.16.1.101:6200 (0x101)
    
84. listening
    
85. 
    
86. DIAGNOSTIC ANALYSIS:
    
87. --------------------
    
88. 
    
89. ons.config shows
    
90. 
    
91. localport=6113
    
92. remoteport=6200
    
93. loglevel=5
    
94. useocr=on
    
95. 
    
96. This is same problem reported in the bug 6429778.
    
97. 
    
98. The problem happens on boht of the customer's production and development
    
99. cluster.

复制代码

就note 来看是 一个bug 容易发生在AIX上

5. ons.log日志
[zwq_kfdb1:/oracle10/app/product/crs/10.2.0/opmn/logs]tail -20 ons.log
instanceName: databaseInstanceName

12/02/23 13:20:17 [2] Passive connection 0,134.100.9.11,6200 invalid connect server IP format
2254702859,6201,6114,0
ONSinfo: !!2254702859!0!6201!0!6114
hostName: zwq_kfdb1
clusterId: databaseClusterId
clusterName: databaseClusterName
instanceId: databaseInstanceId
instanceName: databaseInstanceName

12/02/23 13:21:24 [2] Passive connection 0,134.100.9.11,6200 invalid connect server IP format
2254702861,6200,6113,0
ONSinfo: !!2254702861!0!6200!0!6113
hostName: zwq_kfdb2
clusterId: databaseClusterId
clusterName: databaseClusterName
instanceId: databaseInstanceId
instanceName: databaseInstanceName

6. ora.zwq_kfdb1.ons.log文件
[zwq_kfdb1:/oracle10/app/product/crs/10.2.0/log/zwq_kfdb1/racg]ls -l
total 200
-rw-r--r--    1 oracle10 oinstall          0 Apr 30 2011  evtf.log
-rw-r--r--    1 oracle10 oinstall       4169 Apr 29 2011  ora.ahsheet.db.log
-rwxrwxr-x    1 oracle10 oinstall        564 Jan 16 2011  ora.ahunicom.db.log
-rwxrwxr-x    1 oracle10 dba           40887 Apr 29 2011  ora.zwq_kfdb1.gsd.log
-rwxrwxr-x    1 oracle10 dba           20480 Apr 30 2011  ora.zwq_kfdb1.ons.log
-rwxrwxr-x    1 root     system        16384 Apr 30 2011  ora.zwq_kfdb1.vip.log
-rwxrwxr-x    1 root     system         9966 Jan 16 2011  ora.zwq_kfdb2.vip.log
drwxrwxr-x    2 oracle10 dba             256 Jan 15 2011  racgeut
drwxrwxr-x    2 oracle10 dba             256 Jan 15 2011  racgevtf
drwxrwxr-x    2 oracle10 dba             256 Jan 15 2011  racgmain
[zwq_kfdb1:/oracle10/app/product/crs/10.2.0/log/zwq_kfdb1/racg]tail -20 ora.zwq_kfdb1.ons.log

2011-04-21 04:00:56.731: [    RACG][1] [5243026][1][ora.zwq_kfdb1.ons]: Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
   {node = zwq_kfdb1, port = 6200}
Adding remote host zwq_kfdb1:6200
onscfg[1]
   {node = zwq_kfdb2, port = 6200}
Adding remote host zwq_kfdb2:6200

2011-04-21 04:00:56.735: [    RACG][1] [5243026][1][ora.zwq_kfdb1.ons]: Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
   {node = zwq_kfdb1, port = 6200}
Adding remote host zwq_kfdb1:6200
onscfg[1]
   {node = zwq_kfdb2, port = 6200}
Adding remote host zwq_kfdb2:6200
onsctl: ons started

2011-04-30 18:23:57.514: [ CSSCLNT][1]clsssInitNative: connect failed, rc 2
我今天早上让客户执行了onsctl start和onsctl ping，但是都没有看到相关记录，我自己没有权限执行，如同上面提示Permission denied


节点2上相关情况
进程状态
[zwq_kfdb2:/proc]ps -ef|grep "ons -d"
oracle10  7667896  3342636   0   Apr 29      -  6:55 /oracle10/app/product/crs/10.2.0/opmn/bin/ons -d
oracle10  3342636        1   0   Apr 29      -  0:00 /oracle10/app/product/crs/10.2.0/opmn/bin/ons –d

端口和节点1一样，也是ons占用6200
端口配置也和节点1相同

onsctl ping
[zwq_kfdb2:/proc]onsctl ping
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
   {node = zwq_kfdb1, port = 6200}
Adding remote host zwq_kfdb1:6200
onscfg[1]
   {node = zwq_kfdb2, port = 6200}
Adding remote host zwq_kfdb2:6200
/oracle10/app/product/db/10.2.0/opmn/logs/ons.log.zwq_kfdb2: Permission denied
ons is not running ...

ora.zwq_kfdb2.ons.log
[zwq_kfdb2:/oracle10/app/product/crs/10.2.0/log/zwq_kfdb2/racg]tail -20 ora.zwq_kfdb2.ons.log
Adding remote host zwq_kfdb2:6200
onsctl: ons started

2011-04-29 12:06:45.581: [    RACG][1] [7405762][1][ora.zwq_kfdb2.ons]: Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
   {node = zwq_kfdb1, port = 6200}
Adding remote host zwq_kfdb1:6200
onscfg[1]
   {node = zwq_kfdb2, port = 6200}
Adding remote host zwq_kfdb2:6200

2011-04-29 12:06:45.586: [    RACG][1] [7405762][1][ora.zwq_kfdb2.ons]: Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
   {node = zwq_kfdb1, port = 6200}
Adding remote host zwq_kfdb1:6200
onscfg[1]
   {node = zwq_kfdb2, port = 6200}
Adding remote host zwq_kfdb2:6200
onsctl: ons started

[zwq_kfdb2:/oracle10/app/product/crs/10.2.0/opmn/logs]tail -20 ons.log
instanceName: databaseInstanceName

12/02/23 13:29:45 [2] Passive connection 0,134.100.9.13,6200 invalid connect server IP format
2254702859,6200,6113,0
ONSinfo: !!2254702859!0!6200!0!6113
hostName: zwq_kfdb1
clusterId: databaseClusterId
clusterName: databaseClusterName
instanceId: databaseInstanceId
instanceName: databaseInstanceName

12/02/23 13:30:12 [2] Passive connection 0,134.100.9.13,6200 invalid connect server IP format
2254702859,6201,6114,0
ONSinfo: !!2254702859!0!6201!0!6114
hostName: zwq_kfdb1
clusterId: databaseClusterId
clusterName: databaseClusterName
instanceId: databaseInstanceId
instanceName: databaseInstanceName

我知道ons资源不太重要，但是我想知道为什么我不能让ons资源online，请ml帮忙诊断下。