RAC 一节点监听频繁重启

Honcho

操作系统：
p570_svr3:/home/oracle$uname -a
AIX p570_svr3 3 5 00CB4D124C00

数据库版本：
SQL> select * from v$version;
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bi
PL/SQL Release 10.2.0.3.0 - Production
CORE    10.2.0.3.0      Production
TNS for IBM/AIX RISC System/6000: Version 10.2.0.3.0 - Productio
NLSRTL Version 10.2.0.3.0 - Production

cluster版本：
p570_svr3:/home/oracle$crsctl query crs softwareversion
CRS software version on node [p570_svr3] is [10.2.0.5.0]


监听日志：
p570_svr3:/opt/app/oracle/product/10.2.0/db_1/network/log$tail -n 2000 listener_p570_svr3.log|grep -i stop
Listener completed notification to CRS on stop
19-OCT-2011 22:46:26 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
19-OCT-2011 22:58:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
19-OCT-2011 23:30:04 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
19-OCT-2011 23:59:52 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 00:20:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 00:29:52 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 01:14:08 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 01:21:08 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 01:41:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 01:55:08 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 02:23:17 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 02:31:33 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 03:45:50 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 03:52:52 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 04:17:26 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 08:49:35 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
p570_svr3:/opt/app/oracle/product/10.2.0/db_1/network/log$tail -n 3000  listener_p570_svr3.log|grep -i stop
Listener completed notification to CRS on stop
19-OCT-2011 22:46:26 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
19-OCT-2011 22:58:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
19-OCT-2011 23:30:04 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
19-OCT-2011 23:59:52 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 00:20:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 00:29:52 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 01:14:08 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 01:21:08 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 01:41:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 01:55:08 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 02:23:17 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 02:31:33 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 03:45:50 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 03:52:52 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 04:17:26 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0
Listener completed notification to CRS on stop
20-OCT-2011 08:49:35 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=p570_svr3)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER_P570_SVR3)(VERSION=169870080)) * stop * 0

Honcho

数据库alert日志
ALTER SYSTEM SET service_names='orclerp' SCOPE=MEMORY SID='orclerp2';
Wed Oct 19 23:59:18 2011
ALTER SYSTEM SET service_names='' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 00:07:53 2011
ALTER SYSTEM SET service_names='orclerp' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 00:20:17 2011
ALTER SYSTEM SET service_names='' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 00:28:03 2011
ALTER SYSTEM SET service_names='orclerp' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 00:29:19 2011
ALTER SYSTEM SET service_names='' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 00:38:08 2011
ALTER SYSTEM SET service_names='orclerp' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 01:13:34 2011
ALTER SYSTEM SET service_names='' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 01:18:14 2011
ALTER SYSTEM SET service_names='orclerp' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 01:20:34 2011
ALTER SYSTEM SET service_names='' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 01:28:19 2011
ALTER SYSTEM SET service_names='orclerp' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 01:41:17 2011
ALTER SYSTEM SET service_names='' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 01:48:25 2011
ALTER SYSTEM SET service_names='orclerp' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 01:54:35 2011
ALTER SYSTEM SET service_names='' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 01:58:30 2011
ALTER SYSTEM SET service_names='orclerp' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 02:22:44 2011
ALTER SYSTEM SET service_names='' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 02:28:38 2011
ALTER SYSTEM SET service_names='orclerp' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 02:30:59 2011
ALTER SYSTEM SET service_names='' SCOPE=MEMORY SID='orclerp2';
Thu Oct 20 02:38:43 2011
ALTER SYSTEM SET service_names='orclerp' SCOPE=MEMORY SID='orclerp2';

kevinlin.ora

1. crsctl query crs activeversion - lists the CRS software operating version
2. 检查crs log
3. 确认VIP始终可用，必要时使用crsctl debug进行诊断
4. 必要时listener 设trace进行诊断
5. 互联网上查找相关信息。

vmcd_gg

是否有进展  希望及时跟进

162743

啥原因最后说明:victory:

Honcho

在vip log里面看到如下消息：
2011-10-20 08:59:46.481: [    RACG][1] [708914][1][ora.p570_svr3.vip]: clsrcstartorp: Error with malloc
2011-10-20 22:12:44.429: [    RACG][1] [856628][1][ora.p570_svr3.vip]: Invalid parameters, or failed to bring up VIP (host=p570_svr3
)
2011-10-20 22:12:44.429: [    RACG][1] [856628][1][ora.p570_svr3.vip]: clsrcexecut: env ORACLE_CONFIG_HOME=/opt/app/oracle/product/1
0.2.0/crs
2011-10-20 22:12:44.429: [    RACG][1] [856628][1][ora.p570_svr3.vip]: clsrcexecut: cmd = /opt/app/oracle/product/10.2.0/crs/bin/rac
geut -e _USR_ORA_DEBUG=0 54 /opt/app/oracle/product/10.2.0/crs/bin/racgvip check p570_svr3
2011-10-20 22:12:44.429: [    RACG][1] [856628][1][ora.p570_svr3.vip]: clsrcexecut: rc = 1, time = 4.407s
2011-10-20 22:12:44.429: [    RACG][1] [856628][1][ora.p570_svr3.vip]: end for resource = ora.p570_svr3.vip, action = check, status
= 1, time = 4.565s
2011-10-20 22:21:15.528: [    RACG][1] [688588][1][ora.p570_svr3.vip]: clsrcstartorp: Error with malloc
2011-10-20 22:46:20.237: [    RACG][1] [1302944][1][ora.p570_svr3.vip]: Invalid parameters, or failed to bring up VIP (host=p570_svr
3)
2011-10-20 22:46:20.237: [    RACG][1] [1302944][1][ora.p570_svr3.vip]: clsrcexecut: env ORACLE_CONFIG_HOME=/opt/app/oracle/product/
10.2.0/crs
2011-10-20 22:46:20.237: [    RACG][1] [1302944][1][ora.p570_svr3.vip]: clsrcexecut: cmd = /opt/app/oracle/product/10.2.0/crs/bin/ra
cgeut -e _USR_ORA_DEBUG=0 54 /opt/app/oracle/product/10.2.0/crs/bin/racgvip check p570_svr3
2011-10-20 22:46:20.237: [    RACG][1] [1302944][1][ora.p570_svr3.vip]: clsrcexecut: rc = 1, time = 4.305s
2011-10-20 22:46:20.237: [    RACG][1] [1302944][1][ora.p570_svr3.vip]: end for resource = ora.p570_svr3.vip, action = check, status
= 1, time = 4.456s
2011-10-20 22:51:25.563: [    RACG][1] [512252][1][ora.p570_svr3.vip]: clsrcstartorp: Error with malloc
2011-10-20 22:55:47.512: [    RACG][1] [1409774][1][ora.p570_svr3.vip]: Invalid parameters, or failed to bring up VIP (host=p570_svr
3)
2011-10-20 22:55:47.512: [    RACG][1] [1409774][1][ora.p570_svr3.vip]: clsrcexecut: env ORACLE_CONFIG_HOME=/opt/app/oracle/product/
10.2.0/crs
2011-10-20 22:55:47.512: [    RACG][1] [1409774][1][ora.p570_svr3.vip]: clsrcexecut: cmd = /opt/app/oracle/product/10.2.0/crs/bin/ra
cgeut -e _USR_ORA_DEBUG=0 54 /opt/app/oracle/product/10.2.0/crs/bin/racgvip check p570_svr3
2011-10-20 22:55:47.512: [    RACG][1] [1409774][1][ora.p570_svr3.vip]: clsrcexecut: rc = 1, time = 4.307s
2011-10-20 22:55:47.512: [    RACG][1] [1409774][1][ora.p570_svr3.vip]: end for resource = ora.p570_svr3.vip, action = check, status
= 1, time = 4.469s

[ 本帖最后由 Honcho 于 2011-10-21 15:53 编辑 ]

Honcho

在matelink 上找到如下信息：
VIP on AIX 5.3TL9+ Fails to Come Up with "Invalid Parameters, Or Failed To Bring Up VIP" [ID 959746.1]
Applies to:
Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 10.2.0.4 - Release: 10.2 to 10.2
IBM AIX on POWER Systems (64-bit)
IBM AIX Based Systems (64-bit)
Symptoms
On AIX 5.3 TL9+ , AIX 6 or AIX 6.1, the VIP fails to come up, with error "Invalid Parameters, Or Failed To Bring Up VIP".
Tracing the racgvip command shows that it fails when checking to see if the public interface (NIC) is up.
However, the public interface is up.
An error message similar to the following may be seen in logs:
2009-07-23 17:40:05.812: [ RACG][1] [270490][1][ora.srvr0101.vip]: Thu Jul 23 17:40:05 BST 2009 [ 159774 ] IsIfAlive: /usr/bin/entstat -d en0 failed. Return = 1 (host=srvr0101)
Thu Jul 23 17:40:05 BST 2009 [ 159774 ] checkIf: end for if=en0
Invalid parameters, or failed to bring up VIP (host=srvr0101)
Changes
The adapter type for the public network is LHEA (IBM Logical Host Ethernet Adapter):
# /usr/bin/entstat -d en0
-------------------------------------------------------------
ETHERNET STATISTICS (en0) :
Device Type: Host Ethernet Adapter (l-hea)
...
Cause
The entstat output for LHEA is different from a regular adapter.  
The racgvip script uses entstat output to determine if the specified interface is up or not; because the entstat output for LHEA is different, the check fails, therefore the VIP will not come up.
This is a known issue reported in the following bug:
Bug 8725020 - VIP WONT RUN (LHEA) ADAPTER 5.3 TL9

Solution
The following workaround fixes the racgvip script so that it does not fail for LHEA adapters:
1. Backup the racgvip script
2. Edit this line in the script using vi:
$ENTSTAT -d $_IF | $GREP -iEq '.*lan.*state.*:.*operational.*|.*link.*status.*:.*up.*|.*port.*operational.*state.*:.*up.*'
and replace it with this:
$ENTSTAT -d $_IF | $GREP -iEq '.*lan.*state.*:.*operational.*|.*link.*status.*:.*up.*|.*port.*operational.*state.*:.*up.*|.*driver.*flags.*:.*up.*'
(Notice that an extra regexp clause has been tacked on the end of the grep argument.)
3. Make sure that no stray characters have been introduced.
4. Save the racgvip file.

现在没有出现状况，等晚上十一点左右出现状况之后crsctl debug 看看是否是这个情况

Honcho

crsd.log 部分信息：
2011-10-20 14:37:04.787: [  CRSRES][12635]32StopResource: setting CLI values
2011-10-20 22:12:44.435: [  CRSAPP][591]32CheckResource error for ora.p570_svr3.vip error code = 1
2011-10-20 22:12:44.546: [  CRSRES][591]32In stateChanged, ora.p570_svr3.vip target is ONLINE
2011-10-20 22:12:44.546: [  CRSRES][591]32ora.p570_svr3.vip on p570_svr3 went OFFLINE unexpectedly
2011-10-20 22:12:44.547: [  CRSRES][591]32StopResource: setting CLI values
2011-10-20 22:12:44.703: [  CRSRES][591]32Attempting to stop `ora.p570_svr3.vip` on member `p570_svr3`
2011-10-20 22:12:45.307: [  CRSRES][591]32Stop of `ora.p570_svr3.vip` on member `p570_svr3` succeeded.
2011-10-20 22:12:45.307: [  CRSRES][591]32ora.p570_svr3.vip RESTART_COUNT=0 RESTART_ATTEMPTS=0
2011-10-20 22:12:45.330: [  CRSRES][591]32ora.p570_svr3.vip failed on p570_svr3 relocating.
2011-10-20 22:12:45.589: [  CRSRES][591]32StopResource: setting CLI values
2011-10-20 22:12:45.644: [  CRSRES][591]32Attempting to stop `ora.p570_svr3.LISTENER_P570_SVR3.lsnr` on member `p570_svr3`
2011-10-20 22:14:04.127: [  CRSRES][591]32Stop of `ora.p570_svr3.LISTENER_P570_SVR3.lsnr` on member `p570_svr3` succeeded.
2011-10-20 22:14:04.409: [  CRSRES][591]32Attempting to start `ora.p570_svr3.vip` on member `p570_svr1`
2011-10-20 22:14:07.412: [  CRSRES][591]32Start of `ora.p570_svr3.vip` on member `p570_svr1` succeeded.
2011-10-20 22:21:08.886: [  CRSRES][12716]32StopResource: setting CLI values
2011-10-20 22:21:08.934: [  CRSRES][12716]32Attempting to stop `ora.p570_svr3.vip` on member `p570_svr1`
2011-10-20 22:21:09.490: [  CRSRES][12716]32Stop of `ora.p570_svr3.vip` on member `p570_svr1` succeeded.
2011-10-20 22:21:09.496: [  CRSRES][12716]32startRunnable: setting CLI values
2011-10-20 22:21:09.497: [  CRSRES][12716]32Attempting to start `ora.p570_svr3.vip` on member `p570_svr3`
2011-10-20 22:21:12.106: [  CRSRES][12716]32Start of `ora.p570_svr3.vip` on member `p570_svr3` succeeded.
2011-10-20 22:21:12.387: [  CRSRES][12977]32startRunnable: setting CLI values
2011-10-20 22:21:12.415: [  CRSRES][12977]32Attempting to start `ora.p570_svr3.LISTENER_P570_SVR3.lsnr` on member `p570_svr3`
2011-10-20 22:21:15.388: [  CRSRES][12977]32Start of `ora.p570_svr3.LISTENER_P570_SVR3.lsnr` on member `p570_svr3` succeeded.

[ 本帖最后由 Honcho 于 2011-10-21 15:57 编辑 ]

Honcho

当开启一下debug 命令之后，vip 就不会down掉：
p570_svr3:/#crsctl debug log res "ora.p570_svr3.vip:5"
Set Resource Debug Module: ora.p570_svr3.vip  Level: 5

当关闭debug之后vip 就会down掉，有点意思！

yobyin

有结论吗， 我的也出现这种问题