RAC环境下OGG的HA问题请教！

ylw6006

前面搭建好了RAC环境下(source)同单实例数据库的OGG单向同步！refer:http://ylw6006.blog.51cto.com/470441/932308
然后配置source端的HA功能，参考这个文章配置：http://www.rickyzhu.com/2311_config-ha-goldengate.html
配置步骤如下：
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl stat res -p |grep -ie .network |head -4
START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network)
STOP_DEPENDENCIES=hard(ora.net1.network)
NAME=ora.net1.network
TYPE=ora.network.type
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/appvipcfg create -network=1 -ip=192.168.1.150 -vipname=oggvip -user=root
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl setperm resource oggvip -u user:oracle:r-x
[oracle@rac1 ~]$ crsctl start resource oggvip
CRS-2672: Attempting to start 'oggvip' on 'rac1'
CRS-2676: Start of 'oggvip' on 'rac1' succeeded
[oracle@rac1 ~]$ crsctl status resource oggvip
NAME=oggvip
TYPE=app.appvip_net1.type
TARGET=ONLINE
STATE=ONLINE on rac1
[oracle@rac1 ~]$ /sbin/ip a |grep 192.168.1.150
    inet 192.168.1.150/24 brd 192.168.1.255 scope global secondary eth0:2
[oracle@rac1 ~]$ chmod +x /vol2/ogg/11gr2_ogg_action.scr  （帖子篇幅限制，脚本内容贴在2楼！）
[oracle@rac1 ~]$ crsctl add resource oggapp -type cluster_resource \
-attr "ACTION_SCRIPT=/vol2/ogg/11gr2_ogg_action.scr, \
CHECK_INTERVAL=30, START_DEPENDENCIES='hard(oggvip,ora.rac.db) \
pullup(oggvip)', STOP_DEPENDENCIES='hard(oggvip)'"
[oracle@rac1 ~]$ crsctl status resource oggapp
NAME=oggapp
TYPE=cluster_resource
TARGET=OFFLINE
STATE=OFFLINE
[oracle@rac1 ~]$ netstat -ntpl |grep 7809 （source端的mgr未启动）
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
[oracle@dg3 ogg]$ ./ggsci  （target端的mgr和replicat进程已经启动）
GGSCI (dg3.yang.com) 1> info all
Program     Status      Group       Lag at Chkpt  Time Since Chkpt
MANAGER     RUNNING                                          
REPLICAT    RUNNING     TESTRPT     00:00:00      00:00:00   

[oracle@rac1 ~]$ crs_stat -t -v oggvip
Name           Type           R/RA   F/FT   Target    State     Host        
----------------------------------------------------------------------
oggvip         app....t1.type 0/0    0/0    OFFLINE   OFFLINE               
[oracle@rac1 ~]$ crs_stat -t -v oggapp
Name           Type           R/RA   F/FT   Target    State     Host        
----------------------------------------------------------------------
oggapp         clus...esource 0/1    0/0    OFFLINE   OFFLINE  

[oracle@rac1 ~]$ crsctl start resource  oggapp  (启动oggapp失败)
CRS-2672: Attempting to start 'oggvip' on 'rac2'
CRS-2676: Start of 'oggvip' on 'rac2' succeeded
CRS-2672: Attempting to start 'oggapp' on 'rac2'
CRS-2674: Start of 'oggapp' on 'rac2' failed
CRS-2679: Attempting to clean 'oggapp' on 'rac2'
CRS-2681: Clean of 'oggapp' on 'rac2' succeeded
CRS-2673: Attempting to stop 'oggvip' on 'rac2'
CRS-2677: Stop of 'oggvip' on 'rac2' succeeded
CRS-2563: Attempt to start resource 'oggapp' on 'rac2' has failed. Will re-retry on 'rac1' now.
CRS-2672: Attempting to start 'oggvip' on 'rac1'
CRS-2676: Start of 'oggvip' on 'rac1' succeeded
CRS-2672: Attempting to start 'oggapp' on 'rac1'
CRS-2674: Start of 'oggapp' on 'rac1' failed
CRS-2679: Attempting to clean 'oggapp' on 'rac1'
CRS-2681: Clean of 'oggapp' on 'rac1' succeeded
CRS-2632: There are no more servers to try to place resource 'oggapp' on that would satisfy its placement policy
CRS-4000: Command Start failed, or completed with errors.

[oracle@rac1 ~]$ crsctl stop resource oggapp
[oracle@rac1 ~]$ crsctl stop resource oggvip
CRS-2673: Attempting to stop 'oggvip' on 'rac1'
CRS-2677: Stop of 'oggvip' on 'rac1' succeeded
[oracle@rac1 ~]$ crsctl stop resource oggapp
CRS-2500: Cannot stop resource 'oggapp' as it is not running
CRS-4000: Command Stop failed, or completed with errors.
[oracle@rac1 ~]$ crsctl start resource oggapp
CRS-2672: Attempting to start 'oggvip' on 'rac1'
CRS-2676: Start of 'oggvip' on 'rac1' succeeded
CRS-2672: Attempting to start 'oggapp' on 'rac1'
CRS-2674: Start of 'oggapp' on 'rac1' failed
CRS-2679: Attempting to clean 'oggapp' on 'rac1'
CRS-2681: Clean of 'oggapp' on 'rac1' succeeded
CRS-2527: Unable to start 'oggapp' because it has a 'hard' dependency on 'oggvip'
CRS-2525: All instances of the resource 'oggvip' are already running; relocate is not allowed because the force option was not specified
CRS-4000: Command Start failed, or completed with errors.

[oracle@rac1 ~]$ crs_stat -t -v  (在这里发现rac1的ons,gsd服务异常，使用crs_start命令online后继续测试，同样无法启动oggapp)
Name           Type           R/RA   F/FT   Target    State     Host        
----------------------------------------------------------------------
oggapp         clus...esource 0/1    0/0    ONLINE    OFFLINE               
oggvip         app....t1.type 0/0    0/0    ONLINE    ONLINE    rac1        
ora.ACFS.dg    ora....up.type 0/5    0/     ONLINE    ONLINE    rac1        
ora.CRS.dg     ora....up.type 0/5    0/     ONLINE    ONLINE    rac1        
ora.DATA.dg    ora....up.type 0/5    0/     ONLINE    ONLINE    rac1        
ora.FRA.dg     ora....up.type 0/5    0/     ONLINE    ONLINE    rac1        
ora....ER.lsnr ora....er.type 0/5    0/     ONLINE    ONLINE    rac2        
ora....N1.lsnr ora....er.type 0/5    0/0    ONLINE    ONLINE    rac2        
ora....N2.lsnr ora....er.type 0/5    0/0    ONLINE    ONLINE    rac2        
ora....N3.lsnr ora....er.type 0/5    0/0    ONLINE    ONLINE    rac2        
ora.asm        ora.asm.type   0/5    0/     ONLINE    ONLINE    rac1        
ora.cvu        ora.cvu.type   0/5    0/0    ONLINE    ONLINE    rac2        
ora.gsd        ora.gsd.type   0/5    0/     OFFLINE   OFFLINE               
ora....network ora....rk.type 5/5    0/     ONLINE    ONLINE    rac2        
ora.oc4j       ora.oc4j.type  0/1    0/2    ONLINE    ONLINE    rac2        
ora.ons        ora.ons.type   0/3    0/     ONLINE    ONLINE    rac2        
ora.rac.db     ora....se.type 0/1    0/1    ONLINE    ONLINE    rac1        
ora....SM1.asm application    0/5    0/0    ONLINE    ONLINE    rac1        
ora....C1.lsnr application    0/5    0/0    ONLINE    OFFLINE               
ora.rac1.gsd   application    0/5    0/0    OFFLINE   OFFLINE               
ora.rac1.ons   application    0/3    0/0    ONLINE    OFFLINE               
ora.rac1.vip   ora....t1.type 0/0    0/0    ONLINE    ONLINE    rac2        
ora....SM2.asm application    0/5    0/0    ONLINE    ONLINE    rac2        
ora....C2.lsnr application    0/5    0/0    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    0/5    0/0    OFFLINE   OFFLINE               
ora.rac2.ons   application    0/3    0/0    ONLINE    ONLINE    rac2        
ora.rac2.vip   ora....t1.type 0/0    0/0    ONLINE    ONLINE    rac2        
ora....ry.acfs ora....fs.type 0/5    0/     ONLINE    ONLINE    rac1        
ora.scan1.vip  ora....ip.type 0/0    0/0    ONLINE    ONLINE    rac2        
ora.scan2.vip  ora....ip.type 0/0    0/0    ONLINE    ONLINE    rac2        
ora.scan3.vip  ora....ip.type 0/0    0/0    ONLINE    ONLINE    rac2  

[oracle@rac1 ~]$ cd /vol2/ogg/  (直接在ggsci里面启动mgr和extract group正常，source和target间的数据同步也正常)
[oracle@rac1 ogg]$ ./ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.1 OGGCORE_11.2.1.0.1_PLATFORMS_120423.0230_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Apr 23 2012 08:32:14
Copyright (C) 1995, 2012, Oracle and/or its affiliates. All rights reserved.
GGSCI (rac1.yang.com) 1> start mgr
Manager started.
GGSCI (rac1.yang.com) 4> info all
Program     Status      Group       Lag at Chkpt  Time Since Chkpt
MANAGER     RUNNING                                          
EXTRACT     RUNNING     TESTEXT     00:00:00      00:00:02   
EXTRACT     RUNNING     TESTPUMP    00:00:00      00:00:00   
[oracle@rac1 ogg]$ crs_stat -t -v oggapp
Name           Type           R/RA   F/FT   Target    State     Host        
----------------------------------------------------------------------
oggapp         clus...esource 0/1    0/0    ONLINE    OFFLINE     

在我的配置中oggapp resource不能正常启动，请指点下排错的思路，不太熟悉11g rac gi 的管理，谢谢！

[ 本帖最后由 ylw6006 于 2012-7-18 10:43 编辑 ]

ylw6006

[oracle@rac1 ~]$ cat /vol2/ogg/11gr2_ogg_action.scr  (action脚本内容）
#!/bin/sh
#set the Oracle Goldengate installation directory
export GGS_HOME=/vol2/ogg
#set the oracle home to the database to ensure GoldenGate will get the
#right environment settings to be able to connect to the database
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db1
#specify delay after start before checking for successful start
start_delay_secs=5
#Include the GoldenGate home in the library path to start GGSCI
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:${GGS_HOME}:${LD_LIBRARY_PATH}
#check_process validates that a manager process is running at the PID
#that GoldenGate specifies.

check_process () {
if ( [ -f "${GGS_HOME}/dirpcs/MGR.pcm" ] )
then
  pid=`cut -f8 "${GGS_HOME}/dirpcs/MGR.pcm"`
  if [ ${pid} = `ps -e |grep ${pid} |grep mgr |cut -d " " -f2` ]
  then
    #manager process is running on the PID exit success
    exit 0
  else
  if [ ${pid} = `ps -e |grep ${pid} |grep mgr |cut -d " " -f1` ]
  then
    #manager process is running on the PID exit success
    exit 0
  else
    #manager process is not running on the PID
    exit 1
  fi
fi
else
  #manager is not running because there is no PID file
  exit 1
fi
}

#call_ggsci is a generic routine that executes a ggsci command
call_ggsci () {
  ggsci_command=$1
  ggsci_output=`${GGS_HOME}/ggsci < < EOF
  ${ggsci_command}
  exit
  EOF`
}

case $1 in
'start')
  #start manager
  call_ggsci 'start manager'
  #there is a small delay between issuing the start manager command
  #and the process being spawned on the OS. wait before checking
  sleep ${start_delay_secs}
  #check whether manager is running and exit accordingly
  check_process
  ;;
'stop')
  #attempt a clean stop for all non-manager processes
  #call_ggsci 'stop er *'
  #ensure everything is stopped
  call_ggsci 'stop er *!'
  #call_ggsci 'kill er *'
  #stop manager without (y/n) confirmation
  call_ggsci 'stop manager!'
  #exit success
  exit 0
  ;;
'check')
  check_process
  ;;
'clean')
  #attempt a clean stop for all non-manager processes
  #call_ggsci 'stop er *'
  #ensure everything is stopped
  #call_ggsci 'stop er *!'
  #in case there are lingering processes
  call_ggsci 'kill er *'
  #stop manager without (y/n) confirmation
  call_ggsci 'stop manager!'
  #exit success
  exit 0
  ;;
'abort')
  #ensure everything is stopped
  call_ggsci 'stop er *!'
  #in case there are lingering processes
  call_ggsci 'kill er *'
  #stop manager without (y/n) confirmation
  call_ggsci 'stop manager!'
  #exit success
  exit 0
  ;;
esac

clevernby

/vol2/ogg/11gr2_ogg_action.scr  添加些日志信息
先看看CRS能正常调用脚本吗
如果不能调用，则排查clusterware
如果可以则看看脚本是不是有不对的地方

另外start_delay_secs变量调大些，8-10s比较合适

[ 本帖最后由 clevernby 于 2012-7-18 12:37 编辑 ]

ylw6006

对比了下ha-goldengate-whitepaper-128197.pdf文档，应该不是脚本的问题！

我的rac环境是gi是用户是grid，ogg的用户是oracle,可能和这个有关系！
As root, allow the Oracle Grid infrastructure software owner (e.g. oracle) to run the script
to start the VIP.
GRID_HOME/bin/crsctl setperm resource mvggatevip -u user:oracle:r-x

If your Oracle GoldenGate software owner (e.g. mvandewiel) is not the same as the Oracle
Grid infrastructure software owner, then you must set the ownership of the application to the
Oracle GoldenGate software owner. Run this command as root.
GRID_HOME/bin/crsctl setperm resource ggateapp -o mvandewiel

也或许是ASM的问题！
To prevent this scenario with Oracle ASM 11g Release 2 follow the following steps:
1.  Use another dependency to a local resource ora.asm. This resource is available if the
ASM instance is running. This introduces a slight change to the crsctl add
resource command (changes highlighted):
GRID_HOME/bin/crsctl add resource ggateapp \
     -type cluster_resource \
     -attr  
"ACTION_SCRIPT=/mnt/acfs/oracle/grid/11gr2_gg_action.scr,
CHECK_INTERVAL=30, START_DEPENDENCIES='hard(mvggatevip,ora.asm)
pullup(mvggatevip)', STOP_DEPENDENCIES='hard(mvggatevip)'"

All other steps to configure Oracle GoldenGate with Oracle Clusterware remain the
same, and all configurations discussed in the section Oracle GoldenGate with Oracle
Clusterware will work.

下次按白皮书的example测试一下！

文档地址：http://www.oracle.com/technetwork/middleware/goldengate/overview/ha-goldengate-whitepaper-128197.pdf，方便后来人！

[ 本帖最后由 ylw6006 于 2012-7-18 18:00 编辑 ]

ylw6006

原帖由 clevernby 于 2012-7-18 12:34 发表
/vol2/ogg/11gr2_ogg_action.scr  添加些日志信息
先看看CRS能正常调用脚本吗
如果不能调用，则排查clusterware
如果可以则看看脚本是不是有不对的地方

另外start_delay_secs变量调大些，8-10s比较合适 ...

oh my god,今晚测试了下，还真的是shell脚本的问题！
call_ggsci () {
  ggsci_command=$1
  ggsci_output=`${GGS_HOME}/ggsci < < EOF
  ${ggsci_command}
  exit
  EOF`

明显here document两个尖括号中间多了个空格，这么简单的问题居然被我弄的复杂了！因为是直接复制的代码，所以没去检查，要检讨下，总是在最简单的地方犯错！

ShineCQY

自问自答......