Oracle数据库数据恢复、性能优化»论坛 › Oracle › Oracle数据库管理 › RAC一节点操作系统升级后cluster启动失败

0 积分	0 好友	11 主题

发消息

[升级] RAC一节点操作系统升级后cluster启动失败

1^#

发表于 2013-10-3 20:47:44 | 查看: 6357| 回复: 14

Hello Maclean:

我们一RAC数据库，关闭实例1后，将实例1所在的主机升级操作系统后，该实例的集群无法启动。

OS： SunOS bildbdc-1 5.10 Generic_150400-02 sun4v sparc sun4v
DB：oracle 11.2.0.3

grid alert 日志报：

2013-10-03 18:27:00.008
[client(8010)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.3/grid/log/bildbdc-1/client/crsctl_oracle.log.
2013-10-03 18:27:03.964
[client(8031)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2013-10-03 18:27:03.967
[client(8031)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.3/grid/log/bildbdc-1/client/crsctl_oracle.log.
2013-10-03 18:27:07.284
[client(8031)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2013-10-03 18:27:07.288
[client(8031)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.3/grid/log/bildbdc-1/client/crsctl_oracle.log.

cluster 信息：
root@bildbdc-1 # crsctl check crs
CRS-4639: Could not contact Oracle High Availability Services
root@bildbdc-1 # ps -ef | grep d.bin
  oracle  8878  8537 0 18:31:05 ?          0:00 /bin/sh /u01/app/11.2.0.3/grid/bin/cluvfy comp crs -display_status -n bildbdc-1
  oracle  9160  9045 0 18:31:20 ?          0:00 /u01/app/11.2.0.3/grid/bin/crsctl.bin query crs activeversion
root  2055    1 0 17:59:26 ?          0:00 /u01/app/11.2.0.3/grid/bin/ohasd.bin reboot
root  7300    1 0 18:21:57 ?          0:00 /u01/app/11.2.0.3/grid/bin/ohasd.bin exclusive
  oracle  9181  8901 0 18:31:23 ?          0:00 /u01/app/11.2.0.3/grid/bin/crsctl.bin query crs activeversion
root  9192  3193 0 18:31:26 pts/3    0:00 grep d.bin
root@bildbdc-1 #

分享0

收藏0 回复只看该作者道具举报

ccboceb7

2^#

发表于 2013-10-3 20:49:14

无法访问ocr 所在的磁盘组，这个不知道怎么解决?

回复只看该作者道具举报

Maclean Liu(刘相兵

3^#

发表于 2013-10-3 20:53:32

给出 GI的alert log 和cssd.log

回复只看该作者道具举报

xifenfei

4^#

发表于 2013-10-3 20:54:41

主机升级后，是不是asm disk对应的磁盘名称发生改变了？

回复只看该作者道具举报

ccboceb7

5^#

发表于 2013-10-3 21:05:02

这个是alert log 和cssd.log

alert_ocss_log.rar

174.67 KB, 下载次数: 880

回复只看该作者道具举报

Maclean Liu(刘相兵

6^#

发表于 2013-10-3 21:12:48

[cssd(6312)]CRS-1660:The CSS daemon shutdown has completed
2013-10-03 13:34:33.159
[gpnpd(6264)]CRS-2329:GPNPD on node bildbdc-1 shutdown.
2013-10-03 14:01:43.056
[client(1872)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2013-10-03 14:01:43.060
[client(1872)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.3/grid/log/bildbdc-1/client/crsctl_oracle.log.
2013-10-03 14:01:43.100
[client(1966)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2013-10-03 14:01:43.104
[client(1966)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.3/grid/log/bildbdc-1/client/crsctl_oracle.log.
2013-10-03 14:01:46.363
[client(1872)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2013-10-03 14:01:46.367
[client(1872)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0.3/grid/log/bildbdc-1/client/crsctl_oracle.log.
2013-10-03 14:01:46.393
[client(1966)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2013-10-03 14:01:46.411

重启节点后无法访问ASM中的ocr

需要 ASM的alert.log 和ohasd.log

回复只看该作者道具举报

ccboceb7

7^#

发表于 2013-10-3 21:17:29

上一个附件中的alertbildbdc-1.log是不是ASM的alert log ?

ohasd_log.rar

506.41 KB, 下载次数: 761

回复只看该作者道具举报

Maclean Liu(刘相兵

8^#

发表于 2013-10-3 21:29:52

1、你之后再次重启过 OS吗？
2、可以考虑这样做吗？ =》关掉2节点，并重启1节点并观察

3、 asm_diskstring 是多少？并 ls -l 这些设备
4、能否用nocrs方式启动ASM 并mount diskgroup?

回复只看该作者道具举报

ccboceb7

9^#

发表于 2013-10-3 21:48:51

1、开始起不来，又重启过OS还是一样
2、这是生产，现在不能同时都停
3、asm_diskstring 为空
4、我试过您您博客中的nocrs启动，一直不动

回复只看该作者道具举报

Maclean Liu(刘相兵

10^#

发表于 2013-10-3 21:54:03

在2节点上 select path from v$asm_disk;
将path信息放到下面的path.txt中
执行下面的脚本，并生成结果，对比2个节点上的结果
rm /tmp/kfed_DH.out /tmp/kfed_FS.out /tmp/kfed_BK.out /tmp/kfed_FD.out /tmp/kfed_DD.out /tmp/kfed_PST.out
for i in `cat path.txt`
do
echo $i >> /tmp/kfed_DH.out
kfed read $i >> /tmp/kfed_DH.out
echo $i >> /tmp/kfed_FS.out
kfed read $i blkn=1 >> /tmp/kfed_FS.out
echo $i >> /tmp/kfed_BK.out
kfed read $i aun=1 blkn=254 >> /tmp/kfed_BK.out
echo $i >> /tmp/kfed_FD.out
kfed read $i aun=2 blkn=1 >> /tmp/kfed_FD.out
echo $i >> /tmp/kfed_DD.out
kfed read $i aun=2 blkn=2 >> /tmp/kfed_DD.out
echo $i >> /tmp/kfed_PST.out
kfed read $i aun=1 blkn=2 >> /tmp/kfed_PST.out
done

复制代码

回复只看该作者道具举报

ccboceb7

11^#

发表于 2013-10-3 22:25:11

一样的结果，maclean liu

回复只看该作者道具举报

ccboceb7

12^#

发表于 2013-10-3 22:32:32

只是在 kfed_pst.out 文件最后两块盘都读取报错，但应该不是磁盘本身的问题吧.
cat kfed_PST.out
/dev/rdsk/c0t60014380125D89E50000800002740000d0s0
kfbh.endian:                         0 ; 0x000: 0x00
kfbh.hard:                         130 ; 0x001: 0x82
kfbh.type:                         13 ; 0x002: KFBTYP_PST_NONE
kfbh.datfmt:                         1 ; 0x003: 0x01
kfbh.block.blk:             2147483650 ; 0x004: blk=2 (indirect)
kfbh.block.obj:             2147483648 ; 0x008: disk=0
kfbh.check:                   8523011 ; 0x00c: 0x00820d03
kfbh.fcn.base:                      0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                      0 ; 0x014: 0x00000000
kfbh.spare1:                         0 ; 0x018: 0x00000000
kfbh.spare2:                         0 ; 0x01c: 0x00000000
ERROR!!!, failed to get the oracore error message
/dev/rdsk/c0t60014380125D89E50000800001440000d0s0
kfbh.endian:                         0 ; 0x000: 0x00
kfbh.hard:                         130 ; 0x001: 0x82
kfbh.type:                         13 ; 0x002: KFBTYP_PST_NONE
kfbh.datfmt:                         1 ; 0x003: 0x01
kfbh.block.blk:             2147483650 ; 0x004: blk=2 (indirect)
kfbh.block.obj:             2147483648 ; 0x008: disk=0
kfbh.check:                   8523011 ; 0x00c: 0x00820d03
kfbh.fcn.base:                      0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                      0 ; 0x014: 0x00000000
kfbh.spare1:                         0 ; 0x018: 0x00000000
kfbh.spare2:                         0 ; 0x01c: 0x00000000
ERROR!!!, failed to get the oracore error message

回复只看该作者道具举报

ccboceb7

13^#

发表于 2013-10-5 15:32:40

hi , maclean liu:

我定位到是/etc/init.d/init.ohasd run 不能随机自动启动。
跟你的这篇文章类似：
11g的ohasd无法随机启动
http://feed.askmaclean.com/archives/11g%E7%9A%84ohasd%E6%97%A0%E6%B3%95%E9%9A%8F%E6%9C%BA%E5%90%AF%E5%8A%A8.html

我检查过了配置文件如下：
bash-3.2$ cat /etc/inittab
# Copyright 2004 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# The /etc/inittab file controls the configuration of init(1M); for more
# information refer to init(1M) and inittab(4).  It is no longer
# necessary to edit inittab(4) directly; administrators should use the
# Solaris Service Management Facility (SMF) to define services instead.
# Refer to smf(5) and the System Administration Guide for more
# information on SMF.
#
# For modifying parameters passed to ttymon, use svccfg(1m) to modify
# the SMF repository. For example:
#
#    # svccfg
#    svc:> select system/console-login
#    svc:/system/console-login> setprop ttymon/terminal_type = "xterm"
#    svc:/system/console-login> exit
#
#ident  "@(#)inittab 1.41 04/12/14 SMI"
ap::sysinit:/sbin/autopush -f /etc/iu.ap
sp::sysinit:/sbin/soconfig -f /etc/sock2path
smf::sysinit:/lib/svc/bin/svc.startd >/dev/msglog 2<>/dev/msglog </dev/console
p3:s1234:powerfail:/usr/sbin/shutdown -y -i5 -g0 >/dev/msglog 2<>/dev/msglog

h1:3:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
pt:s1234:powerfail:/lib/svc/method/installupdates lock

回复只看该作者道具举报

ccboceb7

14^#

发表于 2013-10-5 15:35:17

通过手动执行：
nohup /etc/init.ohasd run &
后，可以将GI启动起来。

回复只看该作者道具举报

Maclean Liu(刘相兵

15^#

发表于 2013-10-5 19:19:53

1、这篇文章是小荷写的
2、他的问题的核心是 11.2.0.1 对 Linux 6是不认证的，在你这里不存在该问题
3、如果你还有研究该问题我建议你对hasd做trace

回复只看该作者道具举报

返回列表

		自动登录	找回密码
密码			注册