rac 只能启动其中的一个节点

Mille

aix+9208+ha
现象：rac 只能启动其中的一个节点，正常启动节点一后，二节点在open阶段hang住，关闭节点一时，节点二正常启动，反过来先启动节点二也是一样的现象

Maclean Liu(刘相兵

节点1：

Wed Jul  3 15:16:32 2013
ARC0: Completed archiving  log 1 thread 1 sequence 4
Wed Jul  3 15:16:32 2013
Successfully onlined Undo Tablespace 1.
Wed Jul  3 15:16:32 2013
SMON: enabling tx recovery
Wed Jul  3 15:16:32 2013
Database Characterset is US7ASCII
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: ALTER DATABASE OPEN



节点2：

Wed Jul  3 15:18:05 2013
Starting ORACLE instance (normal)
Wed Jul  3 15:18:05 2013
Global Enqueue Service Resources = 167894, pool = 48
Wed Jul  3 15:18:05 2013
Global Enqueue Service Enqueues = 263421
Wed Jul  3 15:19:00 2013
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Wed Jul  3 15:19:22 2013
SCN scheme 3
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.8.0.
System parameters with non-default values:
  processes                = 8500
  sessions                 = 9355
  timed_statistics         = TRUE
  shared_pool_size         = 6224347136
  sga_max_size             = 60086602960
  shared_pool_reserved_size= 134217728
  large_pool_size          = 268435456
  java_pool_size           = 67108864
  lock_sga                 = TRUE
  spfile                   = /dev/rlv_cx_128M_04
  cluster_interconnects    = 192.168.100.14
  control_files            = /dev/rlv_cx_128M_01, /dev/rlv_cx_128M_02, /dev/rlv_cx_128M_03
  db_block_size            = 8192
  db_writer_processes      = 6
  db_cache_size            = 48704258048
  compatible               = 9.2.0.0.0
  log_archive_start        = TRUE
  log_archive_dest_2       = location=/chaxun_arch2
  db_file_multiblock_read_count= 16
  cluster_database         = TRUE
  cluster_database_instances= 2
  thread                   = 2
  fast_start_mttr_target   = 300
  instance_number          = 2
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS2
  undo_retention           = 23800
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  instance_name            = chaxun2
  session_cached_cursors   = 30
  job_queue_processes      = 10
  cursor_sharing           = SIMILAR
  hash_join_enabled        = TRUE
  hash_area_size           = 2097152
  background_dump_dest     = /oracle/admin/chaxun/bdump
  user_dump_dest           = /oracle/admin/chaxun/udump
  core_dump_dest           = /oracle/admin/chaxun/cdump
  sort_area_size           = 31457280
  db_name                  = chaxun
  open_cursors             = 3000
  _b_tree_bitmap_plans     = FALSE
  query_rewrite_enabled    = FALSE
  pga_aggregate_target     = 12442450944
Wed Jul  3 15:19:26 2013
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2 Version 1.0
PMON started with pid=2, OS id=1290364
DIAG started with pid=3, OS id=1204282
LMON started with pid=4, OS id=1105954
LMD0 started with pid=5, OS id=819394
LMS0 started with pid=6, OS id=1245302
LMS1 started with pid=7, OS id=1257528
LMS2 started with pid=8, OS id=1175774
LMS3 started with pid=9, OS id=688364
LMS4 started with pid=10, OS id=1085440
LMS5 started with pid=11, OS id=680140
LMS6 started with pid=12, OS id=475348
LMS7 started with pid=13, OS id=671994
LMS8 started with pid=14, OS id=1163266
LMS9 started with pid=15, OS id=1073394
DBW0 started with pid=16, OS id=962668
DBW1 started with pid=17, OS id=708622
DBW2 started with pid=18, OS id=676034
DBW3 started with pid=19, OS id=1229024
DBW4 started with pid=20, OS id=1118264
DBW5 started with pid=21, OS id=1052832
LGWR started with pid=22, OS id=1306680
CKPT started with pid=23, OS id=786558
SMON started with pid=24, OS id=741422
RECO started with pid=25, OS id=1151102
CJQ0 started with pid=26, OS id=704720
Wed Jul  3 15:19:30 2013
ARCH: STARTING ARCH PROCESSES
ARC0 started with pid=27, OS id=991372
ARC0: Archival started
ARC1 started with pid=28, OS id=745576
ARC1: Archival started
Wed Jul  3 15:19:30 2013
ARCH: STARTING ARCH PROCESSES COMPLETE
Wed Jul  3 15:19:30 2013
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no FAL' ARCHARC1: Thread not mounted
Wed Jul  3 15:19:30 2013
ARC0: Becoming the heartbeat ARCH
ARC0: Becoming the heartbeat ARCHARC0: Thread not mounted
Wed Jul  3 15:20:27 2013
alter database mount
Wed Jul  3 15:20:27 2013
lmon registered with NM - instance id 2 (internal mem no 1)
Wed Jul  3 15:20:28 2013
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0 1
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 0
0 GCS shadows traversed, 0 cancelled, 0 closed
0 GCS resources traversed, 0 cancelled
set master node info
Submitted all remote-enqueue requests
Update rdomain variables
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
0 GCS shadows traversed, 0 replayed, 0 unopened
Submitted all GCS remote-cache requests
0 write requests issued in 7042 GCS resources
0 PIs marked suspect, 0 flush PI msgs
Wed Jul  3 15:20:29 2013
Reconfiguration complete
LCK0 started with pid=30, OS id=1331420
Wed Jul  3 15:20:30 2013
ARC0: Thread not mounted
Wed Jul  3 15:20:33 2013
Successful mount of redo thread 2, with mount id 2533558731
Wed Jul  3 15:20:33 2013
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE).
Completed: alter database mount
Wed Jul  3 15:20:51 2013
alter database open
Wed Jul  3 15:20:51 2013
Picked Lamport scheme to generate SCNs
Wed Jul  3 15:20:51 2013
LGWR: Primary database is in CLUSTER CONSISTENT mode
Thread 2 opened at log sequence 1
  Current log# 5 seq# 1 mem# 0: /dev/rlv_cx2_1G_01
  Current log# 5 seq# 1 mem# 1: /dev/rlv_cx2_1G_012
Successful open of redo thread 2
Wed Jul  3 15:20:51 2013
SMON: enabling cache recovery
Wed Jul  3 15:20:52 2013
Successfully onlined Undo Tablespace 23.
Wed Jul  3 15:20:52 2013
SMON: enabling tx recovery
Wed Jul  3 15:20:52 2013
Database Characterset is US7ASCII


Wed Jul  3 15:16:32  1节点打开了
Wed Jul  3 15:20:52 2013  也打开了  而且打开时 实例1该活着

Wed Jul  3 15:20:27 2013
lmon registered with NM - instance id 2 (internal mem no 1)
Wed Jul  3 15:20:28 2013
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0 1
Global Resource Directory frozen

Mille

节点2 没打开，alter database open hang住了
打开的话日记会有Completed: ALTER DATABASE OPEN

Mille

Maclean Liu(刘相兵 发表于 2013-7-3 21:42
节点1：

Wed Jul  3 15:16:32 2013

节点2 没打开，alter database open hang住了
打开的话日记会有Completed: ALTER DATABASE OPEN

Maclean Liu(刘相兵

那么等待事件是什么， 做一个10046 level 12 trace open database的进程 并在HANG时做systemstate dump
压缩打包上传