HP安腾机器上9204升级到9208后，数据库启动有问题

chinadm123

$ sqlplus '/as sysdba'

SQL*Plus: Release 9.2.0.8.0 - Production on ?lh 10? 24 20:49:33 2011

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Connected.
SQL> startup
ORA-01081: cannot start already-running ORACLE - shut it down first
SQL> conn /as sysdba
Connected to an idle instance.
SQL>
------------------------------------------------------------------------------------------------------
alert的报错信息：
Errors in file /u01/app/oracle/admin/test/udump/test_ora_14042.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
Mon Oct 24 20:32:58 2011
Errors in file /u01/app/oracle/admin/test/udump/test_ora_14061.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
Mon Oct 24 20:33:13 2011
Errors in file /u01/app/oracle/admin/test/udump/test_ora_14063.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
Starting ORACLE instance (normal)
Mon Oct 24 20:42:25 2011
Starting ORACLE instance (normal)
Mon Oct 24 20:42:25 2011
Global Enqueue Service Resources = 64, pool = 2
Mon Oct 24 20:42:25 2011
Global Enqueue Service Enqueues = 128
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 3
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.8.0.
System parameters with non-default values:
  processes                = 150
  timed_statistics         = TRUE
  shared_pool_size         = 117440512
  large_pool_size          = 16777216
  java_pool_size           = 117440512
  control_files            = /u01/app/oracle/oradata/test/control01.ctl, /u01/app/oracle/oradata/test/control02.ctl, /u01/app/oracle/oradata/test/control03.ctl
  db_block_size            = 8192
  db_cache_size            = 33554432
  compatible               = 9.2.0.0.0
  log_archive_start        = TRUE
  log_archive_dest_1       = LOCATION=/u01/app/oracle/oradata/test/archive
  log_archive_format       = %t_%s.dbf
  db_file_multiblock_read_count= 16
  fast_start_mttr_target   = 300
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS1
  undo_retention           = 10800
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  instance_name            = test
  dispatchers              = (PROTOCOL=TCP) (SERVICE=testXDB)
  job_queue_processes      = 10
  hash_join_enabled        = TRUE
  background_dump_dest     = /u01/app/oracle/admin/test/bdump
  user_dump_dest           = /u01/app/oracle/admin/test/udump
  core_dump_dest           = /u01/app/oracle/admin/test/cdump
  sort_area_size           = 524288
  db_name                  = test
  open_cursors             = 300
  star_transformation_enabled= FALSE
  query_rewrite_enabled    = FALSE
  pga_aggregate_target     = 25165824
  aq_tm_processes          = 1
-------------------------------------------------------------------------------------
$ ps -ef|grep ora_
  oracle  1950  1895  1 20:52:55 console   0:00 grep ora_
$
------------------------------------------------------------------------
$ ipcs
IPC status from /dev/kmem as of Mon Oct 24 20:51:36 2011
T         ID     KEY        MODE        OWNER     GROUP
Message Queues:
q          0 0x3c1c2b60 -Rrw--w--w-      root      root
q          1 0x3e1c2b60 --rw-r--r--      root      root
Shared Memory:
m          0 0x411c2b62 --rw-rw-rw-      root      root
m          1 0x4e0c0002 --rw-rw-rw-      root      root
m          2 0x412007ba --rw-rw-rw-      root      root
m    1638403 0x8b250904 --rw-r-----    oracle       dba
Semaphores:
s          0 0x4f1c0140 --ra-------      root      root
s          1 0x411c2b62 --ra-ra-ra-      root      root
s          2 0x4e0c0002 --ra-ra-ra-      root      root
s          3 0x412007ba --ra-ra-ra-      root      root
s          4 0x00446f6e --ra-r--r--      root      root
s          5 0x00446f6d --ra-r--r--      root      root
s          6 0x01090522 --ra-r--r--      root      root
s          7 0x411c04e2 --ra-ra-ra-      root      root
s          8 0x61200a1d --ra-ra-ra-      root      root
s          9 0x73200a1d --ra-ra-ra-      root      root
s         10 0x70200a1d --ra-ra-ra-      root      root
s         11 0x69200a1d --ra-ra-ra-      root      root
s         12 0x75200a1d --ra-ra-ra-      root      root
s         13 0x63200a1d --ra-ra-ra-      root      root
s         14 0x64200a1d --ra-ra-ra-      root      root
s         15 0x66200a1d --ra-ra-ra-      root      root
s         16 0x6c200a1d --ra-ra-ra-      root      root
s         17 0x6d200a1d --ra-ra-ra-      root      root
s         18 0x6f200a1d --ra-ra-ra-      root      root
s         19 0x410c056a --ra-ra-ra-      root      root
s         20 0x00000001 --ra-ra-ra-      root      root
s      49173 0xc29855dc --ra-r-----    oracle       dba
$
再次请高手有空时帮小弟分析一下原因，能不能给些建议？多谢版主！:P

Honcho

以下可做参考，不知道给不给力！

[ 本帖最后由 Honcho 于 2011-10-24 22:19 编辑 ]

Maclean Liu(刘相兵

尝试以下手段

1. ipcrm -m 1638403
   
2. 
   
3. ipcrm -s 49173
   
4. 
   
5. 
   
6. ipcs

复制代码

Maclean Liu(刘相兵

Mos 文档 ORA-1081: Starting Instance 介绍了该问题，引用如下

1. ORA-1081: Starting Instance [ID 1010214.6]
   
2.           Modified 07-APR-2011     Type PROBLEM     Status PUBLISHED          
   
3. 
   
4. Applies to:
   
5. Oracle Server - Enterprise Edition - Version: 8.1.7.4 to 11.2.0.2.0 - Release: 8.1.7 to 11.2
   
6. Generic UNIX
   
7. Symptoms
   
8. 
   
9. An attempt to start an instance that is not running results in
   
10. the following error:
    
11. 
    
12.   ORA-01081: "cannot start already-running ORACLE - shut it down first"
    
13. 
    
14. 
    
15. When checking for the instance background processes, none exist.
    
16. 
    
17. 
    
18. Cause
    
19. 
    
20. Orphaned shared memory segments and semaphores did not get cleared properly
    
21. from the last instance shutdown.
    
22. 
    
23. 
    
24. Solution
    
25. 
    
26. 
    
27. Remove the orphaned shared memory segments and semaphores via the ipc command specific to the platform.
    
28. 
    
29. Example:
    
30. 
    
31.         ipcrm -m       - for shared memory
    
32.         ipcrm -s       - for semaphores
    
33. 
    
34. The challenge is to find the ID of the orphaned shared memory and  
    
35. semaphores among the IPC facilities, that are used by other Oracle  
    
36. instances and owned by the same user on the same server.
    
37. 
    
38. Refer to Document 18657.1 ORA 1081 cannot start already-running Oracle - shut it down first
    
39. 
    
40. 
    
41. OR
    
42. 
    
43. Shutdown all the instances belonging to the user. Next, run "ipcs -b" to find the remaining IPC
    
44. facilities, owned by the same user, and remove them.  However, if you're running multiple production
    
45. instances, this may not be  acceptable.
    
46. 
    
47. If so, the following approach to solving the problem can be utilized:
    
48. 
    
49. 1. Run
    
50.           % ipcs -a > /tmp/ipcs_before.out  
    
51. 
    
52. 2. Use SQL*Plus to run a simple query against each Oracle  
    
53.    instance running on the server (e.g., "select * from dual;").
    
54. 
    
55. 3. Run
    
56.           % ipcs -a > /tmp/ipcs_after.out  
    
57. 
    
58. 4. Compare the files "/tmp/ipcs_before.out" and "/tmp/ipcs_after.out"  
    
59.    to find the shared memory segments whose LPID value has not changed.   
    
60.    These are the orphaned shared memory segments you are looking for:  
    
61. 
    
62.    LPID shows the process ID of the last process to attach or detach  
    
63.    the shared memory segment.  
    
64. 
    
65.    Since by running a query in each surviving instance you touched the  
    
66.    shared memory belonging to those instances, the untouched shared  
    
67.    memory segments will be the orphaned ones.
    
68. 
    
69. 5. After identifying the orphaned shared memory segments, you can find  
    
70.    the orphaned semaphores by locating the semaphores with the same  
    
71.    value of CTIME in "ipcs -a" output as the value of CTIME for the  
    
72.    orphaned shared memory segments: CTIME shows the time when the  
    
73.    associated entry was created or changed.
    
74. 
    
75.    NOTE:  This step may not be necessary on the platforms implementing
    
76.           post-wait kernel extension (AT&T Unix, AIX), when semaphores  
    
77.           are not used by Oracle at all.
    
78. 
    
79. 
    
80. If you have started one instance, you must reset the instance name before
    
81. starting up the next instance.                                                
    
82.       
    
83.                                                                      
    
84. For example, for the 2 instances:                                             
    
85.                        
    
86. ORACLE_SID prod                                                               
    
87. ORACLE_SID test                                                               
    
88.   
    
89.                                                                               
    
90. With ORACLE_SID set to prod:                                                   
    
91.   
    
92. % sqldba mode=line                                                            
    
93.   
    
94. sqldba> connect internal           (connects you to prod)           
    
95. sqldba> startup                    (starts up prod)                     
    
96. sqldba> startup pfile=inittest.ora (will still attempt to startup prod)        
    
97.                                                                               
    
98.   
    
99. To change instances from within SQL*DBA, use SET INSTANCE                     
    
100.   
     
101. Example (starting up 2 instances) with ORACLE_SID set to prod:                 
     
102.                                     
     
103. % sqldba mode=line                                                            
     
104.   
     
105. sqldba> connect internal           (connects you to prod)                 
     
106. sqldba> startup                    (starts up prod)           
     
107. sqldba> disconnect                 (disconnects you from prod)      
     
108. sqldba> set instance p:test   
     
109. sqldba> connect internal            
     
110. sqldba> startup                    (starts up test)

复制代码

chinadm123

版主，根据你的提示使用了ipcrm命令之后貌似起到了一定的效果，但又出现了其它问题，经过了若干次折腾之后新的问题就让我无法前进了，报错信息如下 ：
SQL> startup
ORACLE instance started.

Total System Global Area  320290568 bytes
Fixed Size                   724744 bytes
Variable Size             285212672 bytes
Database Buffers           33554432 bytes
Redo Buffers                 798720 bytes
ORA-29701: unable to connect to Cluster Manager
//////////////////////////////////////////////////////////////////////////////////////////////
alert中的信息如下：
$ cat alert_test.log
Tue Oct 25 10:44:13 2011
Starting ORACLE instance (normal)
Tue Oct 25 10:44:13 2011
Global Enqueue Service Resources = 64, pool = 2
Tue Oct 25 10:44:13 2011
Global Enqueue Service Enqueues = 128
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 3
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.8.0.
System parameters with non-default values:
  processes                = 150
  timed_statistics         = TRUE
  shared_pool_size         = 117440512
  large_pool_size          = 16777216
  java_pool_size           = 117440512
  control_files            = /u01/app/oracle/oradata/test/control01.ctl, /u01/app/oracle/oradata/test/control02.ctl, /u01/app/oracle/oradata/test/control03.ctl
  db_block_size            = 8192
  db_cache_size            = 33554432
  compatible               = 9.2.0.0.0
  log_archive_start        = TRUE
  log_archive_dest_1       = LOCATION=/u01/app/oracle/oradata/test/archive
  log_archive_format       = %t_%s.dbf
  db_file_multiblock_read_count= 16
  fast_start_mttr_target   = 300
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS1
  undo_retention           = 10800
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  instance_name            = test
  dispatchers              = (PROTOCOL=TCP) (SERVICE=testXDB)
  job_queue_processes      = 10
  hash_join_enabled        = TRUE
  background_dump_dest     = /u01/app/oracle/admin/test/bdump
  user_dump_dest           = /u01/app/oracle/admin/test/udump
  core_dump_dest           = /u01/app/oracle/admin/test/cdump
  sort_area_size           = 524288
  db_name                  = test
  open_cursors             = 300
  star_transformation_enabled= FALSE
  query_rewrite_enabled    = FALSE
  pga_aggregate_target     = 25165824
  aq_tm_processes          = 1
Tue Oct 25 10:44:13 2011
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2 Version 1.0
PMON started with pid=2, OS id=2387
DIAG started with pid=3, OS id=2389
LMON started with pid=4, OS id=2391
LMD0 started with pid=5, OS id=2393
DBW0 started with pid=6, OS id=2395
LGWR started with pid=7, OS id=2397
CKPT started with pid=8, OS id=2399
SMON started with pid=9, OS id=2401
RECO started with pid=10, OS id=2403
CJQ0 started with pid=11, OS id=2405
QMN0 started with pid=12, OS id=2407
Tue Oct 25 10:44:17 2011
starting up 1 shared server(s) ...
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
ARCH: STARTING ARCH PROCESSES
ARC0 started with pid=15, OS id=2413
ARC0: Archival started
ARC1 started with pid=16, OS id=2415
ARC1: Archival started
Tue Oct 25 10:44:17 2011
ARCH: STARTING ARCH PROCESSES COMPLETE
Tue Oct 25 10:44:17 2011
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no FAL' ARCHARC1: Thread not mounted
Tue Oct 25 10:44:17 2011
ARC0: Becoming the heartbeat ARCH
ARC0: Becoming the heartbeat ARCHARC0: Thread not mounted
Tue Oct 25 10:44:17 2011
ALTER DATABASE   MOUNT
Tue Oct 25 10:44:17 2011
ORA-29701 signalled during: ALTER DATABASE   MOUNT...
再次麻烦版主指点！
我自已也看了一些帖子有的说不让/etc/init.d/xx  文件中的init.cssd启动，可是我的/etc/下面没有init.d文件。
也有使用localconfig的说法，可是我的$ORACLE_HOME/bin下面没有这个命令。

[ 本帖最后由 chinadm123 于 2011-10-25 10:53 编辑 ]

kevinlin.ora

10g的文档

How to Check and Enable/Disable Oracle Binary Options [ID 948061.1]
          修改时间 06-SEP-2011     类型 HOWTO     状态 PUBLISHED         

In this Document
  Goal
  Solution
  References

Applies to:
Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.2.0.2 - Release: 10.2 to 11.2
Other UNIX
Goal

This is for UNIX Only ! Does NOT work on Windows !

This requires advanced skills to make the changes and perform the relink.

The goal of this article is 2 parts:

    Verify if a given option is turned on or not
    Enable/Disable given options

This applies to the following options:

    RAC (Real Appliction Cluster)
    RAT (Real Application Testing)
    OLS (Oracle Label Security)
    DV (Database Vault)

Last, some unusual items to be aware of

    ASM (Automated Storage Management)
    OLAP (Oracle OLAP)
    PART (Oracle Partitioning)
    CTX (Context Management Text)


Solution

Before you can verify options and/or enable/disable them, you will need to:

    % cd $ORACLE_HOME/rdbms/lib
    % ar -t libknlopt.a | grep -c {file}

Where {file} is:

    RAC - kcsm.o
    RAT - kecwr.o
    OLS - kzlibac.o
    DV - kzvidv.o
    ASM - kfon.o
    OLAP - xsyeolap.o
    PART - kkpoban.o
    CTX - kciwcx.o

An example:

    Non-AIX   % ar -t libknlopt.a | grep -c kcsm.o
    AIX       % ar -X64 -t libknlopt.a | grep -c kcsm.o

To enable or disable options, please look at the following commands:

    % make -f ins_rdbms.mk {option}

Where {option} is:

    RAC - rac_on / rac_off
    RAT - rat_on / rat_off
    OLS - lbac_on / lbac_off
    DV - dv_on / dv_off
    ASM - asm_on / asm_off
    OLAP - olap_on / olap_off
    PART - part_on / part_off
    CTX - ctx_on / ctx_off
    DM - dm_on / dm_off

An example:

    % make -f ins_rdbms.mk asm_on

Unusual Items to be Aware Of

    The V$OPTION is unchanged and you will need to refer to other documentation to change it in order to complete the changes to the options. This include but not limited to running sql scripts which changes the DB Catalog.

Lastly, the Oracle binary will need to be recompiled

    % make -f rdbms/lib/ins_rdbms.mk ioracle

Additional information is the chopt utility

This note can be made applicable for all platforms by adopting the chopt utility. Besides working on Windows platforms, another useful feature is the fact that on UNIX/Linux platforms, it neatly displays the commands that need to be executed:
% chopt disable rat Writing to <ORACLE_HOME>/install/disable_rat.log...
%s_unixOSDMakePath% -f /grdbms/64bit/patchdb/app/oracle/product/p11201b/rdbms/lib/ins_rdbms.mk rat_off
%s_unixOSDMakePath% -f /grdbms/64bit/patchdb/app/oracle/product/p11201b/rdbms/lib/ins_rdbms.mk ioracle

Where %s_unixOSDMakePath% = make

Note that the relink commands above are not actually being run by chopt. The user has to manually run them.

Warning: switching on ASM should only be done on Grid Infrastructure Homes in Oracle 11gR2. Doing it on standard RDBMS will result in errors on startup.

kevinlin.ora

How to Check Whether Oracle Binary/Instance is RAC Enabled and Relink Oracle Binary in RAC [ID 284785.1]
          修改时间 01-JUL-2011     类型 HOWTO     状态 PUBLISHED          

In this Document
  Goal
  Solution
     To check whether Oracle binary is RAC enabled:
     To check whether a running instance is a RAC instance:
     Steps to relink oracle binary with RAC option:

Applies to:
Oracle Server - Enterprise Edition - Version: 9.2.0.1 to 11.2.0.2 - Release: 9.2 to 11.2
Information in this document applies to any platform.
Goal
This note is to check whether Real Application Clusters(RAC) option is linked into Oracle binary and whether a database instance is a RAC instance, and how to relink Oracle binary in RAC environment.
Solution
To check whether Oracle binary is RAC enabled:
As ORACLE_HOME owner:

On Linux/UNIX except AIX:

  ar -t $ORACLE_HOME/rdbms/lib/libknlopt.a|grep kcsm.o

On AIX:

  ar -X32_64 -t $ORACLE_HOME/rdbms/lib/libknlopt.a|grep kcsm.o


If above command does not return anything, RAC option is not linked in. A RAC enabled oracle binary should return "kcsm.o".

To check whether a running instance is a RAC instance:
Multiple options here:

1. Check sqlplus banner:

$ORACLE_HOME/bin/sqlplus / as sysdba

SQL*Plus: Release 11.1.0.6.0 - Production on Fri Dec 26 12:11:13 2008
Copyright (c) 1982, 2007, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

A RAC instance will show "Real Application Clusters" option in the banner.


2. Check whether lmon background process exists for the instance

ps -ef| grep lmon | grep <ORACLE_SID>
oracle   627     1  0   Apr 15     ?  1:02 ora_lmon_racdb1

Only RAC instance has lmon background process.


3. Check cluster_database parameter

SQL> show parameter cluster_database

Output "true" means it's RAC instance but this is not reliable as a RAC instance may have cluster_database set to false during maintenance period.


Steps to relink oracle binary with RAC option:
Execute the following on all nodes where the ORACLE_HOME exists:

1. Log in as the ORACLE_HOME owner.

2. Stop all resources (database, listener, ASM etc) that's running from the home. When stopping database, use NORMAL or IMMEDIATE option.

3. If relinking 11gR2 Grid Infrastructure home, unlock GI home as root: $GRID_HOME/crs/install/rootcrs.pl -unlock

4. Execute the following to relink:

  cd $ORACLE_HOME/rdbms/lib
  make -f ins_rdbms.mk rac_on ioracle


If interconnect is infiniband and RDS protocol is being used instead of UDP:

  cd $ORACLE_HOME/rdbms/lib
  make -f ins_rdbms.mk ipc_rds ioracle

Caution: confirm infiniband interconnect and RDS protocol before executing it

5. If relinking 11gR2 Grid Infrastructure home, lock GI home as root: $GRID_HOME/crs/install/rootcrs.pl -patch

ALLSTARS_ORACLE

ding ding ding