数据库非正常关机后起不来了。。

ALLSTARS_ORACLE

数据库服务器非正常关机后起不来了。很奇怪，sid 名字应该是rdmetdev, 锁应该是lkrdmetdev.不知道怎么变成红色的那个了。。没有做过任何改动。。也不存在那个文件


请看日志
Thu Feb 14 09:25:20 EST 2013
sculkget: failed to lock /u01/app/oracle/product/10.2/dbs/lkinstrdmetdev exclusive
Thu Feb 14 09:25:20 EST 2013
sculkget: lock held by PID: 272600512
Thu Feb 14 09:25:20 EST 2013
Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown this Instance.
Thu Feb 14 09:25:20 EST 2013
Failed to acquire instance startup/shutdown serialization primitive
Thu Feb 14 09:35:33 EST 2013
sculkget: failed to lock /u01/app/oracle/product/10.2/dbs/lkinstrdmetdev exclusive
Thu Feb 14 09:35:33 EST 2013
sculkget: lock held by PID: 125960160
Thu Feb 14 09:35:33 EST 2013
Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown this Instance.
Thu Feb 14 09:35:33 EST 2013
Failed to acquire instance startup/shutdown serialization primitive


Thu Feb 14 09:34:57 EST 2013
Errors in file /u01/app/oracle/admin/rdmetdev/udump/rdmetdev_ora_6321.trc:
ORA-27050: function called with invalid FIB/IOV structure
Additional information: 2
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 37: No locks available
Additional information: 10

rdmetdev_ora_6321.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/10.2
System name:    Linux
Node name:      ecamolx1813
Release:        2.6.18-194.el5
Version:        #1 SMP Tue Mar 16 21:52:39 EDT 2010
Machine:        x86_64
Instance name: rdmetdev
Redo thread mounted by this instance: 0 <none>
Oracle process number: 0
Unix process pid: 6321, image: oracle@ecamolx1813

*** 2013-02-14 09:34:57.630
ORA-27050: function called with invalid FIB/IOV structure
Additional information: 2
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 37: No locks available
Additional information: 10
*** 2013-02-14 09:35:33.039
Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown this Instance.
*** 2013-02-14 09:35:33.051
Failed to acquire instance startup/shutdown serialization primitive




bash-3.2$ ps -ef|grep rdmetdev
oracle   19003  6738  0 11:48 pts/1    00:00:00 grep rdmetdev

bash-3.2$ lsof |grep lkinst
lsof: WARNING: can't stat() nfs file system /var/cfengine/SiteInfo
      Output information may be incomplete.

bash-3.2$ ls -ltr $ORACLE_HOME/dbs
total 56936
-rw-r----- 1 oracle dba     8385 Sep 11  1998 init.ora
-rw-r--r-- 1 oracle dba    12920 May  3  2001 initdw.ora
-rw-rw---- 1 oracle dba     1544 Apr 11  2011 hc_test.dat
-rw-r----- 1 oracle dba     1536 Apr 11  2011 orapwtest
-rw-r----- 1 oracle dba     3584 Jun 16  2011 spfiletest.ora
-rw-r--r-- 1 oracle dba     1344 Nov 22  2011 initrdmetdev.ora_bak
-rw-r----- 1 oracle dba     1536 Feb 10  2012 orapwrdmetdev
-rw-r--r-- 1 oracle dba     1277 Oct  4 14:02 initrdmetdev.ora_bak_1004
-rw-r----- 1 oracle dba     3584 Oct 31 10:09 spfilerdmetdev.ora
-rw-r----- 1 oracle dba 58114048 Feb 12 05:32 snapcf_rdmetdev.f
-rw-r--r-- 1 oracle dba     1277 Feb 13 14:24 initrdmetdev.ora
-rw-rw---- 1 oracle dba     1544 Feb 13 15:50 hc_rdmetdev.dat
-rw-r----- 1 oracle dba     3054 Feb 14 09:35 alert_rdmetdev.log

bash-3.2$ env|grep ora
rdmetdev_PFILE=/u01/app/oracle/admin/rdmetdev/pfile
ORACLE_PATH=/u01/app/oracle/product/10.2/bin
OLDPWD=/u01/app/oracle/product/10.2
USER=oracle
LD_LIBRARY_PATH=/usr/openv/netbackup/bin:/u01/app/oracle/product/10.2/lib:/u01/app/oracle/product/10.2/dbjava/lib
LD_LIBRARY_PATH_64=/usr/openv/netbackup/bin:/u01/app/oracle/product/10.2/lib:/u01/app/oracle/product/10.2/dbjava/lib
ORACLE_BASE=/u01/app/oracle
rdmetdev_ARCH=/u01/app/oracle/admin/rdmetdev/arch
rdmetdev_ADMIN=/u01/app/oracle/admin/rdmetdev
TNS_ADMIN=/u01/app/oracle/product/10.2/network/admin
MAIL=/var/spool/mail/oracle
PATH=/usr/ccs/bin:/bin:/usr/bin:/usr/sbin:/usr/ucb:/etc:.:/pub/is/bin:/u01/app/oracle/product/10.2/bin:/pub/is/bin
PWD=/u01/app/oracle/product/10.2/dbs
HOME=/u01/app/oracle
LOGNAME=oracle
ORACLE_HOME=/u01/app/oracle/product/10.2
README=/u01/app/oracle/admin/README
bash-3.2$