Oracle数据服务器遭遇楼层停电，启动后一段时间内反复宕机

ALLSTARS_ORACLE

背景：
  昨天中午，我们公司楼层由于电路原因，突然断电。
  我们开发小组的数据库开发环境（Linux）也遭遇宕机。
问题发现：
  1、同事们在开发库中经常使用PL/SQL连接数据库，从断电重启机器以后，经常登录不上环境

--------------------------------------------------------相关的alert_sid.log------------------------------------------------------------------------
Fri Jul  6 00:40:16 2012
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 11936
Fri Jul  6 00:40:51 2012
Starting ORACLE instance (normal)
Cannot determine all dependent dynamic libraries for /proc/self/exe
Unable to find dynamic library libocr10.so in search paths
RPATH = /ade/aime1_build2101/oracle/has/lib/:/ade/aime1_build2101/oracle/lib/:/ade/aime1_build2101/oracle/has/lib/:
LD_LIBRARY_PATH is not set!
The default library directories are /lib and /usr/lib
Unable to find dynamic library libocrb10.so in search paths
Unable to find dynamic library libocrutl10.so in search paths
Unable to find dynamic library libocrutl10.so in search paths
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /usr/local/oracle/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  processes                = 150
  __shared_pool_size       = 96468992
  __large_pool_size        = 4194304
  __java_pool_size         = 4194304
  __streams_pool_size      = 4194304
  sga_target               = 285212672
  control_files            = /usr/local/oracle/oradata/techdb/control01.ctl, /usr/local/oracle/oradata/techdb/control02.ctl, /usr/local/oracle/oradata/techdb/control03.ctl
  db_block_size            = 8192
  __db_cache_size          = 167772160
  compatible               = 10.2.0.1.0
  db_file_multiblock_read_count= 16
  undo_management          = AUTO
  undo_tablespace          = tbs_undo
  _kgl_large_heap_warning_threshold= 52428800
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  dispatchers              = (PROTOCOL=TCP) (SERVICE=techdbXDB)
  job_queue_processes      = 10
  background_dump_dest     = /usr/local/oracle/admin/techdb/bdump
  user_dump_dest           = /usr/local/oracle/admin/techdb/udump
  core_dump_dest           = /usr/local/oracle/admin/techdb/cdump
  audit_file_dest          = /usr/local/oracle/admin/techdb/adump
  db_name                  = techdb
  open_cursors             = 300
  pga_aggregate_target     = 94371840
PMON started with pid=2, OS id=12303
PSP0 started with pid=3, OS id=12305
MMAN started with pid=4, OS id=12307
DBW0 started with pid=5, OS id=12309
LGWR started with pid=6, OS id=12311
CKPT started with pid=7, OS id=12313
SMON started with pid=8, OS id=12315
RECO started with pid=9, OS id=12317
CJQ0 started with pid=10, OS id=12319
MMON started with pid=11, OS id=12321
Fri Jul  6 00:40:52 2012
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=12323
Fri Jul  6 00:40:52 2012
starting up 1 shared server(s) ...
Fri Jul  6 00:40:52 2012
ALTER DATABASE   MOUNT
Fri Jul  6 00:40:56 2012
Setting recovery target incarnation to 2
Fri Jul  6 00:40:56 2012
Successful mount of redo thread 1, with mount id 3665566100
Fri Jul  6 00:40:56 2012
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Fri Jul  6 00:40:56 2012
ALTER DATABASE OPEN
Fri Jul  6 00:40:56 2012
Beginning crash recovery of 1 threads
parallel recovery started with 2 processes
Fri Jul  6 00:40:56 2012
Started redo scan
Fri Jul  6 00:40:56 2012
Completed redo scan
751 redo blocks read, 132 data blocks need recovery
Fri Jul  6 00:40:56 2012
Started redo application at
Thread 1: logseq 209579, block 583
Fri Jul  6 00:40:56 2012
Recovery of Online Redo Log: Thread 1 Group 1 Seq 209579 Reading mem 0
  Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo01.log
Fri Jul  6 00:40:56 2012
Completed redo application
Fri Jul  6 00:40:57 2012
Completed crash recovery at
Thread 1: logseq 209579, block 1334, scn 7788642518
132 data blocks read, 132 data blocks written, 751 redo blocks read
Fri Jul  6 00:40:57 2012
Thread 1 advanced to log sequence 209580
Thread 1 opened at log sequence 209580
  Current log# 2 seq# 209580 mem# 0: /usr/local/oracle/oradata/techdb/redo02.log
Successful open of redo thread 1
Fri Jul  6 00:40:57 2012
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jul  6 00:40:57 2012
SMON: enabling cache recovery
Fri Jul  6 00:40:57 2012
Successfully onlined Undo Tablespace 8.
Fri Jul  6 00:40:57 2012
SMON: enabling tx recovery
Fri Jul  6 00:40:57 2012
Database Characterset is ZHS16GBK
Fri Jul  6 00:40:58 2012
SMON: Restarting fast_start parallel rollback
Fri Jul  6 00:40:58 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_p000_12331.trc:
ORA-00600: internal error code, arguments: [4198], [9], [], [], [], [], [], []
Fri Jul  6 00:40:58 2012
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=20, OS id=12339
Fri Jul  6 00:40:58 2012
Doing block recovery for file 2 block 41
Block recovery from logseq 209580, block 3 to scn 7788642578
Fri Jul  6 00:40:58 2012
Recovery of Online Redo Log: Thread 1 Group 2 Seq 209580 Reading mem 0
  Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo02.log
Block recovery completed at rba 209580.3.16, scn 1.3493675283
Fri Jul  6 00:40:58 2012
SMON: ignoring slave err,downgrading to serial rollback
Fri Jul  6 00:40:58 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_12315.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
ORACLE Instance techdb (pid = 8) - Error 600 encountered while recovering transaction (3, 17).
Fri Jul  6 00:40:59 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_12315.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
Fri Jul  6 00:40:59 2012
Completed: ALTER DATABASE OPEN
Fri Jul  6 00:47:09 2012
SMON: Restarting fast_start parallel rollback
Fri Jul  6 00:47:09 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_p000_12499.trc:
ORA-00600: internal error code, arguments: [4198], [9], [], [], [], [], [], []
Fri Jul  6 00:47:09 2012
Doing block recovery for file 2 block 41
Block recovery from logseq 209580, block 311 to scn 7788642871
Fri Jul  6 00:47:09 2012
Recovery of Online Redo Log: Thread 1 Group 2 Seq 209580 Reading mem 0
  Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo02.log
Block recovery stopped at EOT rba 209580.314.16
Block recovery completed at rba 209580.314.16, scn 1.3493675574
Fri Jul  6 00:47:09 2012
SMON: ignoring slave err,downgrading to serial rollback
Fri Jul  6 00:47:09 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_12315.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
ORACLE Instance techdb (pid = 8) - Error 600 encountered while recovering transaction (3, 17).
Fri Jul  6 00:47:10 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_12315.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
Fri Jul  6 00:52:38 2012
SMON: Restarting fast_start parallel rollback

ALLSTARS_ORACLE

ri Jul  6 00:52:38 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_p000_12705.trc:
ORA-00600: internal error code, arguments: [4198], [9], [], [], [], [], [], []
Fri Jul  6 00:52:39 2012
Doing block recovery for file 2 block 41
Block recovery from logseq 209580, block 762 to scn 7788643210
Fri Jul  6 00:52:39 2012
Recovery of Online Redo Log: Thread 1 Group 2 Seq 209580 Reading mem 0
  Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo02.log
Block recovery stopped at EOT rba 209580.763.16
Block recovery completed at rba 209580.763.16, scn 1.3493675913
Fri Jul  6 00:52:39 2012
SMON: ignoring slave err,downgrading to serial rollback
Fri Jul  6 00:52:41 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_12315.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
ORACLE Instance techdb (pid = 8) - Error 600 encountered while recovering transaction (3, 17).
Fri Jul  6 00:52:41 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_12315.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
Fri Jul  6 00:57:42 2012
SMON: Restarting fast_start parallel rollback
Fri Jul  6 00:57:42 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_p000_12764.trc:
ORA-00600: internal error code, arguments: [4198], [9], [], [], [], [], [], []
Fri Jul  6 00:57:43 2012
Doing block recovery for file 2 block 41
Block recovery from logseq 209580, block 762 to scn 7788643215
Fri Jul  6 00:57:43 2012
Recovery of Online Redo Log: Thread 1 Group 2 Seq 209580 Reading mem 0
  Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo02.log
Block recovery completed at rba 209580.765.16, scn 1.3493675920
Fri Jul  6 00:57:43 2012
SMON: ignoring slave err,downgrading to serial rollback
Fri Jul  6 00:57:43 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_12315.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
ORACLE Instance techdb (pid = 8) - Error 600 encountered while recovering transaction (3, 17).
Fri Jul  6 00:57:44 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_12315.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
Fri Jul  6 01:02:45 2012
SMON: Restarting fast_start parallel rollback
Fri Jul  6 01:02:45 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_p000_12827.trc:
ORA-00600: internal error code, arguments: [4198], [9], [], [], [], [], [], []
Fri Jul  6 01:02:45 2012
Doing block recovery for file 2 block 41
Block recovery from logseq 209580, block 4135 to scn 7788644402
Fri Jul  6 01:02:45 2012
Recovery of Online Redo Log: Thread 1 Group 2 Seq 209580 Reading mem 0
  Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo02.log
Block recovery stopped at EOT rba 209580.4136.16
Block recovery completed at rba 209580.4136.16, scn 1.3493677105
Fri Jul  6 01:02:45 2012
SMON: ignoring slave err,downgrading to serial rollback
Fri Jul  6 01:02:45 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_12315.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
Fri Jul  6 01:02:46 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_pmon_12303.trc:
ORA-00474: SMON process terminated with error
Fri Jul  6 01:02:46 2012
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 12303