数据库无缘无故的down掉

ALLSTARS_ORACLE

数据库昨天晚上无缘无故的down掉，不知道什么原因，求解!!!

Checkpoint not complete
  Current log# 2 seq# 1679 mem# 0: /u01/oradata/empi/redo02.log
Thread 1 advanced to log sequence 1680 (LGWR switch)
  Current log# 1 seq# 1680 mem# 0: /u01/oradata/empi/redo01.log
Thu Nov 22 22:28:29 2012
Archived Log entry 1789 added for thread 1 sequence 1679 ID 0x409b3deb dest 1:
Thu Nov 22 22:29:07 2012
KCF: read, write or open error, block=0x7054c online=1
        file=5 '/u01/oradata/empi/data_empi.dbf'
        error=27072 txt: 'Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 460092
Additional information: -1'
Errors in file /u01/diag/rdbms/empi/empi/trace/empi_dbw0_2444.trc:
Errors in file /u01/diag/rdbms/empi/empi/trace/empi_dbw0_2444.trc:
ORA-63999: data file suffered media failure
ORA-01114: IO error writing block to file 5 (block # 460108)
ORA-01110: data file 5: '/u01/oradata/empi/data_empi.dbf'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 460092
Additional information: -1
DBW0 (ospid: 2444): terminating the instance due to error 63999
Thu Nov 22 22:29:08 2012
System state dump requested by (instance=1, osid=2444 (DBW0)), summary=[abnormal instance termination].
System State dumped to trace file /u01/diag/rdbms/empi/empi/trace/empi_diag_2436.trc
Dumping diagnostic data in directory=[cdmp_20121122222908], requested by (instance=1, osid=2444 (DBW0)), summary=[abnormal instance termination].
Instance terminated by DBW0, pid = 2444
Thu Nov 22 23:04:03 2012
Starting ORACLE instance (normal)


磁盘没有坏。可以写的!!!



BUG: soft lockup - CPU#0 stuck for 2843s! [oracle:8475]
Modules linked in: autofs4 sunrpc vsock(U) ipv6 dm_mirror dm_region_hash dm_log uinput ppdev parport_pc parport vmware_balloon sg vmci(U) i2c_piix4 i2c_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi ata_generic pata_acpi ata_piix dm_mod [last unloaded: speedstep_lib]
CPU 0:
Modules linked in: autofs4 sunrpc vsock(U) ipv6 dm_mirror dm_region_hash dm_log uinput ppdev parport_pc parport vmware_balloon sg vmci(U) i2c_piix4 i2c_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi ata_generic pata_acpi ata_piix dm_mod [last unloaded: speedstep_lib]
Pid: 8475, comm: oracle Tainted: G        W  ----------------  2.6.32-71.el6.x86_64 #1 VMware Virtual Platform
RIP: 0010:[<ffffffff8111f446>]  [<ffffffff8111f446>] clear_page_dirty_for_io+0x6/0x100
RSP: 0018:ffff88020df11c00  EFLAGS: 00000246
RAX: 004000000000083d RBX: ffff88020df11c08 RCX: 000000000000000c
RDX: ffffea0012c95128 RSI: 00000000000e2d36 RDI: ffffea0012c95128
RBP: ffffffff81013c8e R08: 00000000000e2d36 R09: 0000000000000000
R10: ffffea0012c95210 R11: 0000000000000003 R12: 0000000000000000
R13: 000000000df11ce0 R14: ffff88020df11c68 R15: 00000000000e2d3c
FS:  00007fdc38652700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000002cc6ba8 CR3: 000000056c6c6000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa00a4acf>] ? write_cache_pages_da+0x18f/0x430 [ext4]
[<ffffffffa00a5012>] ? ext4_da_writepages+0x2a2/0x600 [ext4]
[<ffffffff8110c177>] ? unlock_page+0x27/0x30
[<ffffffff81135837>] ? handle_pte_fault+0xf7/0xad0
[<ffffffff81120951>] ? do_writepages+0x21/0x40
[<ffffffff8110c73b>] ? __filemap_fdatawrite_range+0x5b/0x60
[<ffffffff8110c79a>] ? filemap_write_and_wait_range+0x5a/0x90
[<ffffffff8119aa9e>] ? vfs_fsync_range+0x7e/0xe0
[<ffffffff8119ab6d>] ? vfs_fsync+0x1d/0x20
[<ffffffff8119abae>] ? do_fsync+0x3e/0x60
[<ffffffff8119ac00>] ? sys_fsync+0x10/0x20
[<ffffffff81013172>] ? system_call_fastpath+0x16/0x1b
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 6c 4f 00 00 08 00
end_request: I/O error, dev sdb, sector 128543823
Buffer I/O error on device dm-0, logical block 26469506
lost page write due to I/O error on dm-0
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 64 4f 00 04 00 00
end_request: I/O error, dev sdb, sector 128541775
Buffer I/O error on device dm-0, logical block 26469250
lost page write due to I/O error on dm-0
Buffer I/O error on device dm-0, logical block 26469251
lost page write due to I/O error on dm-0
Buffer I/O error on device dm-0, logical block 26469252
lost page write due to I/O error on dm-0
Buffer I/O error on device dm-0, logical block 26469253
lost page write due to I/O error on dm-0
Buffer I/O error on device dm-0, logical block 26469254
lost page write due to I/O error on dm-0
Buffer I/O error on device dm-0, logical block 26469255
lost page write due to I/O error on dm-0
Buffer I/O error on device dm-0, logical block 26469256
lost page write due to I/O error on dm-0
Buffer I/O error on device dm-0, logical block 26469257
lost page write due to I/O error on dm-0
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 60 47 00 00 08 00
end_request: I/O error, dev sdb, sector 128540743
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 74 57 00 04 00 00
end_request: I/O error, dev sdb, sector 128545879
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 78 5f 00 04 00 00
end_request: I/O error, dev sdb, sector 128546911
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 84 5f 00 00 08 00
end_request: I/O error, dev sdb, sector 128549983
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 5c 47 00 04 00 00
end_request: I/O error, dev sdb, sector 128539719
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 86 2f 00 02 10 00
end_request: I/O error, dev sdb, sector 128550447
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 78 57 00 00 08 00
end_request: I/O error, dev sdb, sector 128546903
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 7c 5f 00 04 00 00
end_request: I/O error, dev sdb, sector 128547935
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 6c 57 00 04 00 00
end_request: I/O error, dev sdb, sector 128543831
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 60 4f 00 04 00 00
end_request: I/O error, dev sdb, sector 128540751
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 68 4f 00 04 00 00
end_request: I/O error, dev sdb, sector 128542799
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 84 67 00 01 c8 00
end_request: I/O error, dev sdb, sector 128549991
sd 2:0:1:0: timing out command, waited 1080s
sd 2:0:1:0: [sdb] Unhandled error code
sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 a9 70 57 00 04 00 00
end_request: I/O error, dev sdb, sector 128544855
JBD2: Detected IO errors while flushing file data on dm-0-8