RMAN备份报错ORA-19501（无备份）解决方法

ALLSTARS_ORACLE

环境：软件  oracle 10.1.0.4.2         系统  linux
最近刚接手的一个库，没有备份，用rman备份时候报ORA-19501错误
后来查着说是有坏块，用dbv检查没有坏块，用rman检查有坏块，搞不懂
怎么回事，有查过有说是假坏块的，但是情况又不完全相同，跪求大神们出手
相救！！！！！如果能解决，还会追加悬赏。（求解决方法，最好能把原理或者
原因也说下）


以下是一些内容


RMAN> run {
2> backup database format '/infra/flash_recovery_area/ORCL/backupset/%U.dbf';
3> }


Starting backup at 01-AUG-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=296 devtype=DISK
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00046 name=/infra/oradata/ORCL/datafile/o1_mf_esbigtbl_1q6k0sp9_.dbf
input datafile fno=00002 name=/infra/oradata/ORCL/datafile/o1_mf_undotbs1_1q6jqcko_.dbf
input datafile fno=00063 name=/infra/oradata/ORCL/datafile/o1_mf_content__1q6k05ym_.dbf
input datafile fno=00038 name=/infra/oradata/ORCL/datafile/o1_mf_rtc_tran_1q6jrmdp_.dbf
input datafile fno=00044 name=/infra/oradata/ORCL/datafile/o1_mf_rtc_im_d_1q6jmwh1_.dbf
input datafile fno=00009 name=/infra/oradata/ORCL/datafile/o1_mf_ias_meta_1q6jmw5f_.dbf
input datafile fno=00034 name=/infra/oradata/ORCL/datafile/o1_mf_rtc_docu_1q6jxrqh_.dbf
input datafile fno=00062 name=/infra/oradata/ORCL/datafile/o1_mf_content__1q6k1spm_.dbf
input datafile fno=00064 name=/infra/oradata/ORCL/datafile/o1_mf_content__1q6jr6nb_.dbf
input datafile fno=00007 name=/infra/oradata/ORCL/datafile/o1_mf_portal_i_1q6jyy6o_.dbf
input datafile fno=00019 name=/infra/oradata/ORCL/datafile/o1_mf_orabpel_1q6jvror_.dbf
input datafile fno=00048 name=/infra/oradata/ORCL/datafile/o1_mf_esfreqtb_1q6k1glg_.dbf
input datafile fno=00015 name=/infra/oradata/ORCL/datafile/o1_mf_b2b_rt_1q6jwkjp_.dbf
input datafile fno=00016 name=/infra/oradata/ORCL/datafile/o1_mf_b2b_dt_1q6jwc6d_.dbf
input datafile fno=00042 name=/infra/oradata/ORCL/datafile/o1_mf_rtc_repo_1q6jxkbj_.dbf
input datafile fno=00023 name=/infra/oradata/ORCL/datafile/o1_mf_olts_att_1q6k1dmp_.dbf
input datafile fno=00022 name=/infra/oradata/ORCL/datafile/o1_mf_uddisys__1q6jy8l9_.dbf
input datafile fno=00014 name=/infra/oradata/ORCL/datafile/o1_mf_wcrsys_t_1q6jz4bp_.dbf
input datafile fno=00006 name=/infra/oradata/ORCL/datafile/o1_mf_portal_d_1q6jzf3o_.dbf
input datafile fno=00018 name=/infra/oradata/ORCL/datafile/o1_mf_b2b_lob_1q6jz845_.dbf
input datafile fno=00047 name=/infra/oradata/ORCL/datafile/o1_mf_esfreqid_1q6k1q3m_.dbf
input datafile fno=00053 name=/infra/oradata/ORCL/datafile/o1_mf_esperftb_1q6k0j48_.dbf
input datafile fno=00057 name=/infra/oradata/ORCL/datafile/o1_mf_cwsys_ma_1q6k09o7_.dbf
input datafile fno=00066 name=/infra/oradata/ORCL/datafile/o1_mf_ic_metri_1q6k0v7c_.dbf
input datafile fno=00033 name=/infra/oradata/ORCL/datafile/o1_mf_rtc_arch_1q6k11tf_.dbf
input datafile fno=00027 name=/infra/oradata/ORCL/datafile/o1_mf_olts_svr_1q6k14m8_.dbf
input datafile fno=00011 name=/infra/oradata/ORCL/datafile/o1_mf_disco_pt_1q6k1r44_.dbf
input datafile fno=00051 name=/infra/oradata/ORCL/datafile/o1_mf_esnews_1q6k1nrp_.dbf
input datafile fno=00029 name=/infra/oradata/ORCL/datafile/o1_mf_rtc_look_1q6k1vd4_.dbf
input datafile fno=00041 name=/infra/oradata/ORCL/datafile/o1_mf_rtc_tran_1q6k1xrm_.dbf
input datafile fno=00069 name=/infra/oradata/ORCL/datafile/PROJECT_STAT.dbf
channel ORA_DISK_1: starting piece 1 at 01-AUG-13
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 08/01/2013 15:48:46
ORA-19501: read error on file "/infra/oradata/ORCL/datafile/o1_mf_content__1q6k05ym_.dbf", blockno 15233 (blocksize=8192)
ORA-27072: File I/O error
Linux Error: 2: No such file or directory
Additional information: 15232



经检查，/infra/oradata/ORCL/datafile/o1_mf_content__1q6k05ym_.dbf物理上存在


[oracle@infra bin]$ oerr ora 1950
01950, 00000, "no privileges on tablespace '%s'"
// *Cause:  User does not have privileges to allocate an extent in the
//          specified tablespace.
// *Action: Grant the user the appropriate system privileges or grant the user
//          space resource on the tablespace.
[oracle@infra bin]$ oerr ora 27072
27072, 00000, "File I/O error"
// *Cause:  read/write/readv/writev system call returned error, additional
//          information indicates starting block number of I/O
// *Action: check errno


用dbv检查没有坏块


[oracle@infra bin]$ dbv file=/infra/oradata/ORCL/datafile/o1_mf_content__1q6k05ym_.dbf  blocksize=8192

DBVERIFY: Release 10.1.0.4.2 - Production on Thu Aug 1 16:26:07 2013

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

DBVERIFY - Verification starting : FILE = /infra/oradata/ORCL/datafile/o1_mf_content__1q6k05ym_.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 15258
Total Pages Processed (Data) : 13427
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 20
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 1707
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 104
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Highest block SCN            : 1978186924 (0.1978186924)



用如下方法备份检查，还是报相同错误，说明确实存在坏块


RMAN> run {
2> backup validate datafile 63 format '/infra/flash_recovery_area/ORCL/backupset/%U.dbf';
3> }


Starting backup at 01-AUG-13
using target database controlfile instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=367 devtype=DISK
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00063 name=/infra/oradata/ORCL/datafile/o1_mf_content__1q6k05ym_.dbf
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 08/01/2013 16:32:35
ORA-19501: read error on file "/infra/oradata/ORCL/datafile/o1_mf_content__1q6k05ym_.dbf", blockno 15233 (blocksize=8192)
ORA-27072: File I/O error
Additional information: 15232


RMAN> backup check logical validate datafile 63;
Starting backup at 01-AUG-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00063 name=/infra/oradata/ORCL/datafile/o1_mf_content__1q6k05ym_.dbf
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 08/01/2013 17:42:10
ORA-19501: read error on file "/infra/oradata/ORCL/datafile/o1_mf_content__1q6k05ym_.dbf", blockno 15233 (blocksize=8192)
ORA-27072: File I/O error
Additional information: 15232





SQL> SELECT OWNER, SEGMENT_NAME, SEGMENT_TYPE, TABLESPACE_NAME, A.PARTITION_NAME
  2    FROM DBA_EXTENTS A
  3   WHERE FILE_ID = &FILE_ID
  4     AND &BLOCK_ID BETWEEN BLOCK_ID AND BLOCK_ID + BLOCKS - 1;
Enter value for file_id: 63
old   3:  WHERE FILE_ID = &FILE_ID
new   3:  WHERE FILE_ID = 63
Enter value for block_id: 15233
old   4:    AND &BLOCK_ID BETWEEN BLOCK_ID AND BLOCK_ID + BLOCKS - 1
new   4:    AND 15233 BETWEEN BLOCK_ID AND BLOCK_ID + BLOCKS - 1


OWNER                          SEGMENT_NAME                                       SEGMENT_TYPE       TABLESPACE_NAME
------------------------------ -------------------------------------------------- ------------------ ------------------------------
PARTITION_NAME
------------------------------
CONTENT                        DR$IFS_TEXT$I                                      TABLE              CONTENT_IFS_CTX_K

ALLSTARS_ORACLE

小弟菜鸟一枚，对linux不是特别熟悉，不知道下面那个是系统日志

[oracle@infra log]$ pwd
/var/log
[oracle@infra log]$ ls
boot.log    cron.1  gdm      ksyms.4  maillog.1   messages.2               rpmpkgs.1  scrollkeeper.log  spooler    vbox               XFree86.1.log
boot.log.1  cron.2  httpd    ksyms.5  maillog.2   messages.3               rpmpkgs.2  secure            spooler.1  wtmp
boot.log.2  cron.3  ksyms.0  ksyms.6  maillog.3   messages.4               rpmpkgs.3  secure.1          spooler.2  wtmp.1
boot.log.3  cron.4  ksyms.1  lastlog  maillog.4   redhat-config-network    rpmpkgs.4  secure.2          spooler.3  xferlog
boot.log.4  cups    ksyms.2  mail     messages    redhat-config-network.1  sa         secure.3          spooler.4  XFree86.0.log
cron        dmesg   ksyms.3  maillog  messages.1  rpmpkgs                  samba      secure.4          squid      XFree86.0.log.old

又重新执行了下备份，alert太大，将alert的最后200行贴出来

[oracle@infra bdump]$ tail -200 alert_orcl.log
ARCH: Connecting to console port...
Fri Aug  2 09:31:06 2013
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383090_%u_.arc' (thread 1 sequence 383090)
(orcl)
ARCH: Connecting to console port...
Fri Aug  2 09:31:06 2013
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383090_8zp2ytc2_.arc'
(orcl)
ARCH: Connecting to console port...
Committing creation of archivelog '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383090_8zp2ytc2_.arc'
Created Oracle managed file /infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383090_8zp2ytc2_.arc
Fri Aug  2 09:31:07 2013
ARC0: Completed archiving thread 1 sequence 383090 (2489055534-2489063904) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 09:51:10 2013
Private_strands 7 at log switch
Thread 1 advanced to log sequence 383092
  Current log# 3 seq# 383092 mem# 0: /infra/oradata/ORCL/onlinelog/o1_mf_3_1q6k3bk3_.log
  Current log# 3 seq# 383092 mem# 1: /infra/flash_recovery_area/ORCL/onlinelog/o1_mf_3_1q6k3c8w_.log
Fri Aug  2 09:51:10 2013
ARC0: Evaluating archive thread 1 sequence 383091
Fri Aug  2 09:51:10 2013
ARC0: Beginning to archive thread 1 sequence 383091 (2489063904-2489073051) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 09:51:10 2013
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383091_%u_.arc' (thread 1 sequence 383091)
(orcl)
ARCH: Connecting to console port...
Fri Aug  2 09:51:10 2013
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383091_8zp44gvj_.arc'
(orcl)
ARCH: Connecting to console port...
Committing creation of archivelog '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383091_8zp44gvj_.arc'
Created Oracle managed file /infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383091_8zp44gvj_.arc
Fri Aug  2 09:51:11 2013
ARC0: Completed archiving thread 1 sequence 383091 (2489063904-2489073051) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 09:57:40 2013
Private_strands 7 at log switch
Thread 1 advanced to log sequence 383093
  Current log# 2 seq# 383093 mem# 0: /infra/oradata/ORCL/onlinelog/o1_mf_2_1q6k38t2_.log
  Current log# 2 seq# 383093 mem# 1: /infra/flash_recovery_area/ORCL/onlinelog/o1_mf_2_1q6k39n5_.log
Fri Aug  2 09:57:40 2013
ARC0: Evaluating archive thread 1 sequence 383092
Fri Aug  2 09:57:40 2013
ARC0: Beginning to archive thread 1 sequence 383092 (2489073051-2489079570) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 09:57:40 2013
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383092_%u_.arc' (thread 1 sequence 383092)
(orcl)
ARCH: Connecting to console port...
Fri Aug  2 09:57:40 2013
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383092_8zp4jn29_.arc'
(orcl)
ARCH: Connecting to console port...
Committing creation of archivelog '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383092_8zp4jn29_.arc'
Created Oracle managed file /infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383092_8zp4jn29_.arc
Fri Aug  2 09:57:41 2013
ARC0: Completed archiving thread 1 sequence 383092 (2489073051-2489079570) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 09:58:31 2013
Thread 1 cannot allocate new log, sequence 383094
Checkpoint not complete
  Current log# 2 seq# 383093 mem# 0: /infra/oradata/ORCL/onlinelog/o1_mf_2_1q6k38t2_.log
  Current log# 2 seq# 383093 mem# 1: /infra/flash_recovery_area/ORCL/onlinelog/o1_mf_2_1q6k39n5_.log
Private_strands 7 at log switch
Thread 1 advanced to log sequence 383094
  Current log# 1 seq# 383094 mem# 0: /infra/oradata/ORCL/onlinelog/o1_mf_1_1q6k375k_.log
  Current log# 1 seq# 383094 mem# 1: /infra/flash_recovery_area/ORCL/onlinelog/o1_mf_1_1q6k381q_.log
Fri Aug  2 09:58:34 2013
ARC0: Evaluating archive thread 1 sequence 383093
Fri Aug  2 09:58:34 2013
ARC0: Beginning to archive thread 1 sequence 383093 (2489079570-2489082293) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 09:58:34 2013
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383093_%u_.arc' (thread 1 sequence 383093)
(orcl)
ARCH: Connecting to console port...
Fri Aug  2 09:58:35 2013
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383093_8zp4lbs3_.arc'
(orcl)
ARCH: Connecting to console port...
Committing creation of archivelog '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383093_8zp4lbs3_.arc'
Created Oracle managed file /infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383093_8zp4lbs3_.arc
Fri Aug  2 09:58:36 2013
ARC0: Completed archiving thread 1 sequence 383093 (2489079570-2489082293) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 10:05:14 2013
Private_strands 7 at log switch
Thread 1 advanced to log sequence 383095
  Current log# 4 seq# 383095 mem# 0: /infra/oradata/<Default>
Fri Aug  2 10:05:14 2013
ARC0: Evaluating archive thread 1 sequence 383094
Fri Aug  2 10:05:14 2013
ARC0: Beginning to archive thread 1 sequence 383094 (2489082293-2489086683) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 10:05:14 2013
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383094_%u_.arc' (thread 1 sequence 383094)
(orcl)
ARCH: Connecting to console port...
Fri Aug  2 10:05:15 2013
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383094_8zp4ytsg_.arc'
(orcl)
ARCH: Connecting to console port...
Committing creation of archivelog '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383094_8zp4ytsg_.arc'
Created Oracle managed file /infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383094_8zp4ytsg_.arc
Fri Aug  2 10:05:16 2013
ARC0: Completed archiving thread 1 sequence 383094 (2489082293-2489086683) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 10:21:40 2013
Private_strands 7 at log switch
Thread 1 advanced to log sequence 383096
  Current log# 3 seq# 383096 mem# 0: /infra/oradata/ORCL/onlinelog/o1_mf_3_1q6k3bk3_.log
  Current log# 3 seq# 383096 mem# 1: /infra/flash_recovery_area/ORCL/onlinelog/o1_mf_3_1q6k3c8w_.log
Fri Aug  2 10:21:40 2013
ARC0: Evaluating archive thread 1 sequence 383095
Fri Aug  2 10:21:40 2013
ARC0: Beginning to archive thread 1 sequence 383095 (2489086683-2489094961) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 10:21:40 2013
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383095_%u_.arc' (thread 1 sequence 383095)
(orcl)
ARCH: Connecting to console port...
Fri Aug  2 10:21:41 2013
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383095_8zp5xnw0_.arc'
(orcl)
ARCH: Connecting to console port...
Committing creation of archivelog '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383095_8zp5xnw0_.arc'
Created Oracle managed file /infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383095_8zp5xnw0_.arc
Fri Aug  2 10:21:42 2013
ARC0: Completed archiving thread 1 sequence 383095 (2489086683-2489094961) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 10:42:09 2013
Private_strands 7 at log switch
Thread 1 advanced to log sequence 383097
  Current log# 2 seq# 383097 mem# 0: /infra/oradata/ORCL/onlinelog/o1_mf_2_1q6k38t2_.log
  Current log# 2 seq# 383097 mem# 1: /infra/flash_recovery_area/ORCL/onlinelog/o1_mf_2_1q6k39n5_.log
Fri Aug  2 10:42:09 2013
ARC0: Evaluating archive thread 1 sequence 383096
Fri Aug  2 10:42:09 2013
ARC0: Beginning to archive thread 1 sequence 383096 (2489094961-2489104368) (orcl)
ARCH: Connecting to console port...
Fri Aug  2 10:42:09 2013
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_10: '/infra/flash_recovery_area/ORCL/archivelog/2013_08_02/o1_mf_1_383096_%u_.arc' (thread 1 sequence 383096)
(orcl)
ARCH: Connecting to console port...
Fri Aug  2 10:42:09 2013
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_10: