[Exadata常规管理] ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O

2^#

发表于 2013-3-13 15:02:45

[root@aaadb03 trace]# cat /etc/sysctl.conf | grep fs.aio
fs.aio-max-nr = 1048576

是否这个参数有点下，ORACLE推荐设置fs.aio-max-nr= 3145728

3^#

发表于 2013-3-13 15:53:01

就日志看 12月3日也出过类似的问题，特征之一也是并行子进程遇到ORA-27090：

Mon Dec 03 09:57:58 2012
Setting recovery pair for thread 1: nab 19850 seq 50549
Thread 1 advanced to log sequence 50550 (thread recovery)
Redo thread 1 internally disabled at seq 50550 (SMON)
Errors in file /u01/app/oracle/diag/rdbms/aaadb/aaadb3/trace/aaadb3_p005_28947.trc:
ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 3
Additional information: 128
Additional information: 145315736
Errors in file /u01/app/oracle/diag/rdbms/aaadb/aaadb3/trace/aaadb3_p006_28949.trc:
ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 3
Additional information: 128
Additional information: 145315736
Errors in file /u01/app/oracle/diag/rdbms/aaadb/aaadb3/trace/aaadb3_p000_28926.trc:
ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 3
Additional information: 128
Additional information: 145315736
Errors in file /u01/app/oracle/diag/rdbms/aaadb/aaadb3/trace/aaadb3_p008_28953.trc:
ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O
Linux-x86_64 Error: 11: Resource temporarily unavailable

4^#

发表于 2013-3-13 15:54:18

有几点值得明确的：

1。并行进程在跑什么SQL
2. 这个并行SQL是否普遍引起问题

该aaadb3_p011_15078.trc 和ASH、AWR 都是诊断的线索

5^#

发表于 2013-3-13 16:01:47

[root@aaadb03 trace]# cat /etc/sysctl.conf | grep fs.aio
fs.aio-max-nr = 1048576

1048576 确实低了

more info :

dr: 14606456 11.2.0.2 RDBMS 11.2.0.2 VOS PRODID-5 PORTID-226 ORA-27090
Abstract: EXADATA: AIO-MAX-NR RECOMMENDATION FOR X2-8

*** 09/10/12 12:30 pm ***

PROBLEM:
--------
Customer gets the following error . The same error is reported from all
instances including ASM.

ORA-27090: Unable to reserve kernel resources for asynchronous disk I/O
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 3
Additional information: 128

DIAGNOSTIC ANALYSIS:
--------------------
It seems  from following output that aio resources are exhausted

[root@xdb2db01 fs]# cat /proc/sys/fs/aio-nr
3145726
[root@xdb2db01 fs]# cat /proc/sys/fs/aio-max-nr
3145728
AIO-MAX-NR determines the maximum number of allowable concurrent requests.

Question here  is can we increase the value of  aio-max-nr  to  large value
, probably double the size of current setting without worrying about any
side effects? (System is X2-8)

Exadata V2 Starter Kit (Doc ID 1244344.1) says to set this to 4 million.
However  even this is not specific for X2-8.  Can we set this  parameter to a
value like  6 million or even 8 million on x2-8? If customer has such high
concurrent IO. Or should this need to be kept under certain limits.

WORKAROUND:
-----------
Increase AIO-MAX-NR
or
disk_asynch_io=false

6^#

发表于 2013-3-13 17:33:09

State Type Rebal  Sector  Block    AU  Total_MB Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N       512 4096  4194304  33996800 775016       4803404       -2014194             0          N  DATA/
MOUNTED  NORMAL  N       512 4096  4194304  10937408  10651284          994309       4828487             0          N  RECO/
MOUNTED  NORMAL  N       512 4096  4194304 2087680 2084312          189789       947261             0          N  SYSTEMDG/

usable_file_MB 都出现负值了，和这有关系吗？这个负值不太理解什么含义呀？我配置的时NORMAL的冗余模式，还需要担心单个磁盘的损坏吗？

7^#

发表于 2013-3-13 17:45:48

AWR报告(xxxxxxxxxxxxxxxxxxxxx)

awr.tar.gz

3.45 MB, 下载次数: 592

8^#

发表于 2013-3-13 17:54:58

这是是日志（xxxxxxxxxxxxxxxx）

aaadb3_p011_15078.rar

33.24 KB, 下载次数: 764

9^#

发表于 2013-3-13 18:01:14

11:00~14:00 三个小时的ASH报告

ash20130313.rar

6.73 KB, 下载次数: 731

10^#

发表于 2013-3-13 23:47:58

查了一下 11.2.3.1 默认fs.aio-max-nr = 3145728

[root@dm01db01 ~]# imageinfo

Kernel version: 2.6.18-274.18.1.0.1.el5 #1 SMP Thu Feb 9 19:07:16 EST 2012 x86_64
Image version: 11.2.3.1.1.120607
Image activated: 2012-09-07 03:12:44 -0400
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys1

[root@dm01db01 ~]# cat /etc/sysctl.conf|grep aio
# 8311668 file-max and aio-max-nr also from oracle-validated-1.0.0-24.el5.x86_64.rpm
fs.aio-max-nr = 3145728