richard

87 积分	0 好友	3 主题

发消息

[Exadata硬件] 关于awr中% of flow controlled messages参数的问题

1^#

发表于 2013-4-15 19:47:24 | 查看: 5039| 回复: 6

1、rhel5.5 x64 rac
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit

2、根据您的awr 教学

% of flow controlled messages  流控制最常见的原因是网络状况不佳， % of flow controlled messages应当小于1%

3、我这边有个awr：
db1

Global Cache and Enqueue Services - Messaging Statistics

Avg message sent queue time (ms):	0.1
Avg message sent queue time on ksxp (ms):	0.2
Avg message received queue time (ms):	0.0
Avg GCS message process time (ms):	0.0
Avg GES message process time (ms):	0.0
% of direct sent messages:	12.26
% of indirect sent messages:	35.56
% of flow controlled messages:	52.17

db2

Global Cache and Enqueue Services - Messaging Statistics

Avg message sent queue time (ms):	0.0
Avg message sent queue time on ksxp (ms):	0.2
Avg message received queue time (ms):	0.1
Avg GCS message process time (ms):	0.0
Avg GES message process time (ms):	0.0
% of direct sent messages:	46.03
% of indirect sent messages:	46.33
% of flow controlled messages:	7.64

显然是有问题的

4、receivetime都不高
db1
[tr][/tr]

Avg global cache cr block receive time (ms):	0.4
Avg global cache current block receive time (ms):	0.6

db2

Avg global cache cr block receive time (ms):	0.3
Avg global cache current block receive time (ms):	0.4

问题：有可能造成这个问题的原因？

分享0

收藏0 回复只看该作者道具举报

richard

2^#

发表于 2013-4-16 08:47:09

本帖最后由 richard 于 2013-4-16 08:52 编辑

5、
关于RAC interconnect之Flow Control
根据 http://feed.askmaclean.com/archives/%E5%85%B3%E4%BA%8Erac-interconnect%E4%B9%8Bflow-control.html

主机内核版本：2.6.18-194.el5

两边的awr报告中：
gc blocks lost 0 0.00 0.00

是否可以无视flow control打开与否？

6、internal network 走了千兆的交换机
互相ping private IP 延时<0.05ms

回复只看该作者道具举报

Maclean Liu(刘相兵

3^#

发表于 2013-4-16 16:56:56

2个节点的

ifconfig

netstat -s

netstat -in

发一下

回复只看该作者道具举报

richard

4^#

发表于 2013-4-16 17:22:50

db1：
bond0    Link encap:Ethernet  HWaddr 00:25:B5:70:01:6D
      inet addr:192.168.108.11  Bcast:192.168.111.255  Mask:255.255.252.0
      inet6 addr: fe80::225:b5ff:fe70:16d/64 Scope:Link
      UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
      RX packets:2679636801 errors:0 dropped:0 overruns:0 frame:0
      TX packets:2053648627 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:1329094819282 (1.2 TiB)  TX bytes:2026867922237 (1.8 TiB)

bond0:1 Link encap:Ethernet  HWaddr 00:25:B5:70:01:6D
      inet addr:192.168.108.12  Bcast:192.168.111.255  Mask:255.255.252.0
      UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

bond1    Link encap:Ethernet  HWaddr 00:25:B5:80:00:8D
      inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0
      inet6 addr: fe80::225:b5ff:fe80:8d/64 Scope:Link
      UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
      RX packets:3709268404 errors:0 dropped:0 overruns:0 frame:0
      TX packets:5031666807 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:2208025910745 (2.0 TiB)  TX bytes:3694033039705 (3.3 TiB)

eth0    Link encap:Ethernet  HWaddr 00:25:B5:70:01:6D
      UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
      RX packets:2678799226 errors:0 dropped:0 overruns:0 frame:0
      TX packets:2053648627 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1024
      RX bytes:1328971400560 (1.2 TiB)  TX bytes:2026867922237 (1.8 TiB)
      Interrupt:177

eth1    Link encap:Ethernet  HWaddr 00:25:B5:80:00:8D
      UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
      RX packets:3709268370 errors:0 dropped:0 overruns:0 frame:0
      TX packets:5031666807 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1024
      RX bytes:2208025908705 (2.0 TiB)  TX bytes:3694033039705 (3.3 TiB)
      Interrupt:186

eth2    Link encap:Ethernet  HWaddr 00:25:B5:80:00:8D
      UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
      RX packets:34 errors:0 dropped:0 overruns:0 frame:0
      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1024
      RX bytes:2040 (1.9 KiB)  TX bytes:0 (0.0 b)
      Interrupt:185

eth3    Link encap:Ethernet  HWaddr 00:25:B5:70:01:6D
      UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
      RX packets:837577 errors:0 dropped:0 overruns:0 frame:0
      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1024
      RX bytes:123419892 (117.7 MiB)  TX bytes:0 (0.0 b)
      Interrupt:194
lo       Link encap:Local Loopback
      inet addr:127.0.0.1  Mask:255.0.0.0
      inet6 addr: ::1/128 Scope:Host
      UP LOOPBACK RUNNING  MTU:16436  Metric:1
      RX packets:54916148 errors:0 dropped:0 overruns:0 frame:0
      TX packets:54916148 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:46166347268 (42.9 GiB)  TX bytes:46166347268 (42.9 GiB)
virbr0 Link encap:Ethernet  HWaddr 00:00:00:00:00:00
      inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
      inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
      TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:0 (0.0 b)  TX bytes:8843 (8.6 KiB)
============================================
      Ip:
1785420005 total packets received
0 forwarded
0 incoming packets discarded
1014319789 incoming packets delivered
1154579706 requests sent out
644 fragments dropped after timeout
1007333984 reassemblies required
236515179 packets reassembled ok
1466 packet reassembles failed
423609330 fragments received ok
2114126743 fragments created
Icmp:
444803 ICMP messages received
2 input ICMP message failed.
ICMP input histogram:
      destination unreachable: 303050
      timeout in transit: 5
      echo requests: 141746
      echo replies: 2
444686 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
      destination unreachable: 302931
      time exceeded: 3
      echo request: 6
      echo replies: 141746
IcmpMsg:
      InType0: 2
      InType3: 303050
      InType8: 141746
      InType11: 5
      OutType0: 141746
      OutType3: 302931
      OutType8: 6
      OutType11: 3
Tcp:
797601 active connections openings
1754621 passive connection openings
379 failed connection attempts
333627 connection resets received
332 connections established
2351662824 segments received
2088857923 segments send out
48092 segments retransmited
0 bad segments received.
737154 resets sent
Udp:
2956817269 packets received
302915 packets to unknown port received.
1 packet receive errors
3360196295 packets sent
TcpExt:
39506 invalid SYN cookies received
3 resets received for embryonic SYN_RECV sockets
98 packets pruned from receive queue because of socket buffer overrun
482738 TCP sockets finished time wait in fast timer
239 time wait sockets recycled by time stamp
38 packets rejects in established connections because of timestamp
3275876 delayed acks sent
27176 delayed acks further delayed because of locked socket
Quick ack mode was activated 129913 times
1521424051 packets directly queued to recvmsg prequeue.
2156854021 packets directly received from backlog
1119771939 packets directly received from prequeue
43 packets dropped from prequeue
322539085 packets header predicted
1393978878 packets header predicted and directly queued to user
299779420 acknowledgments not containing data received
1452242874 predicted acknowledgments
1045 times recovered from packet loss due to fast retransmit
3517 times recovered from packet loss due to SACK data
Detected reordering 1 times using FACK
Detected reordering 17 times using SACK
Detected reordering 38 times using time stamp
191 congestion windows fully recovered
4739 congestion windows partially recovered using Hoe heuristic
14567 congestion windows recovered after partial ack
199030 TCP data loss events
TCPLostRetransmit: 2
4 timeouts after reno fast retransmit
994 timeouts after SACK recovery
17605 fast retransmits
341 forward retransmits
2920 retransmits in slow start
21555 other TCP timeouts
TCPRenoRecoveryFail: 707
1 sack retransmits failed
50 times receiver scheduled too late for direct processing
39074 packets collapsed in receive queue due to low socket buffer
130261 DSACKs sent for old packets
39045 DSACKs sent for out of order packets
13968 DSACKs received
231139 connections reset due to unexpected data
77910 connections reset due to early user close
14 connections aborted due to timeout
IpExt:
InMcastPkts: 514
OutMcastPkts: 95
InBcastPkts: 281446
=============================================================
Kernel Interface table
Iface    MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
bond0    1500 0 2679670871    0    0    0 2053677901    0    0    0 BMmRU
bond0:1 1500 0    - no statistics available -                         BMmRU
bond1    1500 0 3709316124    0    0    0 5031734053    0    0    0 BMmRU
eth0    1500 0 2678833293    0    0    0 2053677901    0    0    0 BMsRU
eth1    1500 0 3709316090    0    0    0 5031734053    0    0    0 BMsRU
eth2    1500 0    34    0    0    0       0    0    0    0 BMsRU
eth3    1500 0 837581    0    0    0       0    0    0    0 BMsRU
lo       16436 0 54916496    0    0    0 54916496    0    0    0 LRU
virbr0    1500 0       0    0    0    0    53    0    0    0 BMRU

回复只看该作者道具举报

richard

5^#

发表于 2013-4-16 17:23:29

db2：
bond0    Link encap:Ethernet  HWaddr 00:25:B5:70:00:6D
      inet addr:192.168.108.13  Bcast:192.168.111.255  Mask:255.255.252.0
      inet6 addr: fe80::225:b5ff:fe70:6d/64 Scope:Link
      UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
      RX packets:342685291 errors:0 dropped:0 overruns:0 frame:0
      TX packets:389412509 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:140736949621 (131.0 GiB)  TX bytes:406254457085 (378.3 GiB)

bond0:1 Link encap:Ethernet  HWaddr 00:25:B5:70:00:6D
      inet addr:192.168.108.14  Bcast:192.168.111.255  Mask:255.255.252.0
      UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

bond1    Link encap:Ethernet  HWaddr 00:25:B5:80:00:2D
      inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
      inet6 addr: fe80::225:b5ff:fe80:2d/64 Scope:Link
      UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
      RX packets:5033841023 errors:0 dropped:0 overruns:0 frame:0
      TX packets:3708076434 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:3696052757199 (3.3 TiB)  TX bytes:2207200178801 (2.0 TiB)

eth0    Link encap:Ethernet  HWaddr 00:25:B5:70:00:6D
      UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
      RX packets:341847688 errors:0 dropped:0 overruns:0 frame:0
      TX packets:389412526 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1024
      RX bytes:140613526917 (130.9 GiB)  TX bytes:406254465141 (378.3 GiB)
      Interrupt:177

eth1    Link encap:Ethernet  HWaddr 00:25:B5:80:00:2D
      UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
      RX packets:5033840989 errors:0 dropped:0 overruns:0 frame:0
      TX packets:3708076434 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1024
      RX bytes:3696052755159 (3.3 TiB)  TX bytes:2207200178801 (2.0 TiB)
      Interrupt:186

eth2    Link encap:Ethernet  HWaddr 00:25:B5:80:00:2D
      UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
      RX packets:34 errors:0 dropped:0 overruns:0 frame:0
      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1024
      RX bytes:2040 (1.9 KiB)  TX bytes:0 (0.0 b)
      Interrupt:185

eth3    Link encap:Ethernet  HWaddr 00:25:B5:70:00:6D
      UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
      RX packets:837611 errors:0 dropped:0 overruns:0 frame:0
      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1024
      RX bytes:123423232 (117.7 MiB)  TX bytes:0 (0.0 b)
      Interrupt:194

lo       Link encap:Local Loopback
      inet addr:127.0.0.1  Mask:255.0.0.0
      inet6 addr: ::1/128 Scope:Host
      UP LOOPBACK RUNNING  MTU:16436  Metric:1
      RX packets:44039137 errors:0 dropped:0 overruns:0 frame:0
      TX packets:44039137 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:12959690285 (12.0 GiB)  TX bytes:12959690285 (12.0 GiB)

virbr0 Link encap:Ethernet  HWaddr 00:00:00:00:00:00
      inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
      inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
      TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:0 (0.0 b)  TX bytes:8843 (8.6 KiB)
===============================================
Ip:
1096304036 total packets received
0 forwarded
0 incoming packets discarded
3701347580 incoming packets delivered
3371725953 requests sent out
435 fragments dropped after timeout
2112307185 reassemblies required
422664951 packets reassembled ok
1195 packet reassembles failed
235559670 fragments received ok
1005439803 fragments created
Icmp:
444641 ICMP messages received
3 input ICMP message failed.
ICMP input histogram:
      destination unreachable: 302230
      timeout in transit: 7
      echo requests: 142317
      echo replies: 87
444491 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
      destination unreachable: 302075
      time exceeded: 5
      echo request: 94
      echo replies: 142317
IcmpMsg:
      InType0: 87
      InType3: 302230
      InType8: 142317
      InType11: 7
      OutType0: 142317
      OutType3: 302075
      OutType8: 94
      OutType11: 5
Tcp:
761963 active connections openings
789583 passive connection openings
374 failed connection attempts
333851 connection resets received
50 connections established
342108188 segments received
415960793 segments send out
133163 segments retransmited
0 bad segments received.
588806 resets sent
Udp:
3358432736 packets received
302053 packets to unknown port received.
1 packet receive errors
2955187503 packets sent
TcpExt:
39411 invalid SYN cookies received
165 packets pruned from receive queue because of socket buffer overrun
307242 TCP sockets finished time wait in fast timer
141 time wait sockets recycled by time stamp
112 packets rejects in established connections because of timestamp
8410120 delayed acks sent
5903 delayed acks further delayed because of locked socket
Quick ack mode was activated 114212 times
210798895 packets directly queued to recvmsg prequeue.
1543350707 packets directly received from backlog
3357983888 packets directly received from prequeue
44390060 packets header predicted
109314221 packets header predicted and directly queued to user
55194868 acknowledgments not containing data received
150502784 predicted acknowledgments
18 times recovered from packet loss due to fast retransmit
1339 times recovered from packet loss due to SACK data
Detected reordering 8 times using SACK
Detected reordering 4 times using time stamp
66 congestion windows fully recovered
1142 congestion windows partially recovered using Hoe heuristic
2614 congestion windows recovered after partial ack
86936 TCP data loss events
83 timeouts after SACK recovery
3 timeouts in loss state
6066 fast retransmits
2701 forward retransmits
89 retransmits in slow start
43449 other TCP timeouts
TCPRenoRecoveryFail: 15
1774 times receiver scheduled too late for direct processing
7360 packets collapsed in receive queue due to low socket buffer
114471 DSACKs sent for old packets
36051 DSACKs sent for out of order packets
3203 DSACKs received
153155 connections reset due to unexpected data
81317 connections reset due to early user close
2 connections aborted due to timeout
IpExt:
InMcastPkts: 559
OutMcastPkts: 92
InBcastPkts: 281457
=================================================
Kernel Interface table
Iface    MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
bond0    1500 0 342695656    0    0    0 389421887    0    0    0 BMmRU
bond0:1 1500 0    - no statistics available -                         BMmRU
bond1    1500 0 5034074353    0    0    0 3708263806    0    0    0 BMmRU
eth0    1500 0 341858034    0    0    0 389421887    0    0    0 BMsRU
eth1    1500 0 5034074319    0    0    0 3708263806    0    0    0 BMsRU
eth2    1500 0    34    0    0    0       0    0    0    0 BMsRU
eth3    1500 0 837624    0    0    0       0    0    0    0 BMsRU
lo       16436 0 44040050    0    0    0 44040050    0    0    0 LRU
virbr0    1500 0       0    0    0    0    52    0    0    0 BMRU

回复只看该作者道具举报

Maclean Liu(刘相兵

6^#

发表于 2013-4-16 20:34:41

644 fragments dropped after timeout
1466 packet reassembles failed

435 fragments dropped after timeout
1195 packet reassembles failed

drop fragments 还是有一些的

实际cluster 等待如何、是否有gc block lost?

回复只看该作者道具举报

richard

7^#

发表于 2013-4-17 15:47:54

不好意思，回复晚了。

Wait Class	Waits	%Time -outs	Total Wait Time (s)	Avg wait (ms)	%Total Call Time
CPU time			919		48.15
User I/O	103,693		505	5	26.45
Commit	668,273	0	437	1	22.88
System I/O	703,823		206	0	10.78
Cluster	201,396		75	0	3.96
Other	64,587	72	28	0	1.49
Network	1,559,392		19	0	0.99
Concurrency	4,410		4	1	0.20
Application	2,508	0	3	1	0.16
Configuration	1,510	0	1	1	0.08

gc blocks lost

0

0.00

Cluster wait 还是有一些的，是否需要全部awr报告，可否单独发于您？

回复只看该作者道具举报

返回列表

		自动登录	找回密码
密码			注册