关于11G RAC新特性haip的理解

kim_meng

从11.2.0.2之后，RAC又引入了一个新特性HAIP，用来实现RAC私网高可用性的，metalink上面说HAIP可以实现私网的负载均衡和故障切换，最大可以支持四个网卡同时活跃状态。
详情metalink：11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip [ID 1210883.1]

那么这是否可以说：以后RAC安装私网没有必要使用第三方的技术来实现私网的高可用性了，如bonding、etherchannel等。。
HAIP既可以负载均衡，有可以故障切换，还不用配置，RAC安装完后自动实现，十分方面。

另外，使用四个网卡的私网HAIP技术，是否对私网交换机有特殊的要求？

谢谢高人回答。

[ 本帖最后由 kim_meng 于 2012-2-15 22:22 编辑 ]

Maclean Liu(刘相兵

=== ODM Research ===
(Note: This is INTERNAL ONLY research. No action should be taken by the customer on this information.
This is research only, and may NOT be applicable to your specific situation.)
KM SEARCH
--------------------------
Keywords: 11.2 haip
Found:
11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip (Doc ID 1210883.1)

Product Documentation
====================
#1.
Oracle® Clusterware Administration and Deployment Guide
11g Release 2 (11.2)
Part Number E16794-15
2 Administering Oracle Clusterware
Redundant Interconnect Usage
Note:

Redundant Interconnect Usage enables load-balancing and high availability across multiple (up to four) private networks (also known as interconnects).
Oracle Clusterware uses at most four interfaces at any given point, regardless of the number of interfaces defined. If one of the interfaces fails, then the HAIP address moves to another one of the configured interfaces in the defined set.

When there is only a single HAIP address and multiple interfaces from which to select, the interface to which the HAIP address moves is no longer the original interface upon which it was configured. Oracle Clusterware selects the interface with the lowest numerical subnet to which to add the HAIP address.

#2.
Oracle® Grid Infrastructure Installation Guide
11g Release 2 (11.2) for IBM AIX on POWER Systems (64-Bit)
Part Number E24614-01

Redundant Interconnect Usage

In previous releases, to make use of redundant networks for the interconnect, bonding, trunking, teaming, or similar technology was required. Oracle Grid Infrastructure and Oracle RAC can now make use of redundant network interconnects, without the use of other network technology, to enhance optimal communication in the cluster. This functionality is available starting with Oracle Database 11g Release 2 (11.2.0.2).

If you define more than four interfaces as private network interfaces, be aware that Oracle Clusterware activates only four of the interfaces at a time. However, if one of the four active interfaces fails, then Oracle Clusterware transitions the HAIP addresses configured to the failed interface to one of the reserve interfaces in the defined set of private interfaces.

#3.
Oracle® Grid Infrastructure Installation Guide
11g Release 2 (11.2) for IBM AIX on POWER Systems (64-Bit)
Part Number E24614-01
2 Advanced Installation Oracle Grid Infrastructure for a Cluster Preinstallation Tasks

Oracle does not support token-rings or crossover cables for the interconnect.

Please refer to MOS note: 11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip (Doc ID 1210883.1)
It has configuration example.

You can also refer to product documentation:
Oracle® Grid Infrastructure Installation Guide
11g Release 2 (11.2) for IBM AIX on POWER Systems (64-Bit)
Part Number E24614-01
2 Advanced Installation Oracle Grid Infrastructure for a Cluster Preinstallation Tasks

Maclean Liu(刘相兵

"那么这是否可以说：以后RAC安装私网没有必要使用第三方的技术来实现私网的高可用性了，如bonding、etherchannel等。。
HAIP既可以负载均衡，有可以故障切换，还不用配置，RAC安装完后自动实现，十分方面。"

正确， 并非完全不用配置， 在安装过程中 选择private network即可


"是否对私网交换机有特殊的要求？"

一般interconnect network 推荐至少 千兆交换机

Maclean Liu(刘相兵

注意存在 11.2 上  同一subnet下的network interface failover不成功的 bug

具体参考

bug 10389682 (further, please check bug 11770054 ), according to this bug, interface with same subnet may be failed to failover

=== ODM Answer ===
Bug 10389682 points to its base bug 1170054, the later is closed as not a bug.
But these bugs really help us to understand following:
The background for this issue is that in Note:1210883.1 or product documentation,
Oracle did not go into details like
- whether the NICs should be in same subnet
- how to wire all these NICs to one or more switches (network topology)
- relationship between routing table and NICs order
,etc.

In general, HAIP only provide a general solution to replace NIC bonding on OS level.
So practice is needed in specific customer's environment in order to finally avoid the following problem in alert log when starting db instance:

No connectivity to other instances in the cluster during startup. Hence, LMON is terminating the instance. Please check the LMON trace file for details. Also, please check the network logs of this instance along with clusterwide network health for problems and then re-start this instance.
LMON (ospid: 1220768): terminating the instance

kim_meng

据我所知，AIX上面要使用etherchannel的负载均衡的话，必须交换机支持，并且在交换机上面做网口绑定。
这里的HAIP貌似普通交换机均可以实现负载均衡呀。

Maclean Liu(刘相兵

Bug 11770054: NEED CLARIFICATION ON NETWORK REQUIREMENT OF PRIVATE INTERCONNECT

Type         D - Documentation         Fixed in Product Version         -
Severity         2 - Severe Loss of Service         Product Version         11.2.0.2.0
Status         92 - Closed, Not a Bug         Platform         46 - Linux x86
Created         14-Feb-2011         Platform Version         NO DATA
Updated         18-Mar-2011         Base Bug         -
Database Version         11.2.0.2.0                
Affects Platforms         Generic                
Product Source         Oracle                

Show Related Products Related Products
Line         Oracle Database Products         Family         Oracle Database
Area         Oracle Database         Product         5 - Oracle Server - Enterprise Edition

Hdr: 11770054 11.2.0.2.0 PCW 11.2.0.2.0 PRODID-5 PORTID-46
Abstract: NEED CLARIFICATION ON NETWORK REQUIREMENT OF PRIVATE INTERCONNECT


PROBLEM:
--------
From 11.2.0.2, we support HAIP, the multiple private NICs natively (not
bonding required). With this env, we have seen several bugs shows that when

1). subnet on these multiple NICs are the same
2). When cable pulled from one of these NICs, OS is rebooted, which is not
expected.

these bugs are: bug 10638686, bug 10389682, and bug:10277115.

DIAGNOSTIC ANALYSIS:
--------------------
In bug 10277115, Dev AHABBAS commented the following and the bug was closed
as 32.


============================
The problem is that if you have multiple private NICs on the same subnet,
then there are requirements for the way the hardware is cabled.  This is not
specifically an HAIP or CRS issue, but a networking issue in general.

Since the movement of packets is going to be based on routing tables and
routes, when multiple nics with the same subnet are used, it is required that
either nic should be able to communicate to either nic on the other box.

That or some solution to ensure that the routing tables are moved properly.  
Otherwise, when any cable is pulled things will fail.

If on the other hand we are using separate subnets, then traffic is routed
more "naturally".
=============================

There is no mention in our doc about the above. So the purpose of this doc
bug is to clarify:

1). Should we ask ct to use different subnet on different NIC on HAIP?

2). If dev think the same subnet is ok but there is extra requirement as "
when multiple nics with the same subnet are used, it is required that either
nic should be able to communicate to either nic on the other box" then we
need to make it clear on the documentation, and also what does it really mean
by "either nic should be able to communicate to either nic on the other box",
what ct needs to check in order to make sure "either nic should be able to
communicate to either nic on the other box"? Any example of command to check
this?

We have run into situation that ct asked the exact question as how to check
"either nic should be able to communicate to either nic on the other box"



Bug 10389682: HAIP INTERCONNECT DOES NOT WORK IF IP ADDRESSES ARE IN THE SAME SUBNET

Type         B - Defect         Fixed in Product Version         -
Severity         2 - Severe Loss of Service         Product Version         11.2.0.2
Status         96 - Closed, Duplicate Bug         Platform         226 - Linux x86-64
Created         10-Dec-2010         Platform Version         NO DATA
Updated         16-May-2011         Base Bug         11770054
Database Version         11.2.0.2                
Affects Platforms         Generic                
Product Source         Oracle                

Show Related Products Related Products
Line         Oracle Database Products         Family         Oracle Database
Area         Oracle Database         Product         5 - Oracle Server - Enterprise Edition

Hdr: 10389682 11.2.0.2 PCW 11.2.0.2 GIPC PRODID-5 PORTID-226 11770054
Abstract: HAIP INTERCONNECT DOES NOT WORK IF IP ADDRESSES ARE IN THE SAME SUBNET

*** 12/10/10 05:45 am ***
*** 12/10/10 05:45 am *** (CHG: RDBMS Ver.-> NULL -> 11.2.0.2)
*** 12/10/10 05:45 am ***
  
  
  BUG TYPE CHOSEN
  ===============
  Supportability
  
  Component: Portable ClusterWare
  ===============================
  DETAILED PROBLEM DESCRIPTION
  ============================
  With 11.2.0.2 you can create a redundant interconnect by classifying
  multiple (up to 4) interfaces as private.
  The clusterware (ora.cluster_interconnect.haip) will then create virtual
  addresses) on top of these interfaces. When a interface fails the virtual
  interface is relocated on one of the other interfaces.
  
  in a separate subnet.
  If this is not the case, the clusterware will relocate the virtual
  interface, but if the failed interface is listed as first in the routing
  table the other node(s) will still not be able to ping/connect on any of the
  private addresses, resulting in a cluster node failure.
  
  Note that you will not see this behaviour when the interface is stopped with
  ifdown as this will also remove its entry from the routing table.
  
  This requirement is however not specified in the documentation nor does
  oifcfg gives an error when multiple interconnect interfaces with the same
  subnet are given
  
  We need to confirm if it is indeed a requirement to use different subnets
  for different interconnect interfaces or not
  
  DIAGNOSTIC ANALYSIS
  ===================
  We aplied the following test:
  
  -Collect the below output, when all the interconnect interfaces are working
  fine,
  
  oifcfg getif
  oifcfg iflist -p -n
  ifconfig -q
  route -n
  
  - Reproduce the issue, by making one of the interconnect interface to fail
  and get the below output:
  ifconfig -a
  route -n
  
  The customer feedback:
  
  I have uploaded the results of the requested tests, together with the
  logfiles from the clusterware.
  I have also included some additional comments in the test results.
  
  The following observations where made:
  
  When multiple subnets are used, a disconnect on one node results in the
  relocating of the HAIP vip to another nic on all the nodes.
  When only one subnet is used, this is not the case (unless the nic is
  unplumbed).