LACP Bonding in XenServer - Configuration and Troubleshooting

[cicong]

In XenServer 6.1, LACP link aggregation was added to existing bonding modes for vSwitch. This article describes dynamic link aggregation (LACP) between a XenServer host and a switch, giving a high-level overview of the 802.3ad protocol, explaining configuration and diagnosis tools.

Background

LACP Bonds

Configuring LACP on a switch

Troubleshooting

Diagnostics

Background NIC Bonding

NIC bonding is a technique where two or more network cards (NICs) are configured together in order to logically function as one. It can be used for the following reasons:

• Redundancy: in case of link failure all traffic should get shifted seamlessly to the remaining NIC(s).
• Throughput aggregation: it is usually more cost-effective to bundle a few 1Gb NICs than to upgrade from 1Gb to 10Gb.

Other terms frequently used for NIC bonding are: NIC teaming, network bonding, link aggregation, link bundling or port bundling. However, it is perceived to be more correct to use link aggregation/bundling to describe configurations where set-up is required on both sides, so both endpoints are aware of the aggregation (for example, when configuration is done on both on the server side and the switch side).

Bonding modes in XenServer 6.0

In XenServer 6.0, only two bonding modes were supported for both network stacks (Linux Bridge and vSwitch):

• active-backup (active-passive): only one NIC would be actively used for traffic and only in case of its failure would an inactive NIC take over,
• balance-slb (active-active): traffic load balancing based on the source MAC address of each Virtual Network Interface (VIF) of a guest.

Both active-backup and balance-slb do not require configuration on the switch side.

In addition to the preceding options, a few unsupported bonding modes existed in Linux Bridge, notably link aggregation 802.3ad.

The default bonding mode, balance-slb, has the benefit of throughput aggregation and failover, but load balancing can work well only for a sufficient number of guests(or rather VIFs), since traffic from one VIF is never split between multiple NICs. Also, its frequent rebalancing was known to cause issues in some switches (see CTX134947 - Adjusting the bond balance interval in XenServer 6.1.0).

Bonding modes in XenServer 6.1

In XenServer 6.1, LACP support was implemented for the vSwitch network stack and LACP link aggregation was added to existing bond modes. LACP is supported only for vSwitch.

LACP bonds Link aggregation

Link aggregation is defined as a configuration in which a group of ports is aggregated together and treated like a single interface. Its advantages are: throughput aggregation, load balancing and failover.

Static and dynamic LAG

On a switch, ports are usually grouped together by assigning them the same LAG (Link Aggregation Group) number.

There are two types of LAGs:

• Static LAG: ports have LACP disabled and become automatically active members of the bond. Static LAG is not widely used, as it is often considered obsolete and inferior to dynamic LAG. With static LAG on the switch, the bond mode should be balance-slbrather than lacp. Note that use of static LAG is not supported.
• Dynamic LAG: Link Aggregation Control Protocol (LACP) is used for switch-server communication, in order to negotiate dynamically which links should be active and which should be in stand-by mode.

Two names

IEEE 802.3ad, introduced in 2000, was the original standard describing link aggregation and LACP. In order to resolve a layer ordering discrepancy, it was later formally transferred to 802.1 group, becoming IEEE 802.1AX-2008 (with no technical changes). However, the name 802.3ad remains widely used.

LACPDU frames

When using LACP protocol, both sides (the switch and the server) regularly exchange LACPDU frames. They contain all the information necessary to negotiate active and stand-by links, monitor the state of the links and notify the partner about a potential failure, providing a prompt failover.

Actor and Partner

Terms “Actor” and “Partner” are frequently seen when using LACP protocol. These are relative definitions: “Actor” means “this device” and “Partner” is “other device”, so the server will describe itself as the Actor with the switch being its Partner, while the switch sees itself as the Actor and refers to the server as the Partner.

Fallback to balance-slb

In the current implementation, in case of unsuccessful LACP negotiation, XenServer will automatically revert to balance-slb behavior. The server still keeps monitoring the traffic for LACP frames, so if a handshake with the switch is achieved, the bond mode will change to LACP.

Load balancing

Outgoing packets are distributed between active links. Both sides of the link — the switch and the server — are responsible for balancing their respective outgoing traffic. In XenServer, load balancing between links can be described as follows.

• A hashing algorithm assigns a hash number (0-255) to each packet, based on a mix of MAC, IP and TCP/UDP port as required.
• Each hash is assigned to one of the NICs on the bond, which means packets with the same hash are always sent through the same NIC.
• If a new hash is found, it is assigned to the NIC that currently has the lowest utilization.
• Rebalancing occurs at a regular interval — hashes are redistributed between the NICs to ensure all NICs are utilized to approximately the same extent.

Hashing algorithm

The hashing algorithm is chosen independently on both switch and server side, as each device balances its outgoing traffic. Rather than using the same method on both sides, hashing algorithm choice should reflect traffic patterns in each direction.

On the server side, vSwitch provides two hashing algorithms to choose from:

• tcpudp_ports (default), based on source and destination IP, TCP/UDP ports and source MAC.
• src_mac, based on source MAC address — this is the same mechanism as the one used already in balance-slb mode (traffic to/from a guest is not split).

On the switch side, the hashing algorithms depend on the switch brand and can have different names.

Use of TCP/IP parameters for hashing should provide load balancing for management traffic as well as improve it for the case of a low number of guest, as it allows packets from the same guest to be distributed over different NICs at the same time. However, neither of the hashing algorithms presents benefits for storage traffic, as in a typical configuration large amounts of data are sent using the same source and destination IP and ports, and while the endpoints remain the same, all packets will have the same hash assigned and so will be sent through the same NIC. Consider using multipathing rather than bonding for storage traffic.

In the case of src_mac hashing, there is a non-negligible probability that two different MAC addresses will get the same hash number assigned — and when such “MAC clash” occurs, the two VIFs will be always using the same link. In general, src_mac hashing cannot provide even traffic distribution over the links if the number of VIFs is smaller than the number of NICs.

Limitations

The following limitations apply:

• LACP for Linux Bridge is not supported.
• Citrix supports bonding of up to four NICs in XenServer.
• Cisco EtherChannel is not supported.
• Cisco Port Aggregation Protocol (PAgP) is not supported.

Creating LACP bond using XenCenter

It is possible to create a LACP bond in XenCenter. The procedure is similar to creating bonds of any other type. Either go to the "Networking" tab and choose "Add Network", and then "Bonded Network", or alternatively, click the "Create Bond" button on the"NICs" tab. Two types of LACP bonds should be displayed: "LACP with load balancing based on IP and port of source and destination" and "LACP with load balancing based on source MAC address". Load balancing based on IP and port of source and destination is the default hashing algorithm for LACP and it should work well for most typical configurations.

Creating LACP bond using XenServer command line

LACP bond can be created in dom0 command line as follows.

xe bond-create mode=lacp network-uuid=<network-uuid> pif-uuids=<pif-uuids>

Hashing algorithm can be specified at the creation time (default is tcpudp_ports):

xe bond-create mode=lacp properties:hashing-algorithm=<halg> network-uuid=<network-uuid> pif-uuids=<pif-uuids>

where <halg> is src_mac or tcpudp_ports.

You can also change the hashing algorithm for an existing bond, as follows:

xe bond-param-set uuid=<bond-uuid> properties:hashing_algorithm=<halg>

It is possible to customize the rebalancing interval by changing the bond PIF parameter other-config:bond-rebalance-interval and then re-plugging the PIF. The value should be expressed in millisecond. For example, following commands change the rebalancing interval to 30 seconds.

xe pif-param-set other-config:bond-rebalance-interval=30000 uuid=<pif-uuid> 
xe pif-plug uuid=<pif-uuid>

The two LACP bond modes will not be displayed if you use a version older than XenCenter 6.1 or if you use Linux Bridge network stack.