Load Balancing Digest

EtherChannel is a strange creature.  It’s generally known as a way to load balance Ethernet traffic over multiple links.  Need more than 1 Gigabit of throughput, but don’t have 10 Gigabit interfaces yet?  Then bond two Gigabit links together in an EtherChannel link, and you’ve got 2 Gigabits.  That’s the idea, at least.  However, it’s not quite that simple. In many cases, it doesn’t behave the way that people would expect it to, and for that, EtherChannel is both feared and loathed.

Recently, this came up on the load balancing mailing list.  An Alteon was setup with EtherChannel, but the traffic only went over one link.   As strange as it seems at first, this is actually the way EtherChannel works in his environment, and technically it’s working correctly.  To understand why, you need to understand EtherChannel.

I could go into a long treatise on the various components of EtherChannel, but Wikipedia has it pretty well covered, so instead, I came up with the 3 Things You Need To Know About Etherchannel. Know these 3 things, and you can Google the rest.

#1: Etherchannel may not always balance load evenly (or at all) across multiple links.

Simply turning on EtherChannel does not necessarily create even load distribution across multiple bonded links.   You may get nearly perfectly distributed load, or you may get all your traffic running over a single link.  There are tweaks you can do to resolve uneven distribution depending on the situation, but there are also situations where load distribution isn’t possible at all.

#2: How Etherchannel divvies up the traffic

EtherChannel does not work by round-robin, but instead works by hashing a characteristic of an Ethernet frame. The hash determines which of the links the frame travels to in order to get to the other side.

The most basic characteristic to hash on is the source or the destination MAC address. Most devices that are capable of EtherChannel support one of those two modes (or both).

In addition, some advanced switches can hash based on source or destination IP address, and even source or destination IP address and port configuration. These are typically more advanced Layer 2/3 combo devices.  Someone from the load balancing mailing list posted a link to this article on Cisco’s site which explains which modes are available on their line of switches.

As you can probably guess at this point, that a lack of diversity of whatever characteristic you use can cause uneven load distribution.  If you’re doing the hash on the MAC address, but only one MAC address is used to send traffic to only one other MAC address, that traffic will always go over the same link.  The other links in the EtherChannel group won’t get used at all.

In fact, if you have a single MAC address, IP address, and TCP port sending traffic to a single MAC address, IP address, and TCP port (such as an FTP transfer from one host to another) load distribution is actually impossible.  Everything hashes out the same link.

Most likely, however, one of the methods (source or destination on the MAC, IP, or IP/port) will typically get you pretty good load distribution.  If it’s available on your device, that is.  Again, many devices only support source/desination MAC addresses.

#3: The sending side alone decides which link the traffic goes over

When a device determines that a frame is to go over an EtherChannel link to reach its destination, the device sending the frame is the one that chooses which of the available individual links the frame goes down. The receiving end has absolutely no say in the matter.

As a result, to obtain proper load distribution, you must configure each end individually for the best hashing characteristic available.  One device may support only MAC address hashing, while the other device may support MAC and IP.  You can mix and match.  The other ends don’t care how the frames get to them.