Tales of Load Balancing Horror

It’s halloween, and I think it’s time for some scary stories.  I give you…

TALES OF LOAD BALANCING HORRORS!

For tales of persistent terror, challenging your keep-alive, I give you the following vignettes. (Names have been changed to protect my ass, as well as to punch up some bone dry material.)

The Default Gateway To The Abandoned Zone

Several years ago, on a dark and stormy night, a dashingly handsome young hot-shot system administrator had just finished setting up a new web infrastructure for a client.  They were moving their web infrastructure from their own facility, to the co-location facility where Mr. Sysadmin worked.  Mr Sysadmin was also responsible for the load balancer.  Running bravely into the load balancing realm while both the other sysadmins and the network admins dared not tread.

“It’s cursed!” They said.  Or maybe they used curse words to describe it.  It was a long time ago.

He powered up the system, tested the traffic, and cried out into the night “It’s alive! It’s pushing traffic!”  Overly pleased with his unnatural creation, he emailed the client to tell them their configuration was ready.  They moved in, with administrator access to all systems.

At first, the infrastructure worked as promised.  Sites were served, and loads were balanced.  Then, a call came from beyond the datacenter.

“The load balancer is screwing up. The site is down.” said the customer.  From beyond the data center.

Our hero was not convinced.  Many plagues have been blamed upon the load balancer, only to later find out the culprit was elsewhere.  So he punched up the website, and sure enough, nothing. He logged into the load balancer, and found it to be operating correctly, with no changes from when it was working.  He then checked the servers.  And there was the problem.

He shot up from his haunted Aeron chair.  “By Zeus, the default gateway has been changed!”

For you see, as traffic comes into the load balancer, it must also return through the load balancer on the way out.  This is the way of things.  This can be done a number of ways, and the method chosen for this infrastructure was by making the load balancer the default gateway.  But when someone changes the default gateway to a device other than the load balancer, the packets are doomed to wander the network, never to find their destination.  They were damend to the bit bucket.

So our mad sysadmin (he was pretty mad,  as he had specifically instructed them not to change the default gateway) changed the default gateways correctly, so that more packets would not suffer the same ghastly fate.  Once this task had been completed, the packets found their way back the client, and all worked.   (The lost packets still haunt the data center to this day!)

Teh End…

or is it?