In large to medium sized organizations, those responsible for load balancers typically are in the networking group. However, because of the rather unique function of load balancers, their operation is tied closely to the applications in ways that switches, routers, and even firewalls aren’t.
To fully understand load balancers require skills from career paths that don’t typically converge: Those of a server administrator (and application developer) and those of a network engineer.
This strange convergence in skill requirements often means that load balancers are misunderstood. It’s one of the reasons why “It’s always the load balancer“.
I’ve found though that there are a few concepts and tricks that if learned by opposing skill sets can go a long way into understanding/dealing with/troubleshooting load balancing technology.
What a network engineer load balancer administrator needs to know about servers
- Default routes of the servers. I don’t keep statistics on the hundreds of issues I’ve diagnosed over the years, but I’d guess fully 50% involved default routes on the servers as either the primary problem, or a related problem. This is true for every vendor I’ve worked with (or for).
- Learn to use liveHTTP headers for Firefox and/or ieHTTPheaders for Internet Explorer. This shows you exactly what the server is doing, and can help you answer questions like “why is the load balancer redirecting me to Google?†(The answer is no, it’s not, the application is doing that, and here’s the HTTP 302 in the header to prove it.)
- If you’re setting up a non-transparent setup (known as full-NAT) where the IP address of the web clients are hidden, make sure that your server people are aware of this (and OK with this). Otherwise, when the server people call asking where the IP addresses of their clients are, and you have to explain to them that the IPs are lost forever, and that the load balancer doesn’t log them, it’s going to get unpleasant.
- Setup MRTG for your load balancer. Pull as many stats as you can get your grubby little paws on.
What a server admin/application developer needs to know about load balancers
- If you’ve got a tricky problem that can be solved by a solution in a load balancer and in an application, do it in an application. It may seem seductive to have the load balancer take care of specialized cases with specialized URL parsing or redirects, but it can greatly complicate your setup. For one thing, you have to go to a different group in order to make a change. This doesn’t include the core functionality of load balancers, such as cookie persistence of course.
- Know how to do a network capture, using TCPDump or WindDump. This is a language network engineers understand, and even if you don’t understand what it means, the network group can often pick through it and figure it out. It would help greatly to learn to parse it yourself, as well. It would help to know how to read Ethereal (available for Linux/BSD/Unix and Windows).
- Load balancers don’t log client connections. There’s only one load balancer that I know of that could even configure such logging, but it wouldn’t be recommended and would likely impact performance.
Have a tip? Leave a comment.
