This post covers network topology, which is how the load balancer fits into the network.  How a device fits into the network is usually a difficult concept to get, and often that’s simply because people make it tougher than it need be. Basically, for a load balancer to be put into a network effectively, two things need to happen.

1. Traffic needs to flow through the load balancer on the way in

1. Traffic needs to flow through the load balancer on the way out

The first part is easy, as there’s only one way.  We direct traffic to the virtual IP (VIP) and port sitting on the load balancer.  This is the IP and port that pretends to be the server.  Getting traffic through the load balancer on the way out is probably one of the toughest concepts to grasp when learning load balancers, as there are several ways to accomplish this.

There’s on method of getting traffic through the load balancer on the way out that’s a quick way to drop a load balancer into an existing infrastructure with minimal changes to the network topology.  This is called one-armed, route-path.

One-armed, route path is not as popular as some of the other methods, although it has the distinct benefit of being a good, quick “drop-in” deployment.  Here’s how it works.

Let’s say you’ve got a network with a couple of servers sitting behind a firewall.  This firewall does NAT from a public address space to private IPs. This is a pretty common scenario for a small to medium sized business.

In the example shown above, the default gateway for the servers is the firewall, at 192.168.1.1.  To network admins, The concept of a default gateway is second nature.  To server folks, keep this in mind:  If you want to send IP traffic to a system not on your local network, you need a router to handle delivery.  That is your default gateway.  Without a default gateway for your servers, you can’t communicate with the Internet.

So now lets say we want to drop a load balancer into the network.  There are several options, and for the most part the advantages of one over another are logistical, not performance related.  For example, to do two-armed, Layer 3 path (arguably the most common topology), you would need to put in a new IP network between the firewall and the servers, and one new Layer 2 network.  This would require re-addressing the IPs on all the servers.

And while adding a new Layer 2 and Layer 3 network would certainly work, we can use one-armed, Layer 3 path without the need to re-IP all the servers or adding new networks.

In the figure above, you see that we’ve changed the default gateway on the servers to that of the administrative IP of the load balancer (if there were two load balancers, they would have a floating administrative IP which you would use as the default gateway).   The default gateway of the load balancer is that of the firewall.

This seems a little odd, as we’ve got two default gateways on the same IP network.  While unusual, it works, and it’s a handy way to drop a load balancer into a network with minimal changes.

The idea is that if everything goes terribly wrong, at least your visitors will see something, instead of nothing.

Which begs the question: How do you like to fail?  Fail fast or fail slow? Would it be better to fail slow, where your site becomes slower and slower, or possibly just unresponsive, or would it be better to put up a quick-serving sorry page if the infrastructure melts?

A wildly successful website can easily become a victim of its own success.  Take the case of two sites that experienced exponential growth in a relatively short period of time:  Twitter.com and Myspace.com.

They took two different paths in the realm of failure.  One failed fast, and one failed slow.

Although Myspace has lost most of its lead to Facebook, it’s still a wildly popular social media site.  They had exponential growth from their start in 2003, and there were many periods of time when Myspace.com was just… slow.  Really really slow. You can’t really blame them.  It’s tough when users come faster than you can install servers and provision bandwidth.  It’s a happy problem to have usually, but it’s still a logistical challenge.

Twitter.com came around a bit later, but it also had exponential growth and problems coping.  But for the most part, they failed in a different way:  Fail Whale. When something went terribly awry, instead of a slow site, you’d get a very quick fail whale image.

Perhaps this is a matter of personal opinion, but I think if you’re going to fail, it’s better to fail quick than fail slow.  That is, have a sorry page or sorry site that comes up quick, rather than a site that is too slow for anyone to use.

The quick sorry page can be done with many of the load balancing/ADC vendors by using the backup/sorry serverfarm feature.  Keeping a group of reserve servers, serving up only a “oops, sorry about that” type of page, your own fail whale, can be better than having a really slow or unresponsive web site.

Of course, you won’t always be able to choose the method of your failure.  If your upstream ISP goes dark, there’s not much you can do (unless you have an offsite fail site).  But I personally think having a fail site is a more “professional” way to fail than having a slow or unresponsive site when things go belly up (and we all know they will).

Typically, Apache Bench (ab) is installed with the base Apache install, from at least Apache 1.3 on.  This includes when Apache is installed on Windows.

You can check all of the available options on the ab documentation page, but here’s a (very) quick reference to using it.

Two of the most important options are “-n” for the number of total connections, and “-c” for how many concurrent connections are done at the same time.

For instance, using the option “-n 1000″ will do 1,000 requests, one at a time, to a target URL.

ab -n 1000 http://website.com/

One at a time is rarely an effective test, so it’s best to use the “-c” option to specify a high number of concurrent connections, such as 100.

ab -n 1000 -c 100 http://website.com/

If you use concurrency, ab will split the total number of requests up amongst the concurrent settings.  For instance, using the option “-n 1000″ will do 1,000 connections, but “-n 2000 -c 100″ will only do 20 requests from 100 different connections (2,000 / 100 = 20).  So it’s best to use a much larger number of total connections if you’re doing concurrency.

ab -n 100000 -c 100 http://website.com/

When ab is finished running, it will spit out a performance report, including such info as the time taken for tests, requests per second, wait time, etc.

Finished 1000 requests

Server Software:        Apache/2.2.9
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        45 bytes

Concurrency Level:      10
Time taken for tests:   0.427 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      320640 bytes
HTML transferred:       45090 bytes
Requests per second:    2341.45 [#/sec] (mean)
Time per request:       4.271 [ms] (mean)
Time per request:       0.427 [ms] (mean, across all concurrent requests)
Transfer rate:          733.17 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   0.4      2       3
Processing:     0    2   0.5      2       7
Waiting:        0    2   0.5      2       6
Total:          0    4   0.7      4       8

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      5
  80%      5
  90%      5
  95%      5
  98%      5
  99%      5
 100%      8 (longest request)

The ab utility defaults to one request per TCP connection (KeepAlive turned off).  If you want to use KeepAlive, where multiple requests are made through a TCP connection, use the “-K” option, open up as many TCP connections as you specify in concurrency (“-c”) and make the total number of quests through those few open TCP connections.

The utility is a simple but power tool for testing load balancers and web servers.  It doesn’t tend to reflect real-world usage, but it can be useful for baseline testing and troubleshooting.  I’ve found it quite useful over the years.

Among the things I like about the article (including the word Asshattery, which is one of my favorite words, ever) is how he talks honestly and from experience the issues dealing with a rapidly growing site.  You’ve got all sorts of constraints in terms of application, database, network, CPU, disk I/O etc., and like a gangly teenager, you hit different growth limitations at different and unexpected times.

I’ve also faced the challenges he describes, although from a slightly difference perspective, and I know exactly what he’s talking about.  Things will come up that you can’t plan for, and there will be issues that you can plan for, but choose not to because of resource and time constraints.

Sometimes you’ve got to say “Yeah, this will probably bite me in the ass later, but what’s more important are the four things that are biting my ass right now.”

Users might balk at 1,000 TPS. After all, 1,000 SSL TPS just doesn’t sound like a lot. So how much is it, really?

To answer this, just do a simple calculation: Take the average page size and calculate the maximum number of TPS possible through a given connection (T1, 10 Meg line, DS3). I think you’ll find that the numbers are a bit surprising.

Let’s take a page size of 20K. That’s 20,000 Bytes, or 160,000 bits. Add about 10% for SSL overhead, and that gets you 176,000 bits. Take the example of a T1 line, at 1.5 Megabits per second. Divide 1,500,000 bits by 176,000 bits, and you get about 8.5. So in a T1 line, the maximum number of 8.5 TPS in a T1 line without over-subscribing the line.

• T1: 8.5 TPS
• 10 Mbps: 56.8 TPS
• DS3 (45 Mbps): 255.7
• 100 Mbps/Fast Ethernet: 568.1
• OC3 (155 Mbps): 880.7
• Gigabit Ethernet (1,00Mbps): 5681.8

So, as you can see, to get to really insane levels of TPS you either have to have a huge pipe, or you have to have insanely small pages.