Typically, Apache Bench (ab) is installed with the base Apache install, from at least Apache 1.3 on.  This includes when Apache is installed on Windows.

You can check all of the available options on the ab documentation page, but here’s a (very) quick reference to using it.

Two of the most important options are “-n” for the number of total connections, and “-c” for how many concurrent connections are done at the same time.

For instance, using the option “-n 1000″ will do 1,000 requests, one at a time, to a target URL.

ab -n 1000 http://website.com/

One at a time is rarely an effective test, so it’s best to use the “-c” option to specify a high number of concurrent connections, such as 100.

ab -n 1000 -c 100 http://website.com/

If you use concurrency, ab will split the total number of requests up amongst the concurrent settings.  For instance, using the option “-n 1000″ will do 1,000 connections, but “-n 2000 -c 100″ will only do 20 requests from 100 different connections (2,000 / 100 = 20).  So it’s best to use a much larger number of total connections if you’re doing concurrency.

ab -n 100000 -c 100 http://website.com/

When ab is finished running, it will spit out a performance report, including such info as the time taken for tests, requests per second, wait time, etc.

Finished 1000 requests

Server Software:        Apache/2.2.9
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        45 bytes

Concurrency Level:      10
Time taken for tests:   0.427 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      320640 bytes
HTML transferred:       45090 bytes
Requests per second:    2341.45 [#/sec] (mean)
Time per request:       4.271 [ms] (mean)
Time per request:       0.427 [ms] (mean, across all concurrent requests)
Transfer rate:          733.17 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   0.4      2       3
Processing:     0    2   0.5      2       7
Waiting:        0    2   0.5      2       6
Total:          0    4   0.7      4       8

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      5
  80%      5
  90%      5
  95%      5
  98%      5
  99%      5
 100%      8 (longest request)

The ab utility defaults to one request per TCP connection (KeepAlive turned off).  If you want to use KeepAlive, where multiple requests are made through a TCP connection, use the “-K” option, open up as many TCP connections as you specify in concurrency (“-c”) and make the total number of quests through those few open TCP connections.

The utility is a simple but power tool for testing load balancers and web servers.  It doesn’t tend to reflect real-world usage, but it can be useful for baseline testing and troubleshooting.  I’ve found it quite useful over the years.

I was on a run in my new home of Portland, Oregon when I came upon an AMC Pacer parked on the street.  And I thought, how like the old Alteons the AMC Pacer is.  They’re both rather odd looking, and have a rather fanatical fan following (as evidenced in it being in a starring role in the Wayne’s World movies).

There was a post on the lb-l mailing list recently about them, and it’s interesting to see how the old Alteons are still going strong in the market.

Recently, this came up on the load balancing mailing list.  An Alteon was setup with EtherChannel, but the traffic only went over one link.   As strange as it seems at first, this is actually the way EtherChannel works in his environment, and technically it’s working correctly.  To understand why, you need to understand EtherChannel.

I could go into a long treatise on the various components of EtherChannel, but Wikipedia has it pretty well covered, so instead, I came up with the 3 Things You Need To Know About Etherchannel. Know these 3 things, and you can Google the rest.

#1: Etherchannel may not always balance load evenly (or at all) across multiple links.

Simply turning on EtherChannel does not necessarily create even load distribution across multiple bonded links.   You may get nearly perfectly distributed load, or you may get all your traffic running over a single link.  There are tweaks you can do to resolve uneven distribution depending on the situation, but there are also situations where load distribution isn’t possible at all.

#2: How Etherchannel divvies up the traffic

EtherChannel does not work by round-robin, but instead works by hashing a characteristic of an Ethernet frame. The hash determines which of the links the frame travels to in order to get to the other side.

The most basic characteristic to hash on is the source or the destination MAC address. Most devices that are capable of EtherChannel support one of those two modes (or both).

In addition, some advanced switches can hash based on source or destination IP address, and even source or destination IP address and port configuration. These are typically more advanced Layer 2/3 combo devices.  Someone from the load balancing mailing list posted a link to this article on Cisco’s site which explains which modes are available on their line of switches.

As you can probably guess at this point, that a lack of diversity of whatever characteristic you use can cause uneven load distribution.  If you’re doing the hash on the MAC address, but only one MAC address is used to send traffic to only one other MAC address, that traffic will always go over the same link.  The other links in the EtherChannel group won’t get used at all.

In fact, if you have a single MAC address, IP address, and TCP port sending traffic to a single MAC address, IP address, and TCP port (such as an FTP transfer from one host to another) load distribution is actually impossible.  Everything hashes out the same link.

Most likely, however, one of the methods (source or destination on the MAC, IP, or IP/port) will typically get you pretty good load distribution.  If it’s available on your device, that is.  Again, many devices only support source/desination MAC addresses.

#3: The sending side alone decides which link the traffic goes over

When a device determines that a frame is to go over an EtherChannel link to reach its destination, the device sending the frame is the one that chooses which of the available individual links the frame goes down. The receiving end has absolutely no say in the matter.

As a result, to obtain proper load distribution, you must configure each end individually for the best hashing characteristic available.  One device may support only MAC address hashing, while the other device may support MAC and IP.  You can mix and match.  The other ends don’t care how the frames get to them.

If the virtual server is setup as “transparent”, where the client IP address is preserved (i.e., the load balancer doesn’t act like proxy), then the clients cannot respond correctly.

The load balancer performs network address translation on the way in, and on the way out. This is necessary for the connection to work correctly. However, if the client is on the same subnet as the real server, the server will respond to the client directly, without hitting the load balancer on the way out. The source IP address will appear to be the server, and not the virtual service. Because the client sent the connection to the virtual service, it will drop all incoming packets from the server.

The client can be on the same subnet as the virtual service and browsing will work, provided that the virtual service is on a separate subnet from the real servers.

There are a few solutions for this:

The first is to disable transparency, and have the load balancer act like a proxy. The downside to this is that you lose the source IP address for your server logs (all connections appear to be coming from the load balancer).

Another solution is to use DSR (Direct Server Return). The downside to this is that you cannot do cookie persistence, or any other fancy Layer 7 features.

Depending on your situation, a compromise might be to setup two virtual IPs, each with identical real servers sitting behind them.  One would be for the public, which would preserve the IP address, and the other would be for client systems that sit on the same subnet, and that one would be set for non-transparent/proxy mode.

* Another SLB vendor has hit my radar, a company called A10 Networks. They have a Layer 4-7 load balancer from the looks of it, but I’m not sure of what their price point is.

* Also, I’ve completed my Buyer’s Guide, and it looks like it’s going to get published in a major online publication. I’ll of course post links when that is complete.

* An, an SLB Wiki is nearing completion. That should be a good resource for users out there.

Steve Lothspeich was kind enough to post a fix for the DST problem.

Recently member Cihan Subasi posted an interesting question on the state of Firewall Load Balancing. The poster asked if people used external load balancing appliances, or if they used the firewall vendor’s solution.

Several people responded, and among the opinions that I share is that FWLB isn’t all that common anymore, nor is it needed.

There was a time when most firewalls were basically PCs. Even Cisco’s PIX ran on Intel/AMD chips, and they all suffered the same performance problems I mentioned in “Parsing for Precision“. They just couldn’t keep up with the traffic levels that were being imposed upon them.

Firewall Load Balancers (FWLB) came in as a solution to solve these issues. They were typically web switches (ASIC-based load balancers) from the likes of Alteon and Foundry Networks, and would allow scaling of firewall capabilities by simply adding more firewalls.

While they worked on paper, they were notoriously complicated setups, and the wiring alone was enough to make one’s head spin. So instead, many sites just split up traffic among smaller load balancers. This was architecturally inconvenient, but it was still preferable by many to FWLB.

And as time went on, two things happened: ASIC-based firewalls with the ability to do complex rules at wire-speed appeared (NetScreen, for instance), and more gradually, x86 processor speed increased to be able to handler ever increasing traffic loads, and this was supplemented by encryption acceleration/offloading hardware. One vendor in particular, Stonesoft (disclosure: I once worked for them) had a clustering component built-in; it was its own load balancer.

So, even now, FWLB isn’t all that common. Firewalls on modest hardware can handle tremendous amounts of traffic, and for the really high traffic levels, other solutions exist (ASIC, self-clustering, etc.) to handle scalability issues that are generally simpler.

That doesn’t mean it’s not in use, nor does it mean it’s not a good solution for a given infrastructure, but it’s just not common.