Systems supporting only snort rules and lacking a positive security model are not usually not considered a web application firewall, but rather an intrusion prevention system.

Which is an interesting question: What is a Web Application Firewall exactly? How does that differ from an IPS (Intrusion Prevention System), and is there any meaningful distinction between the two?

Amazingly, Wikipedia doesn’t have an exact definition for Web Application Firewall (just Application Firewalls, but that’s something a bit different).  But here’s the definition of “Web Application Firewall” according to OWASP:

A web application firewall (WAF) is an appliance, server plugin, or filter that applies a set of rules to an HTTP conversation. Generally, these rules cover common attacks such as Cross-site Scripting (XSS) and SQL Injection. By customizing the rules to your application, many attacks can be identified and blocked.

I think the KEMP box qualifies, but the range of devices that qualify are vast.  While the KEMP box has WAF capabilities, it’s clearly an entry level box ($2,500) while the Breach box is higher end.  It’s like the difference between the KEMP LoadMaster and F5′s LTM/BIG-IP: They’re both Application Delivery Controllers, but they’re clearly not in the same league or market.  And that’s fine, if all we had were high-end load balancers, the SMB would be squeezed out.  I’ve not tested a Breach box, but I wouldn’t doubt that they offer superior protection.  It’s just a matter of whether the customer can afford it.  The world needs BMWs as well as Camrys.

What do you think?

The driving force behind Layer 7 persistence (keeping an individual user tied to a specific server in a server group based on HTTP headers instead of IP address) was the dreaded AOL Megaproxy issue.  AOL had the nasty little tendancy of routing all web traffic through a couple of mega proxies located throughout the US and Canada.

This caused a problem with the previous method of persistence, which was to base it on source IP address. Typically, one IP address equaled a single user.  However, with AOL, you could have 20,000 users coming from a single IP address.  The load balancer would think it’s a single user, and if you had 300 servers ready to take orders, all 20,000 users would go to one.  That situation has happened a few times, and it’s hillarious, so long as you aren’t the company with the 300 servers.

I still teach that mega proxy problem, mostly out of muscle memory.  But I stopped to think about it, do we really have a problem with megaproxies anymore?  Does AOL even do this practice, and even if they did, is AOL represent a significant amount of traffic?

The answer to the later question is almost certainly no.  AOL has seen a dramatic drop in subscribers, and most people connect directly to the Internet through their cable modem or DSL provider.  And I don’t know of any major Internet provider that utilizes proxies for their users Internet requests.

Layer 7 persistence is still applicable to situations where you may have multiple users coming from a single IP address (such as a small client base coming from a handful of offices, with each office using on public IP address), but I wonder what doing Layer 4 persistence would do to a major site these days.  I’m thinking, not much.

What do you think?

So here’s your chance to contribute to the questions:  What do you want to know?  Put your suggestions in the comments section below, or email them to tony at lbdigest dawt com.

Cisco CSS users specifically have an interesting choice when it comes to choosing a new product: They can take the high road, or they can take the low road.

Feature-wise, the Cisco CSS is roughly in parity with much of the value market (KEMP, Barracuda, Coyote Point). If a site is looking to keep the same level of functionality along with saving a ton of cash, then the value market may be the way to go.

More advanced load balancers, such as F5′s BIG-IP, A10 networks, and of course, Cisco’s ACE platform, offer a lot more features than the CSS. F5 for example offers their iRule platform, which allows the load balancer to do some pretty sophisticated app-level functions (such as authentication) before ever hitting an application.  They all offer a more flexible network implementation, with multiple in-bound and out-bound routes.  Of course, the prices are significantly more than the value market products.

The good news is, if you’re looking to replace your Cisco CSS load balancers, there are over a dozen appropriate vendors that can replace your infrastructure while giving you similar functionality.  The bad news is, there is over a dozen vendors to choose from.  So don’t be afraid to get picky, and start looking into either saving a lot of money, or getting a lot more features.

What do you do?

I’ve been in that situation so many times, I’ve developed a relatively quick checklist process that can quickly be performed. This check list has a couple of benefits:

• It’s methodical and process-based, so it can pick up both the obvious and the oddity
• When working in an environment where there are different groups responsible for different aspects of the infrastructure, this provides clear demarcation for them and helps with interaction
• If the problem lies with the load balancer, this troubleshooting will point to the problem in about 90% of the cases
• If the problem lies elsewhere, this troubleshooting will provide hard evidence to back up that claim

The heart of this check list is the 4-step process basic to all load balancing:

The 4-step process, basic to all load balancing

This process starts at the beginning, from the clients perspective, and moves through the entire connection from end-to-end, testing to make sure everything is hunky dory along the way.

• Make sure the load balancer sees the connection
• Determine how the load balancer handles incoming connections (Layer 4 or Layer 7)
• Check connectivity from the load balancer to the server

Going through the list, if you find a problem, you resolve the problem before you continue on. There may be other problems, but you’ll need to address the first problem you encounter first before moving on, otherwise there will be too many variables.

This check list is particularly useful in situations where you don’t have access to all of the equipment on the network, such as large enterprise situations where separate groups are responsible for areas like firewalls, network routing, switch infrastructure, and servers.

Prepping for the Check List

The two tools you’ll need to run through this checklist are telnet, openssl, and tcpdump (or some other network sniffer). It’s best if you use use tcpdump on the load balancer itself (which is included in most load balancers), but if that’s not possible, setup a network tap of some sort. For this checklist, we’ll assume you’re using a load balancer with tcpdump.

Step #1: Confirm that the connection is reaching the load balancer

Step 1 is to simply ensure that connections are going to where they are supposed to go. While this is obvious if you’re in a situation where the connection times out, this also works when you get a definate reaction (connection accepted/connection refused), this at least proves that the load balancer is the one sending the response, and not some other device.

In this test, we’re only concerned with whether the connection is reaching the load balancer, step 1 in the diagram above. To do this, run TCP dump on the load balancer with the following attributes:

tcpdump -i [interface] -n host [ip address of virtual] and port [port of virtual]

Then telnet to the IP and port of the virtual service on the load balancer. If you’re doing SSL termination at the load balancer, use telnet anyway, as we’re just testing for a valid TCP connection. It’s best that you do this from a subnet that is not the virtual service, so as to eliminate routing issues.

You can try ping, but it really doesn’t tell us anything. For one, ICMP is not the protocol we’re concerned with. Firewall rules also may block ICMP and not TCP, or it may block TCP and not ICMP. Either way, telnet works much better because on a TCP level and mimics a connection from a browser.

Typically one of three things will happen:

• Connection refused
• Nothing connects, and the operation times out
• A connection is made

What we’re looking for is to see if the load balancer sees the attempted incoming connection. If the load balancer doesn’t see the incoming connection, it may be a routing issue (either Layer 3 or even Layer 2) or it may be that a firewall rule is blocking the connection.

In any event, if you’re not seeing the connection, stop at this step, and figure out why. If you’re dealing with different networking groups, you can bring this tcpdump information to them and they’ll have something substantive to go on.

If you do see the incoming connection, move on to the next step.

Step #2: How Is The Load Balancer Handling The Connection?

As mentioned in a previous blog entry, load balancers exhibit different behavior depending on whether or not the virtual service is configured for Layer 4 or Layer 7. A layer 4-configured virtual service will not complete a TCP connection unless a connection all the way through to a real server. In a Layer 7-configured virtual service, as long as you can reach the IP and port of the load balancer, you’ll probably get an established TCP connection (although some load balancers allow you to change this behavior).

Most load balancers don’t tell you explicitly whether or not you’re running in Layer 4 or Layer 7 mode. They switch between one or the other automatically depending on how you configure the virtual service. Only one load balancer that I know of tells you (KEMP Technologies). With others however, you’re left to pretty much deduce this on your own.

So how do you tell? It depends on the vendor, but generally if you’re using any type of cookie persistence, SSL termination, content rules, or programming language on the load balancer, you’re in Layer 7. In F5′s BIG-IP V9, when you set up a virtual server, there are a few options on the type of virtual server to setup.

Standard and Performance HTTP are Layer 7 configurations, while the others are Layer 4-limited.

So what happens when you try to connect to a Layer 7 virtual service that has connectivity problems with real servers on the back end?

First, the connection will be accepted:

system1> telnet 192.168.0.200 80

Trying 192.168.0.200...

Connected to testvip (192.168.0.200).

Escape character is '^]'.

A valid TCP connection has been established. If you’re troubleshooting and you get this, you may assume that the device you’ve connected to is the server. But this is not the case. You never directly connect to the server when the load balancer operates in Layer 7. In this example, there are no real servers that are on line. The BIG-IP shows all available servers as unavailable. Yet I was still able to make a connection.

Now I’ll do a simple “GET /”. What happens with this “GET /” depends on the vendor, and even on the version. Take for the example BIG-IP Version 4 and BIG-IP Version 9.

With version 9, this happens:

GET /

As soon as I hit <Enter>, the connection is closed by the BIG-IP sending a reset packet.

Connection closed by foreign host.

13:37:36.679409 192.168.0.200:80 > 192.168.2.2.33962: R 1:1(0) ack 7 win 4387 (DF)

With BIG-IP version 4, there is a different behavior. It will hang out for a while, before sending the reset. This can make you think that the web server is hanging, but again, what is happening is that the server.

Step #3: Connectivity From The Load Balancer To The Server

First off, perform some sort of test to see if the real servers are even operational. Open up a browser and plug the IP address (and port) and see if you can bring up a site. If that doesn’t work, or if the server is a non-HTTP protocol, use telnet to see if you can get a TCP connection. If you can’t, you may want to figure out why. If the servers aren’t responding, you’re obviously not going to get far with a load balancer.

If you can get to the servers, log onto the load balancer and telnet from the load balancer to the real server on the port configured. Try connecting to at least one of the servers in a multi-server group.

> telnet 10.0.0.100 80

Again, one of three things will likely happen:

• You’ll get a valid TCP connection. If this occurs, try to make an HTTP request. A simple “GET /” and <enter><enter> (hit enter twice) should suffice to get some sort of response. As long as you get some sort of response, that’s good.
• You’ll get a connection refused. If you’re getting connection refused, it’s either because a firewall is blocking you, or the server isn’t answered.
• Your connection will time out. For whatever reason, packets aren’t getting to the server. This can either be a firewalling issue, or some other routing issue. If you’ve got access to run tcpdump or other type of network trace on the server, see if you can see the incoming connection from the load balancer.

After running these tests, you should have a much better picture on what’s going on with your network. If the issue was caused by the load balancer, this would probably have spotted the root cause. If it wasn’t the load balancer, then you’ve got evidence to prove that it’s not.

That’s not always the case. Depending on how your load balancer is configured, it could simply mean you’ve made a connection to the load balancer’s proxy server.

Modern load balancers have actually two different types of virtual services: Layer 4, and Layer 7. Clients can’t really tell the difference, but under the hood, they’re substantially different in the way they operate.

A Layer 4 virtual service on a load balancer operates much like a router. It’s just re-writing source and/or destination addresses. Not much more than you’re basic broadband wireless router.

Layer 7 code is a type of application proxy, aware of HTTP and a few other protocols depending upon the vendor (such as FTP, SIPS). The Layer 7 code is what handles cookie persistence. When a connection is made to a Layer 7 virtual service and a request is sent, a separate TCP connection is opened to the server, and the request is forwarded. Because of the way Layer 7 operations occur, the load balancer *can’t* send the request to a server until it sees the request, because the request (and items in the header) will tell the load balancer where it’s sending the request.

The tricky part comes when there’s problem with connectivity between the load balancer and the servers. If the servers are unresponsive, you will still get a connection to the virtual service. This can trick you into thinking there’s something wrong with the servers, when the issue may be elsewhere.

Once this connection is made, what happens next is fairly vendor specific. Once you’re connected and make a request, the connection can remain open and stall. On other vendors, the connection will terminate the second you make a request, with no HTTP error code.

Even within the same vendor, different versions will react differently. In BIG-IP V4, you can open a connection and make a request, and you’ll see no response.  Eventually the connection will terminate, but you will probably wait a while.  In BIG-IP V9 however, when you open a connection and send a request, you’re immediately sent a TCP reset packet.

If you’ve setup SSL termination, which would fall into the Layer 7 camp, you’ll get non-HTTP encoded error (such as read:errno=104, which won’t show up in the browser, but will show up in a raw TCP connection) when connecting and sending a request to a virtual service with no active real servers.

So it’s important to know how your virtual service is configured when you’re troubleshooting an issue.

• TCPDump
• HTTP Header dump (such as Live HTTP Headers)
• Telnet

Yup, telnet. I don’t know about anyone else, but I use telnet excessively. Basically, I use it to test TCP connectivity, and with a quick “GET /”, to test to see if the web server is responding.

Most modern telnet implementations, including the one on F5′s BIG-IP version 9, include the ability to choose your source IP address (-b [IP address]). BIG-IP version 4, however, doesn’t seem to have this ability.

This was frustrating when trying to test some firewall and routing issues with the BIG-IP’s SNAT address. One issue that can come up is when you setup health checking, everything works out, because the IPs are generally that of the BIG-IP’s self IPs. But because the SNAT is a different IP, routing or firewall issues may crop up and block the connection.
So I wrote a little Perl script that allows me to test connections with various source IP addresses. It makes a simple TCP connection and reports whether it’s successful or not, while allowing me to specify the source IP address. It’s very utilitarian, without any Jobsian niceties, so I may punch it up some more.

#!/usr/bin/perl

use IO::Socket;

$num = $#ARGV + 1;

if ($num != 3)
{
    die "Usage: ptelnet [source IP] [destination IP] [destination port]";
}
$localIP = $ARGV[0];
$destIP = $ARGV[1];
$destport = $ARGV[2];

$remote = IO::Socket::INET->new(
    Proto => "tcp",
    LocalAddr => "$localIP",
    PeerAddr => "$destIP",
    PeerPort => "$destport",
    Reuse => 1
    )
    or die "Can't connect to port $destport at $destIP from $localIP";
print "Connection successful to ", $remote->peerhost, " on port:
", $remote->peerport, " from ", $remote->sockhost, "n";
close($remote);

But what you don’t know much about is load balancers. So here’s a bit of a primer on load balancing for those involved with application development.

Persistence

The first thing you need to know about is persistence, and specifically, do you require it. If your application is stateful, where information regarding a session is stored on only one server, you’ll need persistence. Virtually all load balancers support this, but you’ll need to know to turn it on (or ask your load balancer administrator to turn it on).

Most applications are stateful, so it’s a fair bet you will. A quick way to test is to start a session on one server, then change the hostname or IP address in your browser to point to another server with the same application installed. Does it break, act freaky, or otherwise malfunction? Then you’ll need persistence.

As I’ve said several times before, you’ll probably want cookie persistence.

What The Load Balancer Passes On To The Server

Sometimes I’m asked what the load balancer changes in the client request to the server.

The answer is: absolutely nothing.

Load balancers will pass all HTTP headers that it receives onto the server. It may add a few items, such as a persistence cookie, but in most configurations, the load balancer won’t change anything (and with many vendors, the load balancer just doesn’t have the ability to change anything).

A load balancer might give out an HTTP 302 redirect. A very common example is redirecting from HTTP to HTTPS.

Virtual Hosting: The Host Header

This is often called software virtual hosting, virtual hosting, etc. Basically, it’s running more than one URL off the same IP address.

Let’s say you’ve got two URLs: www.domain1.com and www.domain2.com. In DNS, they both point to the same IP address, yet when you go to the sites with a browser, two separate web pages come up. How come? It’s all in the HTTP host header.

When the browser makes a request, it includes a “Host:” entry, telling the web server what host it’s looking for. The web server looks at this host, and serves up the appropriate page.

The load balancer will forward this host request along with the entire request. Most load balancers don’t have the ability to even change this.

SSL Termination

If you utilize SSL on your web site, you may want to consider having the load balancer terminate the SSL connection.

You’ll want to check to see if the load balancer has hardware acceleration, which is a special card that removes the SSL encryption/decryption operations from the general CPU and onto a specialized processor.

The two main benefits to SSL termination are the performance benefit by having the load balancer handle the SSL instead of your servers, and by terminating the SSL connection on the load balancer, you can use cookie persistence.

Header Dump

It helps to have a method, in either a standalone page or in your own library as a quick function call, to dump all the HTTP header variables.

In PHP, you can use the built-in phpinfo() function.

<?php
phpinfo();
?>

Have this page/function handy, in case a problem arises.  Point your load balancer administrator there, and they may be able to point out the problem.

I read somewhere that the load balancer you run must support sessions, otherwise you must use cookies.”

This is a common type of question that people who are on the application development side of the fence have regarding load balancers.

With regard to PHP sessions, this typically means that the application is “stateful”, that is, there is information regarding a particular session (such as shopping cart contents, login credentials, etc.) that are stored only on an individual server, and not shared amongst a group of servers.

Most modern applications these days are stateful, whether it’s on PHP, or Websphere, or just about any other platform.

For stateful applications to work with load balancers, the load balancers need to do persistence, which keeps a specific user tied to a single server, even though there may be several or even dozens of other servers available. And the best persistence for web serving is cookie persistence.

It’s very simple to setup, and usually involves a simple check box in the load balancer’s configuration. Cookie persistence (specifically, active-cookie) inserts a cookie into an HTTP header which it then uses to identify which server subsequent requests should go to. It requires not additional configuration on the application, and the inserted cookie will not interfere with any existing cookie.

So the answer is simple: With a PHP application that uses sessions, you’ll need to configure persistence, you’d be wise to select a load balancer that supports cookie persistence.

So, here’s a few tips to help whittle down the vendors.

The first thing to consider is what type of company you are. Are you a huge mega-corporation? A financial? A scrappy upstart? That by itself will dramatically reduce the number of viable vendors. If you’re a Fortune 500, you’re going to want to go with one of the established premium market players. If you’re a scrappy upstart, your budget is probably very limited, so you’ll want a value vendor.

Next, consider the focus of the companies you’re evaluating. Is their primary focus load balancing? Is their core competency networking in general, with a tiny subset dealing with load balancing? There are companies that are intently focused on load balancing, and companies that are generalists with networking products. F5 and Cisco are great examples of each. F5 is intently focused on load balancing, and generally have the best in terms of features and technology. Cisco hasn’t been as focused, and seem to always be playing catch-up in terms of features, but they do have a widely known, highly regarded reputation. Just about everyone has heard of Cisco, few outside of IT have heard of F5. I tend to prefer the better technology, but both aspects have merit, and which has precedence depends highly upon your corporate directives and culture.

Of course, there’s also the new versus used question, but I think you know my answer on that one: Always buy new. Not because of the coveted “new load balancer smell” (although it is delightful), but because new boxes are supported by the vendor in terms of software updates and hardware failures, and used typically aren’t.

The rest of the selection process depends highly upon whether you’re going for the value market or you’re going for the premium market. I will be updating shortly with the process I recommend.