Load Balancing Digest

The ability for a load balancer to peer into (and potentially manipulate) the HTTP headers of incoming connections was once an advanced feature, but now is fairly commonplace.  Most often it’s used in cookie -based persistence, but it’s also used in web switching, true-source IP resolution, and other tasks.

But the ability to look at the HTTP headers doesn’t always work the way you might think it would.  Often, the load balancer can have a short attention span.

In a traditional HTTP 1.1 connection, multiple HTTP requests are sent through a single TCP connection. Most load balancers by default will only look at the first HTTP request, and ignore the rest.  To elaborate on this, let’s take a look at two of the basic concepts of HTTP.

HTTP Basics

The HTTP protocol can be broken up into to parts: HTTP requests and HTTP responses.  Both are comprised of two components: HTTP headers and HTTP content.

In both HTTP requests and HTTP responses, there are always HTTP headers.  In an HTTP request, there is sometimes content, such as a form POST, or uploading a file.  In the HTTP response, there is usually content, but there are cases when there is none (such as with an HTTP HEAD request).

And there’s one more important bit to keep in mind with regard to HTTP: Every object has a separate request and a separate response.  That’s every JPG, GIF, Flash file, HTML file, etc.  So a web page with 20 images will invoke 21 different HTTP requests; one for the HTML page itself, and 20 for the objects (such as images) referenced in the HTML file.

With HTTP 1.1, all of those 21 objects in a web page are typically requested in a single TCP stream, rather than 21 individual connections (which would be fairly inefficient).  But this presents a problem for load balancers.

Do load balancers look at the data in the first request out of the 21? Or does the load balancer look at each request individually?

In my recent review of KEMP’s new LoadMaster software, which includes Web Application Firewall capabilities, Ofer Shezaf (from breach.com) had this to say in the comments section:

Systems supporting only snort rules and lacking a positive security model are not usually not considered a web application firewall, but rather an intrusion prevention system.

Which is an interesting question: What is a Web Application Firewall exactly? How does that differ from an IPS (Intrusion Prevention System), and is there any meaningful distinction between the two?

Amazingly, Wikipedia doesn’t have an exact definition for Web Application Firewall (just Application Firewalls, but that’s something a bit different).  But here’s the definition of “Web Application Firewall” according to OWASP:

A web application firewall (WAF) is an appliance, server plugin, or filter that applies a set of rules to an HTTP conversation. Generally, these rules cover common attacks such as Cross-site Scripting (XSS) and SQL Injection. By customizing the rules to your application, many attacks can be identified and blocked.

I think the KEMP box qualifies, but the range of devices that qualify are vast.  While the KEMP box has WAF capabilities, it’s clearly an entry level box ($2,500) while the Breach box is higher end.  It’s like the difference between the KEMP LoadMaster and F5’s LTM/BIG-IP: They’re both Application Delivery Controllers, but they’re clearly not in the same league or market.  And that’s fine, if all we had were high-end load balancers, the SMB would be squeezed out.  I’ve not tested a Breach box, but I wouldn’t doubt that they offer superior protection.  It’s just a matter of whether the customer can afford it.  The world needs BMWs as well as Camrys.

What do you think?

Greetings from Italy.  I’m here dealing with ASICS of a different kind (as in the shoes), specifically the 23rd ASICS Venice Marathon.  I was in no danger of winning it, but I did complete it in 4 hours, 25 minutes (my 3rd Marathon).

Other than bragging about my nerdrunning abilities, this post is also to see what tools are out there that people use in terms of HTTP analyzers.  The two I commonly use are listed below:

• Firefox: LiveHTTPHeader
• Internet Explorer: ieHTTPHeaders

Leading Romanian Online Service Portal, Neogen, Accelerates Delivery of Web Applications with Crescendo Networks

With 12 websites and more than 760 million page views a month, Neogen relies on Crescendo’s AppBeat™ DC to optimize Web performance

“Before deploying AppBeat DC, our user growth was causing our sites to load very slowly. In addition, we were having redundancy problems whenever a Web server went down,” said Zoltan Farczadi, CTO at Neogen. “Since deploying the AppBeat DC product our page load times have improved dramatically. We’ve gone from a slow/very slow rating in Alexa, to fast/very fast. And more importantly, our customers are experiencing lighting-speed load times for pages and content.”

Neogen is one of Romania’s leading online portals. The company offers searching and matching services for a variety of verticals including employment, home, automotive, social networking and more. Last month, the company received more than 760 million total page views across 12 sites, with an average load time of less than one second for www.Neogen.ro.

“AppBeat DC really allows us to offload some of the processing-intensive tasks from our Web servers, and optimizes and accelerates the delivery of the content to our end-users. When we first tested the solution, the performance was fantastic, and it also fit into our existing architecture, without making any significant modifications. We’ve been very happy with the solution and it’s worked just as we’d expected,” added Farczadi.

Crescendo Networks, the Crescendo Networks Logo and AppBeat are trademarks or registered trademarks of Crescendo Networks in the U.S. and other countries. Other names may be trademarks of their respective owners.

Here’s a question that has come up a few times in the past few months:  Will this “cloud” thing that’s

supposedly on the horizon eat up load balancing?

The cloud is this nebulous platform on which to build your web application. The idea of course is that companies don’t have to maintain/manage their own servers, network infrastucture, datacenters, etc.   And of course, load balancers.

If the cloud computing concept takes off, there’s the potential to really disrupt the sales of load balancers/ADCs.  Instead of 100 companies buying load balancers and utilizing them at 20-50%, a few cloud computing providers could purchase a lot fewer load balancers and push the same amount of aggregrate traffic through consolidation and economies of scale. That all depends on how it plays out, of course.

The value market (SMB-oriented) won’t probably feel this.  If the cloud takes off, it’ll probably be geared towards the enterprise first, and then trickle down to the SMBs.

But I’m not betting on the cloud just yet.  We’ve seen this before with grid computing and companies like Loudcloud.  From what I remember, the high degree of customization most enterprises emply seemed to negate most attempts at cloud computing.

Take American Airlines as an example. Let’s say they wanted to port themselves to the cloud.  They’d have to have the cloud company either completely re-write their application on the cloud platform, or the cloud company would just take on the current platform (likely hardware, OS, and app platform and all).  But that wouldn’t really be cloud computing.  That would be hosting.

Radware was kind enough to send me a headsup regarding their virtual press conference that announces their VirtualDirector product.

You can take a look at it this week at: http://www.radware.com/virtualpressconference.

Apologies for the LOLcatspeak.  I’m incapable of helping myself.

The driving force behind Layer 7 persistence (keeping an individual user tied to a specific server in a server group based on HTTP headers instead of IP address) was the dreaded AOL Megaproxy issue.  AOL had the nasty little tendancy of routing all web traffic through a couple of mega proxies located throughout the US and Canada.

This caused a problem with the previous method of persistence, which was to base it on source IP address. Typically, one IP address equaled a single user.  However, with AOL, you could have 20,000 users coming from a single IP address.  The load balancer would think it’s a single user, and if you had 300 servers ready to take orders, all 20,000 users would go to one.  That situation has happened a few times, and it’s hillarious, so long as you aren’t the company with the 300 servers.

I still teach that mega proxy problem, mostly out of muscle memory.  But I stopped to think about it, do we really have a problem with megaproxies anymore?  Does AOL even do this practice, and even if they did, is AOL represent a significant amount of traffic?

The answer to the later question is almost certainly no.  AOL has seen a dramatic drop in subscribers, and most people connect directly to the Internet through their cable modem or DSL provider.  And I don’t know of any major Internet provider that utilizes proxies for their users Internet requests.

Layer 7 persistence is still applicable to situations where you may have multiple users coming from a single IP address (such as a small client base coming from a handful of offices, with each office using on public IP address), but I wonder what doing Layer 4 persistence would do to a major site these days.  I’m thinking, not much.

What do you think?

Note: Just a quick disclaimer here, I worked for KEMP in 2006.  I’m still Mr. Neutral when it comes to these devices, but I wanted to make sure there was full disclosure.  As with all reviews, I receive no payment whatsoever for reviews.  Only a big suitcase full of $100s would change that.

KEMP LoadMaster 1500

KEMP is a member of the value market vendors, and their LM-1500 is what put them on the map. Released in late 2005, the $2,500 price tag gained it a lot of fans. At the time, it was the only Layer 7 load balancer at that price range (a few more have introduced products at that price).

The LM-1500 has been reviewed before, but they’ve got some new code coming out that introduces some new functionality (including caching and compression), and they were kind enough to send me over a box with the soon-to-be-released code. (If you have a product you’d like to see reviewed on lbdigest.com, feel free to contact me [tony at lb digest dot com]).

Getting Started

The LM-1500 can be initially configured through either the VGA port with a USB keyboard, or through the serial port.  After it’s given an IP address, the rest of the configuration is done through a web interface.  There’s also a way to configure from the start with a web interface, as it comes up with a 192.168.1.100 IP automatically.

Administration

There is a command line interface, but virtually all of what you need to do can be best done through the web interface.  One of the nice touches they’ve added is the “download root certificate”, which installs a trusted CA cert on your browser to get rid of the annoying but ubiquitous self-signed certificate warning.

Creating a Virtual Service (VIP) is pretty straight-forward.  Give it an IP address and port, select your options (L7, cookies, etc.), and so on. There’s also configuration options for a “sorry server” (if non of your regular servers are up, the LM will send traffic to a sorry server, which can have a some sort of “sorry, we’re not working right now page”.

There’s also a handy stats section, reporting on the various performance metrics of the device.  Most of these metrics are also available through SNMP (for PRTG/MRTG, etc.).

App Delivery Features

The LM-1500 does the standard Layer 4 load balancing, as well as the more advanced cookie-based persistence and web switching.  The only drawback is that you cannot do both web switching and cookie-based persistence at the same time.

There are quite a few options available to a virtual service configuration, including transparency (full-NAT or half-NAT), health checking, and a “sorry server”, for when all the servers in your main web app farm goes down.

SSL certificate management works well as well, making it pretty easy to add/remove/change SSL certificates assigned to virtual servers. The LoadMaster automatically installs a self-signed certificate when you turn on SSL termination, and gives you the option to install CA certs and intermediate certs.  The LoadMaster 1500 does not include an RSA SSL ASIC, however (see the box section for more specifics).

Network Architecture

As I’ve said before in other reviews, one of the aspects missing in in a lot of reviews is how the product fits into the network, so I make a point to include some details.

The LM-1500 operates only in Layer-3 route-path mode, it cannot be used as a Layer-2 bridge-path load balancer.  You must either use the LM-1500 as the default gateway, or use it in non-transparent mode.   One of the easiest ways to put an LM-1500 into a network is the one-armed mode, where the Virtual Service IPs (VIPs) and the real servers all sit on the same subnet.

The LM-1500 also works well in two-armed mode, which is typically when you have the Virtual Service IPs on a public subnet and the real servers on a private RFC1918 address space (such as 192.168.x.x).

The Box

The box itself is of solid metal construction, and doesn’t look or feel “cheap”.  The bright gold is unmistakable, and in addition to being prominent in a data center, it would probably keep me safe on my late night training runs.

Stat-wise, the system is powered by a VIA Eden chip, with 512 MB of RAM which is more than sufficient for this class of system.  On-board storage is solid-state flash, so there are no moving parts (other than a fan) and the system boots very quickly.

There are three 10/100 Fast Ethernet interfaces, each operating in routed mode (the LoadMasters don’t do bridged) so each interface would each be on a different subnet.

The system boasts on on-board SSL chip, however the chip only does the symmetric “bulk” encryption algorithm AES.  Most of the heavy lifting in an SSL connection is done on the asymmetric RSA operations when an SSL connection is established.  While the AES bulk encryption would help tremendously in long lasting download-type connections, it’s of little use for rapid-fire of short lived connections, which is what most web connections are.

Caching and Compression

The newest addition to the LM-1500 (and the other LoadMaster models) is the addition of caching and compression as well as web application firewall abilities.

In its initial deployment, there’s not much you can configure with the caching and compression, other than turning it on or off per virtual service.  I was able to verify that cached objects came from the LoadMaster, and not the server.

With the LoadMaster, compression is actually another method of caching.  Instead of the on-the-fly compression that most other products do, objects are cached by the LoadMaster, and compressable objects are compressed once in the cache.  This doesn’t get you the benefits of on-the-fly compression for dynamic web pages, but it’s much easier on the CPU (there’s no compression ASIC on the LM-1500).

For compression, I used the simple apache.gif file that comes with the Apache distribution.  Normally, it’s 2410 bytes.

Content-Type: image/gif
Content-Length: 2410

Turning on compression, and I see the LoadMaster send back a gzip’d image.

Content-Type: image/gif
Content-Encoding: gzip
Content-Length: 1795

That’s a slight decrease, but not really a fair test since there’s not a lot to compress.  So to give a better test, I saved the main page of lbdigest.com, and the HTML file came out to about 38K.  I put it up as a static page on my test server, and accessed it through my browser.

 Content-Length: 38180

Turning on compression, I accessed the page again:

 Content-Encoding: gzip
 Content-Length: 11041

From 38k to 11k, that’s more than a 3 to 1 compression ratio on the main page.  Not too shabby.  If your clients were on a 56k dialup line, or connecting from a PC in say Bali, Indonesia, that 3 to 1 savings could mean a much faster page load.

With caching and compression, your mileage may vary quite a bit, depending on the nature of your users and the nature of your content.  It could help tremendously, or it could end up slowing your site down significantly, so it’s something that’s best to test first (this is true for any product that does caching).

Wish List:

My wish list would be the ability to exclude/include file extensions (no PHP, yes on JPG, etc.) and some better reporting of cache and compression statistics.

IPS/Web Application Firewall

Another new addition to the LoadMaster is the addition of IPS/WAF (Web Application Firewall) capabilities (the only other value market product to have this capability is the Barracuda, which I have not personally tested).  The LoadMaster uses SNORT-compatible rules (you can get a limited set for free at snort.org) in order to catch malicious requests.  For instance, try to go to “/etc/passwd” (http://domain.com/etc/passwd) as the URI, and the connection will be blocked at the LoadMaster, and won’t be forwarded to the server. Reporting is a basic with this first release as well, with blocked requests being reported through SYSLOG.

08-01-2008 10:10:39 Invalid URL '/etc/passwd' - WEB-MISC /etc/passwd

Having a web application firewall is part of the new PCI-DSS recommendations.

Wish List:

The ability to pull automatic updates and the ability to get your subscription through KEMP (rather than finding SNORT rules on your own) would be my wishlist for this feature.

Conclusion

Overall, this is a very solid release.  The new features (caching, compression, WAF) are going to be very handy for the SMB.  They are a bit sparse in their configuration, but work quite well as a first release.  The same code will run on all of the LoadMaster line, with the only difference is that the LM-2500 and up have SSL accelerator cards (for the RSA heavy lifting, not just AES).

Availability of the new release will be in the next few weeks.

I saw this on Slashdot today, where a bunch of hackers developed a tool for stealing session IDs in Gmail.  By default, gmail authentication is encrypted, but the rest of your session is not.  In the requests that you send to gmail is included a session ID cookie, which is in the clear.  With your gmail session cookie, I can put it into my browser, and gmail would think I’m you, without needing to re-authenticate. I could then peruse your craig’s list personal responses.  I’m guessing that would be bad.

So now Gmail will allow you to do all SSL, all the time.  This isn’t just a gmail problem, but one that affects all logged-in sessions.  I’m guessing gmail has a pretty high-end SSL accelerator in operation for this.