I agree, they do more than they’ve done before. From application logic to web application firewalls to VMware integration, modern application delivery controllers do a lot. But they still also load balance.  And that’s what everyone calls them.

A rose by any other name. Pictured: Olfactory stimulation vector

Using the term load balancer saves me the conversation: “What’s an application delivery doo-hickey?”

I still call them load balancers because it serves no purpose to rename them.

Since at least 2006 there’s been an effort to rebrand load balancers as application delivery controllers. Gartner has moved to the new term, as have most of the vendors. Marketing has been heavy to rename them. Some vendors even use the term load balancer as a disparaging term for their competitors.

But here’s the problem: We’ve had at least 5 years of marketing, press releases, and events, and still no one (outside of the vendors and specialists) seems to know what an application delivery controller is. When I teach load balancing classes, very few in the class are even aware of the term.

What network administrators, server administrators, and application developers do know is load balancers. When you say “load balancer”, they universally understand what they do and the benefit they provide. Generally speaking, they have no idea what an ADC is.

I have no problem educating on a new term, I’d even help evangelize the term if it made sense. But it doesn’t. Renaming them ADCs adds nothing substantive to the industry, only confusion and an extra conversation.

If I told you I got a new multi-media climate controlled dynamic geographical device, you’d think I’d be some sort of mad scientist. But no, that’s just another name for a car. Cars today do a lot more than cars 50 years ago did, but they’re still cars.

I understand the though behind the attempt to rename them, but I think it’s a mistake. I don’t mind mistakes, but I think its time to own up to the error and start calling them load balancers again.

Technology is complicated enough, we shouldn’t make it more complicated by adding in terms when none are needed.

I’ll give you a minute because I assume if you were drinking a beverage, it’s now all over your computer screen.

How much damage could someone do? Take Google for example. This means they could set up a fake Gmail-looking server, and collect the username and password of a user. The user might not ever realize that the site was fake, and their passwords were compromised.

So now there are totally legit-looking certificates out there. Your browser, and every other browser in the entire world trusts them.

Remember, SSL gives us two things: Privacy and trust. Privacy comes though symmetric encryption, and trust is done through signed certificate chains.

Trust needs to start somewhere.  With your browser, whether its Firefox, Safari, IE, Chrome, Opera, whatever, they all come with pretty much the same set of root certificate that act as the start of trust.  Essentially, they come trusting several sources.

So what happens if this trust is broken, or if the certificate was issued under false pretenses? There needs to be a way to revoke that trust on a certificate by certificate basis. There’s only two ways to do this: A manually updated CRLs (certificate revocation list), or through the OCSP protocol.

CRLs aren’t a terribly good way to handle it. A CRL is simply of list of certificates that would validate the chain of trust through the regular way (Picard -> LaForge -> Ensign Tony), but aren’t trusted anymore. Each of your browsers have their own CRLs, and they can be updated by an OS or browser patch. This relies on you or your organization reliably updating software and/or OS, which doesn’t always happen in a timely manner. Even if it’s timely, there’s always a period of time between when the certificate is revoked and when you get that revoked certificate added to your CRL.  This could be hours, days, or even weeks where your browser would trust an otherwise bogus certificate.  This just doesn’t scale.

OCSP is a better approach, as it can check with a certificate authority every time it hits a website with an SSL certificate. So not only does the browser do the usually trust check, it double checks by checking with the source (an OCSP server hosted by the certificate authority) that the trust is still valid.

If my browser ran OCSP, I don’t have to worry that I might miss a revoked certificate because I haven’t updated my browser or OS.  It can also check every time, so if a cert has been revoked, my browser finds out right away.

So while OCSP is a better approach to CRLs, it isn’t used universally. And it isn’t “fail closed” by default on some browsers, as shown in this table from David Holmes at F5.

So right now, OCSP and CRLs are no guarantee that you can trust a certificate unless you use Firefox or Chrome.

Wait, what? No guarantee? Shit.

This is what happens when people don’t trust each other

There are some opinions that certificates should be handled differently than they are now. He discusses some interesting ideas, but I think it’s fair to say PKI (public-key infrastructure) needs a bit of overhaul.

The best known product for traffic manipulation is likely F5′s legendary iRules, but other vendors have similar capability such as A10′s aFlex.   Essentially, this puts an application development platform. Typically this is done with a standard programming language, such as a modified TCL for iRules and aFlex.

Some vendors, (Cisco I’m looking at you, as well as Brocade but I’m less familiar with them lately) lack the ability to manipulate traffic using application logic.

For vendors, this is a great feature to have. Its attractive to potential customers, and it makes it difficult to move to a platform that doesn’t have this feature. I call it the Charlton Heston feature, since you’ll only pull it out of a client’s cold dead hands.  Once you use it, you’re fairly dependent on it.

Get your paws off me, you damned dirty network admins!

In general, I’m a fan of iRules and their ilk. There’s just too many situations where the ability to manipulate HTTP content has saved the day.

So what kind of manipulation can you do?  There are rules to scrub credit card numbers, so if a web application tries to display a credit card number such as “5123-1234-1234-1234″ (a big no-no according to PCI-DSS), the load balancer would do a regular expression search for that pattern, and replace it with “XXX-XXX-XXXX-1234″ before sending the response off to the client.

With a programming language and regular expressions, just about anything is possible. And there in lies a problem.

Two Edged Sword

Having application logic on the load balancer is a double-edged sword.  One the one hand, it allows you to have very granular control over headers and content for requests and responses.  The drawback is it allows you to have very granular control over headers and content for requests and responses.

The awesomeness of this manipulation is evident, but there are some caveats.

We’ll fix it in post

There’s a running joke in the film industry called “we’ll fix it in post”.  Essentially, it means who cares if we get it right here, we’ll just fix it with special effects or editing.  Except it hardly ever works.  You’ve got the same hazard in something like iRules; it can be a lousy and lazy way to fix to a problem that really should be fixed in the code.

Capacity

Of course, all this regular expressioning and application logic come at a cost in terms of CPU.  The more of it you do, the lower capacity you’ll have.  A load balancer capable of serving up 4 Gbits of second of traditional Layer 4-7 traffic may have its capacity dropped to 500 Mbps.  And there’s no way of telling what the new performance ceiling would be (until you hit it like Wile E Coyote on a rocket sled).

This is less of an issue than it used to be, as Moore’s law has made processors faster and faster, allowing load balancers to handle increasingly difficult tasks, while bandwidth requirements have not increased nearly as dramatically. Not many organizations have more than a couple hundred megabits per second to the Internet, while load balancers are capable of handling several (even dozens) of Gigabits.

Latency

Latency is a dirty word in networks and applications, and certainly a potential issue with application logic.  If you’re going to look at a cookie, the latency imposed by a load balancer will be minimal.  If you’re going to calculate Pi to the 100th digit on every HTTP request, that’s going to add a certain amount of latency to the transaction.  Like the capacity ceiling, it’s very difficult to predict what that additional latency will be when adding more and more logic. And depending on the load profile, that latency may vary quite a bit over time.

Creep

Capacity and latency are easy enough issues to deal with, but creep is very problematic.  It’s a “Layer 8″ problem, and one that can quickly spiral out of control.

Typically the creep comes into play when you first implement application logic, and it fixes a vexing problem. You’re the hero, and someone perks their ears up and says “wow, what else can you do with it?”

“Anything”, you say confidently.  And anything is what they throw at you. Add semi-colons at the end of paragraphs, replace all double-spaces after a period with single spaces, etc.

It can very easily spiral out of control, so you need to know when to put your foot down (hint: early).

Skills

To develop this application logic, you need to understand HTTP really well. When developing most web applications on a platform like ASP or PHP, you don’t really need to understand HTTP all that well.  But the load balancer manipulates on the HTTP level, so you’ll really need to be up on your HTTP. You also need to have the programming skills to pay the bills. Especially for a network admin, those programming muscles might not get flexed on a regular basis.

I’ve also heard the argument that iRules and their ilk are only there to fix problems that should have been fixed in the code. And there’s a lot of truth in that; they are typically used in situations where the problem could also be solved at the application.

But that’s also like saying that I shouldn’t need a lock on my door, because people should not be jerks and go around stealing stuff.

In IT, there are always going to be situations where its either easier to fix the problem on the load balancer, or it’s the only viable solution (code locked down, client doesn’t control the code as its a third party, developer ran away to Brazil with their secret family and won’t return calls).

Bottom line? I’d rather have it than not have it, but I’m very careful with it. After all, with great power comes great potential for epic fail.

I’m a big fan of virtualization.  There’s a lot to like about it, including consolidation (getting rid of space-heater servers that do nothing running 1% CPU and sucking up electricity and throwing off heat), flexibility, and management.  I’ve even gone and got my VCP4 (VMware Ceritified Professional 4) certification.  (Hear that ladies? I’m certified.)

One aspect of virtualization I’m a fan of is the appliances.  Vendors are taking physical appliances (such as a Vyatta router) and turning it into a VM appliance.  I don’t have to worry about an underlying operating system (and the requisite patches), the appliance vendor handles the software and the OS.

Several load balancing vendors have gotten into that virtualization game.  Vendors that have traditionally offered hardware appliances now have virtual appliances (some for years).  From From F5 to KEMP, from Coyote Point to loadbalancer.org, there are a number of virtual load balancers/ADCs to choose from.   And for the most part, they offer the same features as their hardware brethren.

Their throughput and performance is hampered somewhat by the fact that they’re all software and no silicon. Everything is done in the x86 virtualized CPU(s).  Still, depending on how you provision them, they can generally handle several thousand HTTP requests per second even in Layer 7 mode.

The one caveat to virtual load balancers is that their SSL performance is severely limited.  Even value-market load balancers that do most of their functions in a general purpose CPU will still use SSL ASICs for the asymmetric crypto (even using the general processor for the less CPU-intensive symmetric crypto).

The first part of every new SSL connection is a very CPU-intensive asymmetric operation (about 1000x more CPU intensive than symmetric operations).    CPUs that can normally handle tens of thousand of regular TCP connections per second can only handle a few thousand SSL connections at the most.

It is technically possible to do hardware SSL acceleration on a VM load balancers however.  It requires that the virtual machine host (like VMWare ESXi) have an SSL card installed, and VM Passthrough enabled (where the virtual machine can have direct access to physical hardware).

Unfortunately, these SSL cards are tough to come by.  Cavium is probably the most notable vendor, but cards from them aren’t exactly easy to come by, and they’re fairly expensive.  You may not have the option if you’re using blade systems.   And if you you want to leverage features like HA and DRS (using ESX hosts in a cluster), then every machine in the cluster would need to have such a card.  I’m not aware of any virtual load balancer vendor that even supports this configuration.

There are a lot of situations where virtual load balancers make a lot of sense, but keep in mind that the SSL performance capability is going to be fairly constrained.

TALES OF LOAD BALANCING HORRORS!

For tales of persistent terror, challenging your keep-alive, I give you the following vignettes. (Names have been changed to protect my ass, as well as to punch up some bone dry material.)

The Default Gateway To The Abandoned Zone

Several years ago, on a dark and stormy night, a dashingly handsome young hot-shot system administrator had just finished setting up a new web infrastructure for a client.  They were moving their web infrastructure from their own facility, to the co-location facility where Mr. Sysadmin worked.  Mr Sysadmin was also responsible for the load balancer.  Running bravely into the load balancing realm while both the other sysadmins and the network admins dared not tread.

“It’s cursed!” They said.  Or maybe they used curse words to describe it.  It was a long time ago.

He powered up the system, tested the traffic, and cried out into the night “It’s alive! It’s pushing traffic!”  Overly pleased with his unnatural creation, he emailed the client to tell them their configuration was ready.  They moved in, with administrator access to all systems.

At first, the infrastructure worked as promised.  Sites were served, and loads were balanced.  Then, a call came from beyond the datacenter.

“The load balancer is screwing up. The site is down.” said the customer.  From beyond the data center.

Our hero was not convinced.  Many plagues have been blamed upon the load balancer, only to later find out the culprit was elsewhere.  So he punched up the website, and sure enough, nothing. He logged into the load balancer, and found it to be operating correctly, with no changes from when it was working.  He then checked the servers.  And there was the problem.

He shot up from his haunted Aeron chair.  “By Zeus, the default gateway has been changed!”

For you see, as traffic comes into the load balancer, it must also return through the load balancer on the way out.  This is the way of things.  This can be done a number of ways, and the method chosen for this infrastructure was by making the load balancer the default gateway.  But when someone changes the default gateway to a device other than the load balancer, the packets are doomed to wander the network, never to find their destination.  They were damend to the bit bucket.

So our mad sysadmin (he was pretty mad,  as he had specifically instructed them not to change the default gateway) changed the default gateways correctly, so that more packets would not suffer the same ghastly fate.  Once this task had been completed, the packets found their way back the client, and all worked.   (The lost packets still haunt the data center to this day!)

Teh End…

or is it?

The idea is that if everything goes terribly wrong, at least your visitors will see something, instead of nothing.

Which begs the question: How do you like to fail?  Fail fast or fail slow? Would it be better to fail slow, where your site becomes slower and slower, or possibly just unresponsive, or would it be better to put up a quick-serving sorry page if the infrastructure melts?

A wildly successful website can easily become a victim of its own success.  Take the case of two sites that experienced exponential growth in a relatively short period of time:  Twitter.com and Myspace.com.

They took two different paths in the realm of failure.  One failed fast, and one failed slow.

Although Myspace has lost most of its lead to Facebook, it’s still a wildly popular social media site.  They had exponential growth from their start in 2003, and there were many periods of time when Myspace.com was just… slow.  Really really slow. You can’t really blame them.  It’s tough when users come faster than you can install servers and provision bandwidth.  It’s a happy problem to have usually, but it’s still a logistical challenge.

Twitter.com came around a bit later, but it also had exponential growth and problems coping.  But for the most part, they failed in a different way:  Fail Whale. When something went terribly awry, instead of a slow site, you’d get a very quick fail whale image.

Perhaps this is a matter of personal opinion, but I think if you’re going to fail, it’s better to fail quick than fail slow.  That is, have a sorry page or sorry site that comes up quick, rather than a site that is too slow for anyone to use.

The quick sorry page can be done with many of the load balancing/ADC vendors by using the backup/sorry serverfarm feature.  Keeping a group of reserve servers, serving up only a “oops, sorry about that” type of page, your own fail whale, can be better than having a really slow or unresponsive web site.

Of course, you won’t always be able to choose the method of your failure.  If your upstream ISP goes dark, there’s not much you can do (unless you have an offsite fail site).  But I personally think having a fail site is a more “professional” way to fail than having a slow or unresponsive site when things go belly up (and we all know they will).

Benefiting from Moore’s Law to a great extent are load balancers/ADCs, where the lowest end device from just about every vendor can handle traffic loads in the 50-100 Mbps range.  Of course, throughput isn’t a terrible way of measuring performance capability of a load balancer (100 Mbps of large file downloads is a heckuva lot easier than 100 Mbps of tiny file connections), but it does relate well to one very important factor in web site serving:

How big is your pipe?

Businesses of all sizes have seen steady increases in their available bandwidth to be sure, but while the growth may be steady, for the most part it tends to be a slower growth rate.  Much slower than doubling every 18 months, like in Moore’s Law.   The result is that increasingly that the lower end offerings from vendors are more than sufficient to run a larger share of web sites out there.

There are a number of reasons for this.  We’re no longer experiencing the exponential accross-the-board growth rates in users that occurred during the dot-com boom.  While there are some sites going through a growth explosion, for most websites in this economic environment, growth rates of any kind are fairly extraordinary.  So extra bandwidth isn’t in as high demand. Secondly, if you’re hosting your own data center as many large businesses do, getting extra bandwidth is often time consuming.  Moving from a single DS3 (45 Mbps) to an OC-3 (155 Mbps) is going to take some time to get that order fulfilled.

Cisco’s ACE 4710 appliance comes with a default license of 1 Gbps of throughput.  F5′s entry-level BIG-IP 1600 LTM maxes out at 1 Gbps.   In the Enterprise market, 500-Mbps to 1 Gbps is about the rock bottom in terms of performance capability.  Yet many of the high-end clients of these vendors don’t push nearly that much traffic.

Companies that aren’t media or mega-content providers (such as Google, Youtube, Yahoo!, Facebook, etc.) that have web applications serving customers or businesses typically don’t go above 100-200 Mbps in traffic, even for some Fortune 500 companies.  Of course, there are exceptions, and there are quite a few factors involved in determining the traffic characteristics of a site. Companies that are offering media such as streaming video or audio often use third-party content providers, such as Youtube or Akami, so as to keep that bandwidth off their own pipe.

So we’ve got all this idle CPU time, so why not make use of it?  That’s what many vendors are doing, in both the enterprise and value markets. With the steady rise in CPU power while bandwidth consumption lags behind, vendors are throwing more and more capabilities into these devices to take advantage of the unused CPU cycles, such as caching, compression, Layer-7 inspection, etc.  Vendors are offering more functionality with the greater power they have available to them.

Then I read the follow-up response over at the 03 blog.  And I got that pang.  You know the one.

Someone is wrong on the Internet.

He made a number of errors about F5′s capabilities, and they were cleared up by Lori in her response.  But there are a couple of items I wanted to address.  Take this quote from his blog post:

She claims that L7 is expensive, sure it takes extra processing, but if you read the article, you’d see that I drop hints that Nginx has a very superior way of handling I/O.

Maybe it’s just me, but “oh, I dropped hints!” seems to be talking down to your audience.  At best, it’s condescending to your audience.  At worst, it’s a shallow and transparent attempt to show a depth of knowledge in an area you actually know only superficially.  Either detail the “very superior way of handling I/O”, or post a link to something detailing as such, or get off the pot.

There’s also his claim of being able to do over 25,000 TPS, which Lori rightfully called into question.

My personal favourite is that she quotes a 3 year old article to try to claim that the Opteron can handle “around 1500″ 1024-bit RSA operations per second, I don’t think she understands what is written in that report, as she has mis-quoted it, and picked a report thats over 3 years old. Lets play far shall we, I know marketing people are used to trying to skew reports, but you try that with me, I’m going to call you on it. Yet we have a running test that shows that machine is handling requests on-par with the F5 solution.

I found a more updated article on a processor very similar to the one used in the original O3 article.  For a 4096-bit operation, the value was 343.12 decrypt RSA operations for 8 cores (dual quad-core Opteron 2356, a similar CPU to the one he used).  4096-bit RSA of course being much more expensive than 1024-bit, but by a predictable amount.  Mulitply by about 32, and you get the number of 1024-bit operations (used in most SSL certificates).  The value comes to about 11,000 for 8-cores, which is inline with what she states, and a lot less than his stated 26,000 connections per second.  And as Lori pointed out, 11,000 would be possible if the system were only doing this SSL work.  He also didn’t refute that he used 512-bit certs.

It’d be easy enough for him to test.  The command to test the system’s SSL capabilities with 8 cores is:

openssl speed rsa -multi 8

So he either really did use 512-bit certs, or he’s actually not measuring TPS correctly.  When an SSL vendor measures SSL, they typically use the term TPS, or transactions per second.  This typically refers to the rate at which the system can accept new SSL sessions, with each new connection requiring an asymmetric encryption operation.    SSL/TLS uses two encryption technologies: Symmetric and Asymmetric.  The asymmetric encryption is relatively expensive on a general purpose CPU like an Opteron (about 1,000 times as CPU intensive as symmetric encryption).  That’s why devices like the F5 and other vendors include SSL accelerator cards, which are special processors that keep the encryption operations off the main CPU.  A device with an SSL accelerator won’t “feel” the impact of an SSL connection any different than a non-SSL connection.

What I’m guessing (and it’s just a guess, since he didn’t state how he did his tests) is that he measured HTTP requests per second, which is a bit different than TPS.  If he used HTTP 1.1 connection persistence, he could do 10 or more requests with only one asymmetric operation.   While that’s an absolutely fair measure of HTTP/S performance, it is not TPS, at least not in the generally accepted way.  If you measured an F5 the same way (multiple HTTP requests through a single SSL connection), the F5 (or any other vendor advertising in TPS) would be able to push far beyond 25,000.

If that is the case, then he could do a comparative TPS test by using HTTP 1.0 mode, where each HTTP request required it’s own TCP connection (and thus asymmetric operation).

And finally, we have this.

1. No offense but covering the beat doesn’t exactly equate to 9 years experience with the technology. Sure you look at the trends, products and evaluations, but this is within a sandbox, not day to day real world experience. I’ve got 2 years experience working with Alteon as a customer back in 1998, working on the bleeding edge at the time of L4 switching. I kept F5 out of the customer site where I was working, simply because Alteon offered a much better and more innovative hardware platform. The web guys liked F5 because it did fancy graphs, Alteon got the job done in terms of performance and scalability. Following that, I spent over 6 years working as a Sustaining Engineer for Nortel / Alteon, responsible for thousands of bug fixes, and beating F5 on many occasions. After that I spent about 18 months working on Open Source App Delivery before returning to Nortel to work on their next-gen platform and help Sustain the Application Switch line. So as you can see, not all experience is equal.

He should really have done more research on Lori.  There aren’t a lot of people in the world with a wider breadth of knowledge in the Layer-7/ADC/load balancing world as Lori (and he ain’t among them).  He makes the mistake of dismissing her as “covering the beat”, referring to her time at Network World Computing, as if her job there only involved sitting through power point presentations and hitting the occasional power switch.  Just looking at her F5 devcentral posts, she has an impressive knowledge from such aspects of the technology as HTTP security, application acceleration, and finer aspects of the HTTP protocol, to application-specific issues such as SOA, XML security, and APIs.   I’ve never met her, but I’ve spoken with her on a number of occasions, and she’s the real deal.