So I’ve created a new blog: datacenteroverlords.com. This deals with all the issues surrounding the modern data center, such as virtualization, networking, storage, and of course load balancing. The name comes from the new role that’s forming around the multiple disciplines involved in data centers. Some call it the data center admin.

Behold my mighty data center!

I prefer the term “Data Center Overlord”.

I’ll likely be blogging there exclusively now.

• Lori and I have a (in good fun) BATTLE ROYAL over the term “ADC” versus load balancers.
• I have single-handedly renamed Direct Server Return (DSR) to “Bat-shit Crazy Mode”.
• We talked about challenges with providing accurate health checks
• We were distracted several times by squirrels.

Have a listen, good stuff.

I agree, they do more than they’ve done before. From application logic to web application firewalls to VMware integration, modern application delivery controllers do a lot. But they still also load balance.  And that’s what everyone calls them.

A rose by any other name. Pictured: Olfactory stimulation vector

Using the term load balancer saves me the conversation: “What’s an application delivery doo-hickey?”

I still call them load balancers because it serves no purpose to rename them.

Since at least 2006 there’s been an effort to rebrand load balancers as application delivery controllers. Gartner has moved to the new term, as have most of the vendors. Marketing has been heavy to rename them. Some vendors even use the term load balancer as a disparaging term for their competitors.

But here’s the problem: We’ve had at least 5 years of marketing, press releases, and events, and still no one (outside of the vendors and specialists) seems to know what an application delivery controller is. When I teach load balancing classes, very few in the class are even aware of the term.

What network administrators, server administrators, and application developers do know is load balancers. When you say “load balancer”, they universally understand what they do and the benefit they provide. Generally speaking, they have no idea what an ADC is.

I have no problem educating on a new term, I’d even help evangelize the term if it made sense. But it doesn’t. Renaming them ADCs adds nothing substantive to the industry, only confusion and an extra conversation.

If I told you I got a new multi-media climate controlled dynamic geographical device, you’d think I’d be some sort of mad scientist. But no, that’s just another name for a car. Cars today do a lot more than cars 50 years ago did, but they’re still cars.

I understand the though behind the attempt to rename them, but I think it’s a mistake. I don’t mind mistakes, but I think its time to own up to the error and start calling them load balancers again.

Technology is complicated enough, we shouldn’t make it more complicated by adding in terms when none are needed.

I’ll give you a minute because I assume if you were drinking a beverage, it’s now all over your computer screen.

How much damage could someone do? Take Google for example. This means they could set up a fake Gmail-looking server, and collect the username and password of a user. The user might not ever realize that the site was fake, and their passwords were compromised.

So now there are totally legit-looking certificates out there. Your browser, and every other browser in the entire world trusts them.

Remember, SSL gives us two things: Privacy and trust. Privacy comes though symmetric encryption, and trust is done through signed certificate chains.

Trust needs to start somewhere.  With your browser, whether its Firefox, Safari, IE, Chrome, Opera, whatever, they all come with pretty much the same set of root certificate that act as the start of trust.  Essentially, they come trusting several sources.

So what happens if this trust is broken, or if the certificate was issued under false pretenses? There needs to be a way to revoke that trust on a certificate by certificate basis. There’s only two ways to do this: A manually updated CRLs (certificate revocation list), or through the OCSP protocol.

CRLs aren’t a terribly good way to handle it. A CRL is simply of list of certificates that would validate the chain of trust through the regular way (Picard -> LaForge -> Ensign Tony), but aren’t trusted anymore. Each of your browsers have their own CRLs, and they can be updated by an OS or browser patch. This relies on you or your organization reliably updating software and/or OS, which doesn’t always happen in a timely manner. Even if it’s timely, there’s always a period of time between when the certificate is revoked and when you get that revoked certificate added to your CRL.  This could be hours, days, or even weeks where your browser would trust an otherwise bogus certificate.  This just doesn’t scale.

OCSP is a better approach, as it can check with a certificate authority every time it hits a website with an SSL certificate. So not only does the browser do the usually trust check, it double checks by checking with the source (an OCSP server hosted by the certificate authority) that the trust is still valid.

If my browser ran OCSP, I don’t have to worry that I might miss a revoked certificate because I haven’t updated my browser or OS.  It can also check every time, so if a cert has been revoked, my browser finds out right away.

So while OCSP is a better approach to CRLs, it isn’t used universally. And it isn’t “fail closed” by default on some browsers, as shown in this table from David Holmes at F5.

So right now, OCSP and CRLs are no guarantee that you can trust a certificate unless you use Firefox or Chrome.

Wait, what? No guarantee? Shit.

This is what happens when people don’t trust each other

There are some opinions that certificates should be handled differently than they are now. He discusses some interesting ideas, but I think it’s fair to say PKI (public-key infrastructure) needs a bit of overhaul.

The best known product for traffic manipulation is likely F5′s legendary iRules, but other vendors have similar capability such as A10′s aFlex.   Essentially, this puts an application development platform. Typically this is done with a standard programming language, such as a modified TCL for iRules and aFlex.

Some vendors, (Cisco I’m looking at you, as well as Brocade but I’m less familiar with them lately) lack the ability to manipulate traffic using application logic.

For vendors, this is a great feature to have. Its attractive to potential customers, and it makes it difficult to move to a platform that doesn’t have this feature. I call it the Charlton Heston feature, since you’ll only pull it out of a client’s cold dead hands.  Once you use it, you’re fairly dependent on it.

Get your paws off me, you damned dirty network admins!

In general, I’m a fan of iRules and their ilk. There’s just too many situations where the ability to manipulate HTTP content has saved the day.

So what kind of manipulation can you do?  There are rules to scrub credit card numbers, so if a web application tries to display a credit card number such as “5123-1234-1234-1234″ (a big no-no according to PCI-DSS), the load balancer would do a regular expression search for that pattern, and replace it with “XXX-XXX-XXXX-1234″ before sending the response off to the client.

With a programming language and regular expressions, just about anything is possible. And there in lies a problem.

Two Edged Sword

Having application logic on the load balancer is a double-edged sword.  One the one hand, it allows you to have very granular control over headers and content for requests and responses.  The drawback is it allows you to have very granular control over headers and content for requests and responses.

The awesomeness of this manipulation is evident, but there are some caveats.

We’ll fix it in post

There’s a running joke in the film industry called “we’ll fix it in post”.  Essentially, it means who cares if we get it right here, we’ll just fix it with special effects or editing.  Except it hardly ever works.  You’ve got the same hazard in something like iRules; it can be a lousy and lazy way to fix to a problem that really should be fixed in the code.

Capacity

Of course, all this regular expressioning and application logic come at a cost in terms of CPU.  The more of it you do, the lower capacity you’ll have.  A load balancer capable of serving up 4 Gbits of second of traditional Layer 4-7 traffic may have its capacity dropped to 500 Mbps.  And there’s no way of telling what the new performance ceiling would be (until you hit it like Wile E Coyote on a rocket sled).

This is less of an issue than it used to be, as Moore’s law has made processors faster and faster, allowing load balancers to handle increasingly difficult tasks, while bandwidth requirements have not increased nearly as dramatically. Not many organizations have more than a couple hundred megabits per second to the Internet, while load balancers are capable of handling several (even dozens) of Gigabits.

Latency

Latency is a dirty word in networks and applications, and certainly a potential issue with application logic.  If you’re going to look at a cookie, the latency imposed by a load balancer will be minimal.  If you’re going to calculate Pi to the 100th digit on every HTTP request, that’s going to add a certain amount of latency to the transaction.  Like the capacity ceiling, it’s very difficult to predict what that additional latency will be when adding more and more logic. And depending on the load profile, that latency may vary quite a bit over time.

Creep

Capacity and latency are easy enough issues to deal with, but creep is very problematic.  It’s a “Layer 8″ problem, and one that can quickly spiral out of control.

Typically the creep comes into play when you first implement application logic, and it fixes a vexing problem. You’re the hero, and someone perks their ears up and says “wow, what else can you do with it?”

“Anything”, you say confidently.  And anything is what they throw at you. Add semi-colons at the end of paragraphs, replace all double-spaces after a period with single spaces, etc.

It can very easily spiral out of control, so you need to know when to put your foot down (hint: early).

Skills

To develop this application logic, you need to understand HTTP really well. When developing most web applications on a platform like ASP or PHP, you don’t really need to understand HTTP all that well.  But the load balancer manipulates on the HTTP level, so you’ll really need to be up on your HTTP. You also need to have the programming skills to pay the bills. Especially for a network admin, those programming muscles might not get flexed on a regular basis.

I’ve also heard the argument that iRules and their ilk are only there to fix problems that should have been fixed in the code. And there’s a lot of truth in that; they are typically used in situations where the problem could also be solved at the application.

But that’s also like saying that I shouldn’t need a lock on my door, because people should not be jerks and go around stealing stuff.

In IT, there are always going to be situations where its either easier to fix the problem on the load balancer, or it’s the only viable solution (code locked down, client doesn’t control the code as its a third party, developer ran away to Brazil with their secret family and won’t return calls).

Bottom line? I’d rather have it than not have it, but I’m very careful with it. After all, with great power comes great potential for epic fail.

I’m a big fan of virtualization.  There’s a lot to like about it, including consolidation (getting rid of space-heater servers that do nothing running 1% CPU and sucking up electricity and throwing off heat), flexibility, and management.  I’ve even gone and got my VCP4 (VMware Ceritified Professional 4) certification.  (Hear that ladies? I’m certified.)

One aspect of virtualization I’m a fan of is the appliances.  Vendors are taking physical appliances (such as a Vyatta router) and turning it into a VM appliance.  I don’t have to worry about an underlying operating system (and the requisite patches), the appliance vendor handles the software and the OS.

Several load balancing vendors have gotten into that virtualization game.  Vendors that have traditionally offered hardware appliances now have virtual appliances (some for years).  From From F5 to KEMP, from Coyote Point to loadbalancer.org, there are a number of virtual load balancers/ADCs to choose from.   And for the most part, they offer the same features as their hardware brethren.

Their throughput and performance is hampered somewhat by the fact that they’re all software and no silicon. Everything is done in the x86 virtualized CPU(s).  Still, depending on how you provision them, they can generally handle several thousand HTTP requests per second even in Layer 7 mode.

The one caveat to virtual load balancers is that their SSL performance is severely limited.  Even value-market load balancers that do most of their functions in a general purpose CPU will still use SSL ASICs for the asymmetric crypto (even using the general processor for the less CPU-intensive symmetric crypto).

The first part of every new SSL connection is a very CPU-intensive asymmetric operation (about 1000x more CPU intensive than symmetric operations).    CPUs that can normally handle tens of thousand of regular TCP connections per second can only handle a few thousand SSL connections at the most.

It is technically possible to do hardware SSL acceleration on a VM load balancers however.  It requires that the virtual machine host (like VMWare ESXi) have an SSL card installed, and VM Passthrough enabled (where the virtual machine can have direct access to physical hardware).

Unfortunately, these SSL cards are tough to come by.  Cavium is probably the most notable vendor, but cards from them aren’t exactly easy to come by, and they’re fairly expensive.  You may not have the option if you’re using blade systems.   And if you you want to leverage features like HA and DRS (using ESX hosts in a cluster), then every machine in the cluster would need to have such a card.  I’m not aware of any virtual load balancer vendor that even supports this configuration.

There are a lot of situations where virtual load balancers make a lot of sense, but keep in mind that the SSL performance capability is going to be fairly constrained.

SSL performance is a tricky thing to measure.  There are a lot of different aspects to consider, and on top of that different vendors use slightly different definitions for the same terms. I was talking with a vendor the other day, and I realized as we were talking performance numbers we had different definitions for the term TPS (Transactions Per Second) and CPS (Connections Per Second).

TPS is probably one of the more ill-advised acronyms (boy do we love acronyms) for load balancers, because it’s open to debate what the “transaction” in transactions per second means.

SSL Basics

Before we talk about CPS versus TPS, let’s review some SSL basics.  In a new SSL connection, two things happen: An asymmetric exchange, then a transition to symmetric encryption.

The asymmetric part is why load balancers from F5 to KEMP use SSL accelerator chips.  It’s incredibly CPU intensive; so much so that a server that is capable of tens of thousands of connections per second of regular TCP connections is only capable of several hundred SSL connections.  To keep this from crippling the CPU, SSL accelerator processors are used to offload the cryptographic functions from the main CPU.

So when we measure SSL performance of a load balancer, we typically want to measure three things:

1: How many new SSL connections per second can a device handle, that is how many of the expensive RSA operations per second are possible.

2: How much bandwidth of SSL traffic can the device push.  This is symmetric encryption (such as AES), and is much easier on a CPU.

3: How many HTTP requests per second can the device handle after an SSL connection is established.  In HTTP 1.1, a client is allowed to make multiple HTTP requests off a single TCP/SSL connection.  This is far more efficient than the old HTTP 1.0 standard, that required a separate TCP connection for each object.  Many sites will have pages with literally hundreds of objects (I’m looking at you http://vg.no).

The trick is how many HTTP requests per SSL connection?  Personally, I’d say anywhere between 10 to 20 is a pretty good place to start, but that’s not what everyone uses.  In an epically flawed fight that o3 magazine picked with F5, author John Buswell claimed that an Nginx-based box he built could handle 25,000 TPS.

The system had no problems handling over 26,590 TPS, the test lab ran out of capacity to generate additional transactions. Compare that to the F5 Networks Big-IP 6900 which handles a maximum of 25,000 TPS but carries a starting price tag of $55,000.

The box he used had 2 quad core Opteron CPUs that were in no way capable of doing 26,000+ new SSL connections per second without an SSL accelerator card (which he didn’t use).  A few thousand 1024-bit RSA operations per second at the most, but definitely not 26,000.

He likely made the mistake of opening a couple of SSL connection, and running 25,000+ HTTP request per second off those open TCP/SSL connections (which is not even close to a real world scenario).   25,000 HTTP requests per second is within the capabilities of his box.  This is not how F5 measures TPS, nor how any other vendor measures TPS.  Of course, I’m only guessing at his methodology, as he never bothered to share his testing methodology.  (F5 and other vendors share their methodology).

TPS versus CPS

While SSL throughput is pretty easy to measure and the definition is identical for all vendors, the terms TPS and CPS aren’t.  So what do they mean?  CPS (Connections per Second) can mean a couple of things.  For SSL, it would typically mean new SSL connections, requiring the asymmetric operation.  Some vendors use TPS for this definition (the T standing for an asymmetric transaction), while other vendors use CPS for the SSL portion, and TPS for the HTTP-within-an-SSL connection.

So it’s important to understand what’s being measured.  Different vendors have different ideas of what that means, although they are honest differences of opinions (things reasonable people can disagree on).  Most vendors are reputable and will outline the methodology they used to come up with their numbers (03 magazine did not).

The fact is, we’re more addicted to cookies than even Cookie Monster

Came upon this post on Slashdot.org, a criticism of HTTP cookies entitled “HTTP cookies, or how not to design a protocol“.    It goes into many of the security issues surrounding HTTP cookies, and how the popular criticisms (privacy) aren’t the real problems.

The problem is, HTTP cookies are absolutely vital to everything we do on the Internet. Any website that we visit where the server builds up customized content (even if it’s just selection which region we’re from, like on Fedex.com), depends up on cookies.

Want to play a nasty prank on a co-worker who doesn’t lock their screen when they leave their desk?  Disable cookies on their browser.  Nothing works.

Cookies are just about the only mechanism in use to create a unique relationship between a client and a server. In other words, a cookie is the only way to establish a session.  Other than cookies (or long URLs), the HTTP protocol does not

In the end, we’re worse than cookie monster with our cookie addiction.