Privacy comes through the use of digital encryption (RSA, AES, etc.) to keep your web interactions, such as credit card numbers, emails, passwords, confidential documents, etc., safe from prying eyes.

But having private communications with another party is all for naught if you’re talking to the wrong party.  You also need trust.  Trust that someone is who they say they are. For Internet commerce to work on a practical level, you need to able to trust that when you’re typing your username and password into your bank’s website, that you’re actually connecting to a bank, and not someone pretending to be your bank.

Trust is accomplished through the use of SSL certificates, CAs (certificate authorities), intermediate certificates, and certificate chains which combined is known as PKI (Public Key Infrastructure).   To elaborate on the use of these technologies to provide trust, I’m going to forgo the traditional Bob and Alice encryption examples, and go for something a little closer to your heart.  I’m going to drop some Star Trek on you.

Let’s say you’re in the market for a starship.  You’re looking for a sporty model with warp drive, heated seats, and most importantly, a holodeck. You go to your local Starfleet dealer, and you find this guy.

Ensign Tony.

Seriously Tony, how do you get girls to even talk to you?

The problem is, you don’t trust this guy.  It’s nothing personal, but you just don’t know him. He says he’s Ensign Tony, but you have no idea if it’s really him or not.  But there is one Starfleet officer you do know and trust implicitly, even though you never met him.  You trust Captain Jean-Luc Picard.

If there’s a problem a peace negotiation can’t solve, I haven’t met it yet

Captain Picard is the kind of guy you start out automatically trusting.  His reputation precedes him. Your browser is the same way, in that right out of the gate there are several sources (such as Verisign) that your browser trusts implicitly.

But you’re not dealing with Picard directly.  Instead, you’re dealing with Ensign Tony.  So Picard vouches for Ensign Tony, and thus a trust chain is built.   You trust Picard, and Picard trusts Ensign Tony, so by the transitive property, you can now trust Ensign Tony.

Whether it’s Internet Explorer, Firefox, Safari, Chrome, Opera, or other browsers, they come built-in trusting a number of sources.

Intermediate Certificates

One of the lesser understood concepts in the us of SSL certificates is the intermediate certificates.  These are certificates that sit between the CA (Picard) and the site certificate (Ensign Tony).

You see, Picard is an important man.  The Enterprise has over a thousand crew members and he can’t possibly personally know and trust all of them.  (In Ensign Tony’s case, there’s also the little matter of a restraining order.)  So he farms the trust out to his subordinates. And one crew member he does implicitly trust is Chief Engineer Geordi La Forge.

I have not clever caption for this image, as it is perfect.

Ensign Tony works for Geordi, and Geordi trusts Ensign Tony.   Thus Geordi becomes the intermediate certificate.  You can’t trust Ensign Tony directly through Picard because Picard can’t vouch for Tony, but Geordi can vouch fro Tony, and Picard can vouch for Geordi, so we have built a chain of trust.   This is why load balancers and web servers often require you to install an intermediate certificate.

This may be the greatest SSL diagram ever made.

Here’s what happens when you don’t install an intermediate certificate onto your load balancer/ADC/web server:

You’re 33 years old Tony, you’d think you would have made Lieutenant by now

One of the practical issues that comes up with intermediate certificates is which one do you use?  The various SSL certificate vendors such as Thawte, Digicert, and Verisign have several intermediate certificates depending on the type of certificate you purchase. Sometimes it’s not always obvious.  If you have any doubts, use one of the SSL certificate validation tools from the various vendors , including this one by Digicert.  It will tell you if the certificate chain works or not. Do not let a test from your browser determine whether your certificate works.  Browsers handle certs differently, and a validation tool will tell you if it will work with all browsers.

I’m an idiot.

I migrated from one hosting system to another.  I run regular backups of my MySQL database.

Well, apparently there was a field in one of the wordpress databases that the mysql backup application didn’t like.   So as it dumped the contents of the database, it hit this particular record and then stopped.  So the backups are frozen in time.

By the time I figured this out, I had purged all the data off my old server.

Woops.

Ensign Tony, Didn’t Anyone Tell You To Test Those Backups?

I’ll be reconstructing the old articles as best I can.  But for now, enjoy Picard’s epic facepalm.

This post covers network topology, which is how the load balancer fits into the network.  How a device fits into the network is usually a difficult concept to get, and often that’s simply because people make it tougher than it need be. Basically, for a load balancer to be put into a network effectively, two things need to happen.

1. Traffic needs to flow through the load balancer on the way in

1. Traffic needs to flow through the load balancer on the way out

The first part is easy, as there’s only one way.  We direct traffic to the virtual IP (VIP) and port sitting on the load balancer.  This is the IP and port that pretends to be the server.  Getting traffic through the load balancer on the way out is probably one of the toughest concepts to grasp when learning load balancers, as there are several ways to accomplish this.

There’s on method of getting traffic through the load balancer on the way out that’s a quick way to drop a load balancer into an existing infrastructure with minimal changes to the network topology.  This is called one-armed, route-path.

One-armed, route path is not as popular as some of the other methods, although it has the distinct benefit of being a good, quick “drop-in” deployment.  Here’s how it works.

Let’s say you’ve got a network with a couple of servers sitting behind a firewall.  This firewall does NAT from a public address space to private IPs. This is a pretty common scenario for a small to medium sized business.

In the example shown above, the default gateway for the servers is the firewall, at 192.168.1.1.  To network admins, The concept of a default gateway is second nature.  To server folks, keep this in mind:  If you want to send IP traffic to a system not on your local network, you need a router to handle delivery.  That is your default gateway.  Without a default gateway for your servers, you can’t communicate with the Internet.

So now lets say we want to drop a load balancer into the network.  There are several options, and for the most part the advantages of one over another are logistical, not performance related.  For example, to do two-armed, Layer 3 path (arguably the most common topology), you would need to put in a new IP network between the firewall and the servers, and one new Layer 2 network.  This would require re-addressing the IPs on all the servers.

And while adding a new Layer 2 and Layer 3 network would certainly work, we can use one-armed, Layer 3 path without the need to re-IP all the servers or adding new networks.

In the figure above, you see that we’ve changed the default gateway on the servers to that of the administrative IP of the load balancer (if there were two load balancers, they would have a floating administrative IP which you would use as the default gateway).   The default gateway of the load balancer is that of the firewall.

This seems a little odd, as we’ve got two default gateways on the same IP network.  While unusual, it works, and it’s a handy way to drop a load balancer into a network with minimal changes.

The idea is that if everything goes terribly wrong, at least your visitors will see something, instead of nothing.

Which begs the question: How do you like to fail?  Fail fast or fail slow? Would it be better to fail slow, where your site becomes slower and slower, or possibly just unresponsive, or would it be better to put up a quick-serving sorry page if the infrastructure melts?

A wildly successful website can easily become a victim of its own success.  Take the case of two sites that experienced exponential growth in a relatively short period of time:  Twitter.com and Myspace.com.

They took two different paths in the realm of failure.  One failed fast, and one failed slow.

Although Myspace has lost most of its lead to Facebook, it’s still a wildly popular social media site.  They had exponential growth from their start in 2003, and there were many periods of time when Myspace.com was just… slow.  Really really slow. You can’t really blame them.  It’s tough when users come faster than you can install servers and provision bandwidth.  It’s a happy problem to have usually, but it’s still a logistical challenge.

Twitter.com came around a bit later, but it also had exponential growth and problems coping.  But for the most part, they failed in a different way:  Fail Whale. When something went terribly awry, instead of a slow site, you’d get a very quick fail whale image.

Perhaps this is a matter of personal opinion, but I think if you’re going to fail, it’s better to fail quick than fail slow.  That is, have a sorry page or sorry site that comes up quick, rather than a site that is too slow for anyone to use.

The quick sorry page can be done with many of the load balancing/ADC vendors by using the backup/sorry serverfarm feature.  Keeping a group of reserve servers, serving up only a “oops, sorry about that” type of page, your own fail whale, can be better than having a really slow or unresponsive web site.

Of course, you won’t always be able to choose the method of your failure.  If your upstream ISP goes dark, there’s not much you can do (unless you have an offsite fail site).  But I personally think having a fail site is a more “professional” way to fail than having a slow or unresponsive site when things go belly up (and we all know they will).

In the networking realm, there is the concept of PDUs, or Protocol Data Units.  These are the discreet messages for a particular protocol.   Every protocol has them.  In Layer 2, the Ethernet protocol has the Ethernet Frame, and for Layer 3 the IP protocol has the IP packet, just to name a few examples.   Each of theses PDUs have their own format.

For the HTTP protocol, which operates on Layer 7, we also have a PDU: The HTTP message.  HTTP messages are further divided into two categories: HTTP requests and responses.

When your browser wants an object, such as an JPG or HTML file, it must make a properly formatted HTTP request.  The server responds with an HTTP object.  One object equals one request plus one response.  Typically the response includes the object, although sometimes the response says simply “file not found” (HTTP 404), “object moved” (HTTP 301 or 302), or even “access denied” (HTTP 400), among others.

But the thing to keep in mind is that every object, which is to say every JPG, HTML, Flash, CSS, MPEG, every individual file, will require its own HTTP request and will generate an HTTP response from the server.  A web page consisting of 10 objects will require 11 HTTP requests:  One for the HTML page itself, and 10 additional requests to pull all of the objects referenced in the HTML.

In the next post, we’ll discuss some tools you can use to troubleshoot HTTP requests and responses.

Benefiting from Moore’s Law to a great extent are load balancers/ADCs, where the lowest end device from just about every vendor can handle traffic loads in the 50-100 Mbps range.  Of course, throughput isn’t a terrible way of measuring performance capability of a load balancer (100 Mbps of large file downloads is a heckuva lot easier than 100 Mbps of tiny file connections), but it does relate well to one very important factor in web site serving:

How big is your pipe?

Businesses of all sizes have seen steady increases in their available bandwidth to be sure, but while the growth may be steady, for the most part it tends to be a slower growth rate.  Much slower than doubling every 18 months, like in Moore’s Law.   The result is that increasingly that the lower end offerings from vendors are more than sufficient to run a larger share of web sites out there.

There are a number of reasons for this.  We’re no longer experiencing the exponential accross-the-board growth rates in users that occurred during the dot-com boom.  While there are some sites going through a growth explosion, for most websites in this economic environment, growth rates of any kind are fairly extraordinary.  So extra bandwidth isn’t in as high demand. Secondly, if you’re hosting your own data center as many large businesses do, getting extra bandwidth is often time consuming.  Moving from a single DS3 (45 Mbps) to an OC-3 (155 Mbps) is going to take some time to get that order fulfilled.

Cisco’s ACE 4710 appliance comes with a default license of 1 Gbps of throughput.  F5′s entry-level BIG-IP 1600 LTM maxes out at 1 Gbps.   In the Enterprise market, 500-Mbps to 1 Gbps is about the rock bottom in terms of performance capability.  Yet many of the high-end clients of these vendors don’t push nearly that much traffic.

Companies that aren’t media or mega-content providers (such as Google, Youtube, Yahoo!, Facebook, etc.) that have web applications serving customers or businesses typically don’t go above 100-200 Mbps in traffic, even for some Fortune 500 companies.  Of course, there are exceptions, and there are quite a few factors involved in determining the traffic characteristics of a site. Companies that are offering media such as streaming video or audio often use third-party content providers, such as Youtube or Akami, so as to keep that bandwidth off their own pipe.

So we’ve got all this idle CPU time, so why not make use of it?  That’s what many vendors are doing, in both the enterprise and value markets. With the steady rise in CPU power while bandwidth consumption lags behind, vendors are throwing more and more capabilities into these devices to take advantage of the unused CPU cycles, such as caching, compression, Layer-7 inspection, etc.  Vendors are offering more functionality with the greater power they have available to them.

But that he may have used the wrong terminology for the performance testing he did (saying it was TPS instead of HTTP requests per second) shows that there’s a lot of confusion on benchmark terminology, so I’m going to go over some of the basics.

TPS

In the load balancing world, TPS (Transactions per second) refer to the number of new SSL connections initiated.  The new part is important, because each new SSL connection requires a relatively CPU-expensive asymmetric encryption operation.  This is why most load balancers that do SSL have a separate chip for SSL processing (SSL ASIC), which offloads the SSL functionality from the main CPU.

Once the first step of an SSL connection is completed, the encryption then shifts to the much more CPU-friendly symmetric encryption, which is often referred to as “bulk encryption”.  Pushing bulk encryption throughput is relatively easy for a load balancer, even without an SSL accelerator chip.

However, HTTP/HTTPS typically involve short-lived connections, so there is relatively little throughput, and a lot of connection setup/teardown. Hence the need to know the TPS rating of a given device.

Many vendors will offer tiered licensing for SSL TPS.  So keep in mind when they mean TPS, they usually reffer to *new* SSL connections per second.

Connection Rate versus Request Rate

There are two ways to measure “rates” with load balancers:  Connection rate, and request rate.  While they sound similar, when you get right down to it, they’re actually quite different.

Connection rate refers to the number of TCP connections per second a device can handle.  HTTP request rate refers to the number of HTTP requests the device can handle.  How are they different? You can have multiple HTTP requests in a single TCP connection.

When your browser goes to a web site, it firsts initiates a TCP connection to the server (or in our case, a load balancer load balancing traffic for servers).  In that TCP connection, your browser will typically make several HTTP requests over that connection.

Making multiple requests over a single TCP connection is a lot easier than making a TCP connection for every single request.  In fact, the original HTTP 1.0 specification required on TCP connection per request.  The HTTP 1.1 specification fixed that, by allowing the multiple requests per TCP stream.

Layer-4 versus Layer-7

When a load balancer operates in Layer-4 mode, it’s functioning a lot like a router.  In fact, it’s not doing much more than your wireless access point at home.  Very little memory is consumed with each new connection, and only the TCP/IP header information is evaluated.

When a load balancer operates in Layer-7 mode, it’s functioning more like a server.  The TCP session is terminated at the load balancer, and a new TCP connection is initated to the server.  HTTP requests are buffered in the load balancer’s memory in order to be evaluated.  This requires a lot more processing power and a lot more memory.

Obviously, a load balancer can handle more Layer-4 workload than Layer-7, so it’s important to know which mode you plan on using when it comes to performance.

The deal was pretty what had been theorized, although I think  the biggest surprise was that Radware has stated (and reiterated in my conversation with Mr Zisapel) that the deal is not a pure customer-list purchase, as many had suspected.  Instead, Radware has comitted to supporting and expanding the existing Alteon line, and not just moving everyone onto Radware boxes.   They’ve also committed to a 5-year product support plan of the current Alteon lines.

The deal isn’t a done-deal yet (the bankrupcty courts of the US and Canada has to approve this) but it seems pretty likely to go through.

But the ability to look at the HTTP headers doesn’t always work the way you might think it would.  Often, the load balancer can have a short attention span.

In a traditional HTTP 1.1 connection, multiple HTTP requests are sent through a single TCP connection. Most load balancers by default will only look at the first HTTP request, and ignore the rest.  To elaborate on this, let’s take a look at two of the basic concepts of HTTP.

HTTP Basics

The HTTP protocol can be broken up into to parts: HTTP requests and HTTP responses.  Both are comprised of two components: HTTP headers and HTTP content.

In both HTTP requests and HTTP responses, there are always HTTP headers.  In an HTTP request, there is sometimes content, such as a form POST, or uploading a file.  In the HTTP response, there is usually content, but there are cases when there is none (such as with an HTTP HEAD request).

And there’s one more important bit to keep in mind with regard to HTTP: Every object has a separate request and a separate response.  That’s every JPG, GIF, Flash file, HTML file, etc.  So a web page with 20 images will invoke 21 different HTTP requests; one for the HTML page itself, and 20 for the objects (such as images) referenced in the HTML file.

With HTTP 1.1, all of those 21 objects in a web page are typically requested in a single TCP stream, rather than 21 individual connections (which would be fairly inefficient).  But this presents a problem for load balancers.

Do load balancers look at the data in the first request out of the 21? Or does the load balancer look at each request individually?

The driving force behind Layer 7 persistence (keeping an individual user tied to a specific server in a server group based on HTTP headers instead of IP address) was the dreaded AOL Megaproxy issue.  AOL had the nasty little tendancy of routing all web traffic through a couple of mega proxies located throughout the US and Canada.

This caused a problem with the previous method of persistence, which was to base it on source IP address. Typically, one IP address equaled a single user.  However, with AOL, you could have 20,000 users coming from a single IP address.  The load balancer would think it’s a single user, and if you had 300 servers ready to take orders, all 20,000 users would go to one.  That situation has happened a few times, and it’s hillarious, so long as you aren’t the company with the 300 servers.

I still teach that mega proxy problem, mostly out of muscle memory.  But I stopped to think about it, do we really have a problem with megaproxies anymore?  Does AOL even do this practice, and even if they did, is AOL represent a significant amount of traffic?

The answer to the later question is almost certainly no.  AOL has seen a dramatic drop in subscribers, and most people connect directly to the Internet through their cable modem or DSL provider.  And I don’t know of any major Internet provider that utilizes proxies for their users Internet requests.

Layer 7 persistence is still applicable to situations where you may have multiple users coming from a single IP address (such as a small client base coming from a handful of offices, with each office using on public IP address), but I wonder what doing Layer 4 persistence would do to a major site these days.  I’m thinking, not much.

What do you think?