Privacy comes through the use of digital encryption (RSA, AES, etc.) to keep your web interactions, such as credit card numbers, emails, passwords, confidential documents, etc., safe from prying eyes.

But having private communications with another party is all for naught if you’re talking to the wrong party.  You also need trust.  Trust that someone is who they say they are. For Internet commerce to work on a practical level, you need to able to trust that when you’re typing your username and password into your bank’s website, that you’re actually connecting to a bank, and not someone pretending to be your bank.

Trust is accomplished through the use of SSL certificates, CAs (certificate authorities), intermediate certificates, and certificate chains which combined is known as PKI (Public Key Infrastructure).   To elaborate on the use of these technologies to provide trust, I’m going to forgo the traditional Bob and Alice encryption examples, and go for something a little closer to your heart.  I’m going to drop some Star Trek on you.

Let’s say you’re in the market for a starship.  You’re looking for a sporty model with warp drive, heated seats, and most importantly, a holodeck. You go to your local Starfleet dealer, and you find this guy.

Ensign Tony.

Seriously Tony, how do you get girls to even talk to you?

The problem is, you don’t trust this guy.  It’s nothing personal, but you just don’t know him. He says he’s Ensign Tony, but you have no idea if it’s really him or not.  But there is one Starfleet officer you do know and trust implicitly, even though you never met him.  You trust Captain Jean-Luc Picard.

If there’s a problem a peace negotiation can’t solve, I haven’t met it yet

Captain Picard is the kind of guy you start out automatically trusting.  His reputation precedes him. Your browser is the same way, in that right out of the gate there are several sources (such as Verisign) that your browser trusts implicitly.

But you’re not dealing with Picard directly.  Instead, you’re dealing with Ensign Tony.  So Picard vouches for Ensign Tony, and thus a trust chain is built.   You trust Picard, and Picard trusts Ensign Tony, so by the transitive property, you can now trust Ensign Tony.

Whether it’s Internet Explorer, Firefox, Safari, Chrome, Opera, or other browsers, they come built-in trusting a number of sources.

Intermediate Certificates

One of the lesser understood concepts in the us of SSL certificates is the intermediate certificates.  These are certificates that sit between the CA (Picard) and the site certificate (Ensign Tony).

You see, Picard is an important man.  The Enterprise has over a thousand crew members and he can’t possibly personally know and trust all of them.  (In Ensign Tony’s case, there’s also the little matter of a restraining order.)  So he farms the trust out to his subordinates. And one crew member he does implicitly trust is Chief Engineer Geordi La Forge.

I have not clever caption for this image, as it is perfect.

Ensign Tony works for Geordi, and Geordi trusts Ensign Tony.   Thus Geordi becomes the intermediate certificate.  You can’t trust Ensign Tony directly through Picard because Picard can’t vouch for Tony, but Geordi can vouch fro Tony, and Picard can vouch for Geordi, so we have built a chain of trust.   This is why load balancers and web servers often require you to install an intermediate certificate.

This may be the greatest SSL diagram ever made.

Here’s what happens when you don’t install an intermediate certificate onto your load balancer/ADC/web server:

You’re 33 years old Tony, you’d think you would have made Lieutenant by now

One of the practical issues that comes up with intermediate certificates is which one do you use?  The various SSL certificate vendors such as Thawte, Digicert, and Verisign have several intermediate certificates depending on the type of certificate you purchase. Sometimes it’s not always obvious.  If you have any doubts, use one of the SSL certificate validation tools from the various vendors , including this one by Digicert.  It will tell you if the certificate chain works or not. Do not let a test from your browser determine whether your certificate works.  Browsers handle certs differently, and a validation tool will tell you if it will work with all browsers.

When using Visio to do a network diagram, there is almost always the ubiquitous cloud icon.   The cloud represents the rest of the Internet typically, with no detail other than “The Internet”.  The Internet is of course a ton of detail.  Routing protocols, IP packets, WAN links, interconnects, hundreds of thousands of network devices making sure IP traffic gets from point A to point B.

But unless you’re a network administrator, you don’t care.  And even if you are a network administrator, you only care about your little piece of the Internet.   The rest of it is there, but it’s someone else’s responsibility to make sure it works.  You don’t need to concern yourself with the details.  The Internet is a cloud, full of Shit You Don’t Care About.

Technology in a way is a continual process of making more and more of the technical world Shit I Don’t Care About.  A few decades ago, programmers needed to know machine language to get anything done.  This gave rise to assembly language, and that begat C and other low level languages, to Java, PHP, and Perl.   Each step up abstracts more and more of the base technology. Perl and PHP, for instance, turn memory pointers into Shit I Don’t Care About anymore.

Arthur C. Clarke, a novelist known for his incredible insight into technology’s affect on the future (he proposed the idea of geosynchronous orbit satellites, have you heard of them?) mention this phenomenon in his book 3001: The Final Odyssey.  In the book, the chief engineer of the inter-planetary vessel Goliath can’t explain to the to protagonist Frank Poole much about the inner workings of the ship (Scottie would be appalled!).  The evolution of space craft had made them so incredibly complex (and automated) that only the basic operation needed to be (or could be) understood by its operators.

The iPad and the newest generation of smartphones are devices that transition a layer of Shit I Care About into the realm of Shit I Don’t Care About: Namly the desktop, drivers, and the concept of a PC desktop.

So that’s what The Cloud is; it’s Shit I Don’t Care About.  There’s some type of interface for interaction, (HTTP for web apps, APIs for data retrieval, etc.) and then there’s a bunch of shit behind the scenes that I don’t care about.  Or at least, that’s my definition of The Cloud (among thousands of conflicting definitions).

For instance, does your website run Linux or Windows?  Do your users care?  Most likely not.  They’re not going to interact with the OS, so why would they care? It’s Shit They Don’t Care About.   Oracle or Microsoft SQL?  Again, I don’t care.  Just as I don’t care if the plane I’m flying on is an Airbus 320 or a Boeing 737.

So Cloud Computing is the act of transitioning infrastructure I used to care about into the realm of Shit I Don’t Care About.  Servers, storage, networking, etc.  If I can transition Shit I Care About into Shit I Don’t Care About, I can concentrate on some New Shit.

Cloud Computing is transitioning Shit I Care About into Shit I Don’t Care About so I can spend time on New Shit.

I’m an idiot.

I migrated from one hosting system to another.  I run regular backups of my MySQL database.

Well, apparently there was a field in one of the wordpress databases that the mysql backup application didn’t like.   So as it dumped the contents of the database, it hit this particular record and then stopped.  So the backups are frozen in time.

By the time I figured this out, I had purged all the data off my old server.

Woops.

Ensign Tony, Didn’t Anyone Tell You To Test Those Backups?

I’ll be reconstructing the old articles as best I can.  But for now, enjoy Picard’s epic facepalm.

Recently Cavium purchased MontaVista for $50 million.  More at moblinzone.

Not as prominent in the press release, but I personally think is the neatest feature, is the dynamic transparency.  Transparency is when the source IP address of the client is maintained, which is the default method for most load balancer.  The LoadMaster’s non-transparency is probably known more commonly in the industry as Source NAT, or SNAT.  This is when the client’s IP address is replaced by an IP on the load balancer.

When preserving the true source IP address of your clients, you cannot have clients on the same network as your servers.  This is sometimes referred to as “the same subnet problem”.  The cause of this is that the traffic needs to pass through the load balancer on the way out.  If the client is on the same subnet as the servers, the servers reply directly to the client, rather through the load balancer.

The solution for the same subnet problem is usually to enable SNAT/non-transparency, but you lose the true source IP address of your clients, so the web server logs will show everyone coming from one address.

The higher-end load balancers have the ability to do selective SNAT, and now KEMP has the ability to do selective SNAT automatically. I’ve yet to see it in action, so I can’t attest to how well it works,  but it’s potentially a very nice feature.

But it seems Radware has made good on their promise to keep up the Alteon line, although in slightly different form.  They’ve released the Alteon 5412, a 20-Gigabit Layer 7 device (web acceleration and SSL promised for mid-2010).   It’s the Alteon software running on top of their OnDemand Switch 3 switching platform. Since the acquisition,  they have also released two maintenance releases of the Alteon OS for the previous platforms.  So it seems they’re making good on their promise.

Check out the press releases and promo site (bringing back Alteon marketing materials even).

TALES OF LOAD BALANCING HORRORS!

For tales of persistent terror, challenging your keep-alive, I give you the following vignettes. (Names have been changed to protect my ass, as well as to punch up some bone dry material.)

The Default Gateway To The Abandoned Zone

Several years ago, on a dark and stormy night, a dashingly handsome young hot-shot system administrator had just finished setting up a new web infrastructure for a client.  They were moving their web infrastructure from their own facility, to the co-location facility where Mr. Sysadmin worked.  Mr Sysadmin was also responsible for the load balancer.  Running bravely into the load balancing realm while both the other sysadmins and the network admins dared not tread.

“It’s cursed!” They said.  Or maybe they used curse words to describe it.  It was a long time ago.

He powered up the system, tested the traffic, and cried out into the night “It’s alive! It’s pushing traffic!”  Overly pleased with his unnatural creation, he emailed the client to tell them their configuration was ready.  They moved in, with administrator access to all systems.

At first, the infrastructure worked as promised.  Sites were served, and loads were balanced.  Then, a call came from beyond the datacenter.

“The load balancer is screwing up. The site is down.” said the customer.  From beyond the data center.

Our hero was not convinced.  Many plagues have been blamed upon the load balancer, only to later find out the culprit was elsewhere.  So he punched up the website, and sure enough, nothing. He logged into the load balancer, and found it to be operating correctly, with no changes from when it was working.  He then checked the servers.  And there was the problem.

He shot up from his haunted Aeron chair.  “By Zeus, the default gateway has been changed!”

For you see, as traffic comes into the load balancer, it must also return through the load balancer on the way out.  This is the way of things.  This can be done a number of ways, and the method chosen for this infrastructure was by making the load balancer the default gateway.  But when someone changes the default gateway to a device other than the load balancer, the packets are doomed to wander the network, never to find their destination.  They were damend to the bit bucket.

So our mad sysadmin (he was pretty mad,  as he had specifically instructed them not to change the default gateway) changed the default gateways correctly, so that more packets would not suffer the same ghastly fate.  Once this task had been completed, the packets found their way back the client, and all worked.   (The lost packets still haunt the data center to this day!)

Teh End…

or is it?

But Tony, what about this site? Aren’t I reading a page on your site being served from a web server?

It’s not actually a web server.  It’s a web server/application server/database server. This blog is a web application, and web applications require the three tiers:  Web, App, and DB.  This particular system is running the popular LAMP stack, for Linux, Apache, MySQL, and PHP.  The application this server runs (among others) is the popular WordPress blog.

So, no, it’s not a web server. Not just a web server anyway.  You get the point.

When’s the last time you went to a static web page?  Years ago, your average person would need to know some HTML in order to post pictures of their cats.  Now, a web application handles the presentation, all you need to do is supply the cat pictures (and adorable captions).

Everything is a web application.  Nothing is static anymore.  That’s part of the reason load balancers are being called “Application Delivery Controllers”.

Nothing usable on the Interet today is useful unless it’s a web application.  Blogs, news sites (powered by content management system), social media, web stores.  They’re all web applications.  So no more web servers.  Not by themselves, anyway.

This post covers network topology, which is how the load balancer fits into the network.  How a device fits into the network is usually a difficult concept to get, and often that’s simply because people make it tougher than it need be. Basically, for a load balancer to be put into a network effectively, two things need to happen.

1. Traffic needs to flow through the load balancer on the way in

1. Traffic needs to flow through the load balancer on the way out

The first part is easy, as there’s only one way.  We direct traffic to the virtual IP (VIP) and port sitting on the load balancer.  This is the IP and port that pretends to be the server.  Getting traffic through the load balancer on the way out is probably one of the toughest concepts to grasp when learning load balancers, as there are several ways to accomplish this.

There’s on method of getting traffic through the load balancer on the way out that’s a quick way to drop a load balancer into an existing infrastructure with minimal changes to the network topology.  This is called one-armed, route-path.

One-armed, route path is not as popular as some of the other methods, although it has the distinct benefit of being a good, quick “drop-in” deployment.  Here’s how it works.

Let’s say you’ve got a network with a couple of servers sitting behind a firewall.  This firewall does NAT from a public address space to private IPs. This is a pretty common scenario for a small to medium sized business.

In the example shown above, the default gateway for the servers is the firewall, at 192.168.1.1.  To network admins, The concept of a default gateway is second nature.  To server folks, keep this in mind:  If you want to send IP traffic to a system not on your local network, you need a router to handle delivery.  That is your default gateway.  Without a default gateway for your servers, you can’t communicate with the Internet.

So now lets say we want to drop a load balancer into the network.  There are several options, and for the most part the advantages of one over another are logistical, not performance related.  For example, to do two-armed, Layer 3 path (arguably the most common topology), you would need to put in a new IP network between the firewall and the servers, and one new Layer 2 network.  This would require re-addressing the IPs on all the servers.

And while adding a new Layer 2 and Layer 3 network would certainly work, we can use one-armed, Layer 3 path without the need to re-IP all the servers or adding new networks.

In the figure above, you see that we’ve changed the default gateway on the servers to that of the administrative IP of the load balancer (if there were two load balancers, they would have a floating administrative IP which you would use as the default gateway).   The default gateway of the load balancer is that of the firewall.

This seems a little odd, as we’ve got two default gateways on the same IP network.  While unusual, it works, and it’s a handy way to drop a load balancer into a network with minimal changes.

The idea is that if everything goes terribly wrong, at least your visitors will see something, instead of nothing.

Which begs the question: How do you like to fail?  Fail fast or fail slow? Would it be better to fail slow, where your site becomes slower and slower, or possibly just unresponsive, or would it be better to put up a quick-serving sorry page if the infrastructure melts?

A wildly successful website can easily become a victim of its own success.  Take the case of two sites that experienced exponential growth in a relatively short period of time:  Twitter.com and Myspace.com.

They took two different paths in the realm of failure.  One failed fast, and one failed slow.

Although Myspace has lost most of its lead to Facebook, it’s still a wildly popular social media site.  They had exponential growth from their start in 2003, and there were many periods of time when Myspace.com was just… slow.  Really really slow. You can’t really blame them.  It’s tough when users come faster than you can install servers and provision bandwidth.  It’s a happy problem to have usually, but it’s still a logistical challenge.

Twitter.com came around a bit later, but it also had exponential growth and problems coping.  But for the most part, they failed in a different way:  Fail Whale. When something went terribly awry, instead of a slow site, you’d get a very quick fail whale image.

Perhaps this is a matter of personal opinion, but I think if you’re going to fail, it’s better to fail quick than fail slow.  That is, have a sorry page or sorry site that comes up quick, rather than a site that is too slow for anyone to use.

The quick sorry page can be done with many of the load balancing/ADC vendors by using the backup/sorry serverfarm feature.  Keeping a group of reserve servers, serving up only a “oops, sorry about that” type of page, your own fail whale, can be better than having a really slow or unresponsive web site.

Of course, you won’t always be able to choose the method of your failure.  If your upstream ISP goes dark, there’s not much you can do (unless you have an offsite fail site).  But I personally think having a fail site is a more “professional” way to fail than having a slow or unresponsive site when things go belly up (and we all know they will).