I’ll give you a minute because I assume if you were drinking a beverage, it’s now all over your computer screen.

How much damage could someone do? Take Google for example. This means they could set up a fake Gmail-looking server, and collect the username and password of a user. The user might not ever realize that the site was fake, and their passwords were compromised.

So now there are totally legit-looking certificates out there. Your browser, and every other browser in the entire world trusts them.

Remember, SSL gives us two things: Privacy and trust. Privacy comes though symmetric encryption, and trust is done through signed certificate chains.

Trust needs to start somewhere.  With your browser, whether its Firefox, Safari, IE, Chrome, Opera, whatever, they all come with pretty much the same set of root certificate that act as the start of trust.  Essentially, they come trusting several sources.

So what happens if this trust is broken, or if the certificate was issued under false pretenses? There needs to be a way to revoke that trust on a certificate by certificate basis. There’s only two ways to do this: A manually updated CRLs (certificate revocation list), or through the OCSP protocol.

CRLs aren’t a terribly good way to handle it. A CRL is simply of list of certificates that would validate the chain of trust through the regular way (Picard -> LaForge -> Ensign Tony), but aren’t trusted anymore. Each of your browsers have their own CRLs, and they can be updated by an OS or browser patch. This relies on you or your organization reliably updating software and/or OS, which doesn’t always happen in a timely manner. Even if it’s timely, there’s always a period of time between when the certificate is revoked and when you get that revoked certificate added to your CRL.  This could be hours, days, or even weeks where your browser would trust an otherwise bogus certificate.  This just doesn’t scale.

OCSP is a better approach, as it can check with a certificate authority every time it hits a website with an SSL certificate. So not only does the browser do the usually trust check, it double checks by checking with the source (an OCSP server hosted by the certificate authority) that the trust is still valid.

If my browser ran OCSP, I don’t have to worry that I might miss a revoked certificate because I haven’t updated my browser or OS.  It can also check every time, so if a cert has been revoked, my browser finds out right away.

So while OCSP is a better approach to CRLs, it isn’t used universally. And it isn’t “fail closed” by default on some browsers, as shown in this table from David Holmes at F5.

So right now, OCSP and CRLs are no guarantee that you can trust a certificate unless you use Firefox or Chrome.

Wait, what? No guarantee? Shit.

This is what happens when people don’t trust each other

There are some opinions that certificates should be handled differently than they are now. He discusses some interesting ideas, but I think it’s fair to say PKI (public-key infrastructure) needs a bit of overhaul.

The best known product for traffic manipulation is likely F5′s legendary iRules, but other vendors have similar capability such as A10′s aFlex.   Essentially, this puts an application development platform. Typically this is done with a standard programming language, such as a modified TCL for iRules and aFlex.

Some vendors, (Cisco I’m looking at you, as well as Brocade but I’m less familiar with them lately) lack the ability to manipulate traffic using application logic.

For vendors, this is a great feature to have. Its attractive to potential customers, and it makes it difficult to move to a platform that doesn’t have this feature. I call it the Charlton Heston feature, since you’ll only pull it out of a client’s cold dead hands.  Once you use it, you’re fairly dependent on it.

Get your paws off me, you damned dirty network admins!

In general, I’m a fan of iRules and their ilk. There’s just too many situations where the ability to manipulate HTTP content has saved the day.

So what kind of manipulation can you do?  There are rules to scrub credit card numbers, so if a web application tries to display a credit card number such as “5123-1234-1234-1234″ (a big no-no according to PCI-DSS), the load balancer would do a regular expression search for that pattern, and replace it with “XXX-XXX-XXXX-1234″ before sending the response off to the client.

With a programming language and regular expressions, just about anything is possible. And there in lies a problem.

Two Edged Sword

Having application logic on the load balancer is a double-edged sword.  One the one hand, it allows you to have very granular control over headers and content for requests and responses.  The drawback is it allows you to have very granular control over headers and content for requests and responses.

The awesomeness of this manipulation is evident, but there are some caveats.

We’ll fix it in post

There’s a running joke in the film industry called “we’ll fix it in post”.  Essentially, it means who cares if we get it right here, we’ll just fix it with special effects or editing.  Except it hardly ever works.  You’ve got the same hazard in something like iRules; it can be a lousy and lazy way to fix to a problem that really should be fixed in the code.

Capacity

Of course, all this regular expressioning and application logic come at a cost in terms of CPU.  The more of it you do, the lower capacity you’ll have.  A load balancer capable of serving up 4 Gbits of second of traditional Layer 4-7 traffic may have its capacity dropped to 500 Mbps.  And there’s no way of telling what the new performance ceiling would be (until you hit it like Wile E Coyote on a rocket sled).

This is less of an issue than it used to be, as Moore’s law has made processors faster and faster, allowing load balancers to handle increasingly difficult tasks, while bandwidth requirements have not increased nearly as dramatically. Not many organizations have more than a couple hundred megabits per second to the Internet, while load balancers are capable of handling several (even dozens) of Gigabits.

Latency

Latency is a dirty word in networks and applications, and certainly a potential issue with application logic.  If you’re going to look at a cookie, the latency imposed by a load balancer will be minimal.  If you’re going to calculate Pi to the 100th digit on every HTTP request, that’s going to add a certain amount of latency to the transaction.  Like the capacity ceiling, it’s very difficult to predict what that additional latency will be when adding more and more logic. And depending on the load profile, that latency may vary quite a bit over time.

Creep

Capacity and latency are easy enough issues to deal with, but creep is very problematic.  It’s a “Layer 8″ problem, and one that can quickly spiral out of control.

Typically the creep comes into play when you first implement application logic, and it fixes a vexing problem. You’re the hero, and someone perks their ears up and says “wow, what else can you do with it?”

“Anything”, you say confidently.  And anything is what they throw at you. Add semi-colons at the end of paragraphs, replace all double-spaces after a period with single spaces, etc.

It can very easily spiral out of control, so you need to know when to put your foot down (hint: early).

Skills

To develop this application logic, you need to understand HTTP really well. When developing most web applications on a platform like ASP or PHP, you don’t really need to understand HTTP all that well.  But the load balancer manipulates on the HTTP level, so you’ll really need to be up on your HTTP. You also need to have the programming skills to pay the bills. Especially for a network admin, those programming muscles might not get flexed on a regular basis.

I’ve also heard the argument that iRules and their ilk are only there to fix problems that should have been fixed in the code. And there’s a lot of truth in that; they are typically used in situations where the problem could also be solved at the application.

But that’s also like saying that I shouldn’t need a lock on my door, because people should not be jerks and go around stealing stuff.

In IT, there are always going to be situations where its either easier to fix the problem on the load balancer, or it’s the only viable solution (code locked down, client doesn’t control the code as its a third party, developer ran away to Brazil with their secret family and won’t return calls).

Bottom line? I’d rather have it than not have it, but I’m very careful with it. After all, with great power comes great potential for epic fail.

I’m a big fan of virtualization.  There’s a lot to like about it, including consolidation (getting rid of space-heater servers that do nothing running 1% CPU and sucking up electricity and throwing off heat), flexibility, and management.  I’ve even gone and got my VCP4 (VMware Ceritified Professional 4) certification.  (Hear that ladies? I’m certified.)

One aspect of virtualization I’m a fan of is the appliances.  Vendors are taking physical appliances (such as a Vyatta router) and turning it into a VM appliance.  I don’t have to worry about an underlying operating system (and the requisite patches), the appliance vendor handles the software and the OS.

Several load balancing vendors have gotten into that virtualization game.  Vendors that have traditionally offered hardware appliances now have virtual appliances (some for years).  From From F5 to KEMP, from Coyote Point to loadbalancer.org, there are a number of virtual load balancers/ADCs to choose from.   And for the most part, they offer the same features as their hardware brethren.

Their throughput and performance is hampered somewhat by the fact that they’re all software and no silicon. Everything is done in the x86 virtualized CPU(s).  Still, depending on how you provision them, they can generally handle several thousand HTTP requests per second even in Layer 7 mode.

The one caveat to virtual load balancers is that their SSL performance is severely limited.  Even value-market load balancers that do most of their functions in a general purpose CPU will still use SSL ASICs for the asymmetric crypto (even using the general processor for the less CPU-intensive symmetric crypto).

The first part of every new SSL connection is a very CPU-intensive asymmetric operation (about 1000x more CPU intensive than symmetric operations).    CPUs that can normally handle tens of thousand of regular TCP connections per second can only handle a few thousand SSL connections at the most.

It is technically possible to do hardware SSL acceleration on a VM load balancers however.  It requires that the virtual machine host (like VMWare ESXi) have an SSL card installed, and VM Passthrough enabled (where the virtual machine can have direct access to physical hardware).

Unfortunately, these SSL cards are tough to come by.  Cavium is probably the most notable vendor, but cards from them aren’t exactly easy to come by, and they’re fairly expensive.  You may not have the option if you’re using blade systems.   And if you you want to leverage features like HA and DRS (using ESX hosts in a cluster), then every machine in the cluster would need to have such a card.  I’m not aware of any virtual load balancer vendor that even supports this configuration.

There are a lot of situations where virtual load balancers make a lot of sense, but keep in mind that the SSL performance capability is going to be fairly constrained.

SSL performance is a tricky thing to measure.  There are a lot of different aspects to consider, and on top of that different vendors use slightly different definitions for the same terms. I was talking with a vendor the other day, and I realized as we were talking performance numbers we had different definitions for the term TPS (Transactions Per Second) and CPS (Connections Per Second).

TPS is probably one of the more ill-advised acronyms (boy do we love acronyms) for load balancers, because it’s open to debate what the “transaction” in transactions per second means.

SSL Basics

Before we talk about CPS versus TPS, let’s review some SSL basics.  In a new SSL connection, two things happen: An asymmetric exchange, then a transition to symmetric encryption.

The asymmetric part is why load balancers from F5 to KEMP use SSL accelerator chips.  It’s incredibly CPU intensive; so much so that a server that is capable of tens of thousands of connections per second of regular TCP connections is only capable of several hundred SSL connections.  To keep this from crippling the CPU, SSL accelerator processors are used to offload the cryptographic functions from the main CPU.

So when we measure SSL performance of a load balancer, we typically want to measure three things:

1: How many new SSL connections per second can a device handle, that is how many of the expensive RSA operations per second are possible.

2: How much bandwidth of SSL traffic can the device push.  This is symmetric encryption (such as AES), and is much easier on a CPU.

3: How many HTTP requests per second can the device handle after an SSL connection is established.  In HTTP 1.1, a client is allowed to make multiple HTTP requests off a single TCP/SSL connection.  This is far more efficient than the old HTTP 1.0 standard, that required a separate TCP connection for each object.  Many sites will have pages with literally hundreds of objects (I’m looking at you http://vg.no).

The trick is how many HTTP requests per SSL connection?  Personally, I’d say anywhere between 10 to 20 is a pretty good place to start, but that’s not what everyone uses.  In an epically flawed fight that o3 magazine picked with F5, author John Buswell claimed that an Nginx-based box he built could handle 25,000 TPS.

The system had no problems handling over 26,590 TPS, the test lab ran out of capacity to generate additional transactions. Compare that to the F5 Networks Big-IP 6900 which handles a maximum of 25,000 TPS but carries a starting price tag of $55,000.

The box he used had 2 quad core Opteron CPUs that were in no way capable of doing 26,000+ new SSL connections per second without an SSL accelerator card (which he didn’t use).  A few thousand 1024-bit RSA operations per second at the most, but definitely not 26,000.

He likely made the mistake of opening a couple of SSL connection, and running 25,000+ HTTP request per second off those open TCP/SSL connections (which is not even close to a real world scenario).   25,000 HTTP requests per second is within the capabilities of his box.  This is not how F5 measures TPS, nor how any other vendor measures TPS.  Of course, I’m only guessing at his methodology, as he never bothered to share his testing methodology.  (F5 and other vendors share their methodology).

TPS versus CPS

While SSL throughput is pretty easy to measure and the definition is identical for all vendors, the terms TPS and CPS aren’t.  So what do they mean?  CPS (Connections per Second) can mean a couple of things.  For SSL, it would typically mean new SSL connections, requiring the asymmetric operation.  Some vendors use TPS for this definition (the T standing for an asymmetric transaction), while other vendors use CPS for the SSL portion, and TPS for the HTTP-within-an-SSL connection.

So it’s important to understand what’s being measured.  Different vendors have different ideas of what that means, although they are honest differences of opinions (things reasonable people can disagree on).  Most vendors are reputable and will outline the methodology they used to come up with their numbers (03 magazine did not).

So to help with the common ailments of SSL, and how to resolve and/or avoid them, I give you the 3 most common SSL problems.

1: SSL Scramble

You get a call from your help desk, and they say they’re getting quite a few calls about SSL errors.  You punch up your website, and sure enough, “cannot verify” of some sort comes up on your screen.  You haven’t changed anything, so what gives?  Then you check the date on your SSL certificate.

It expired two days ago.  And thus begins the SSL shuffle, where you frantically try to get the SSL certificate provider to turn around an updated certificate as quickly as possible.

Chaos reigns supreme when your SSL certificates expire

The SSL shuffle has happened to a lot of organizations over the years, including Target.com back in July.

Most SSL shops will turn a certificate request in a day or so, but minutes seem to last quite a long time when your site is running on an expired SSL cert.

It’s not like you don’t know when the certificate is going to expire.  It says so on the certificate you get back from the certificate authority.  So why does this keep happening?  Well, most certificates expire in 1 or 2 years, and that’s about 10-20 years in IT years.  Reminder emails get filled and forgotten, or the person is on vacation, etc.  So put the expiration in you Outlook, Google or iCal calendar.

2: Domain Name Mismatch

Another common issue with SSL is the domain name mismatch.   This is actually part of a larger issue, “To WWW, or not too WWW”.  When you post links to your site, advertise it, write applications that use full domains names in the URL, do you pre-pend your domain name with “www”, or do you just use the domain name?

For example, does this site show up as “www.lbdigest.com”, or “lbdigest.com”.  If you try to punch in “www.lbdigest.com” on this site, you actually get an HTTP 301 redirect to “lbdigest.com”.   I picked non-WWW, because that’s what’s hip and cool, and I’m nothing but hip and cool.  However, it doesn’t matter which one you pick so long as you’re consistent.

See what I did there? I bounced you like a low-rider in a Dr Dre Video

Why do you need to be consistent? Well, one reason is your SSL certificate.  If you get an SSL certificate for “www.lbdigest.com”, and you link to “lbdigest.com”, you’ll get an SSL warning (unless you get a wildcard certificate).

When you apply for a typical SSL certificate, there’s a field called “common-name”, which is the fully qualified domain name (www.lbdigest.com, webmail.lbdigest.com, etc.) that you’re going to use for your site.  Make sure you pick the correct domain name, because getting a new one usually requires buying a new certificate.

3: Intermediate Certificates

This one is the most confusing for people.   Most SSL certificate authorities now require the use of intermediate SSL certificates.  These provide protection to the root CA key (perhaps the most valuable 2048 bits in the world) and also allow companies who haven’t gotten their root certificates into your browser to sell SSL certificates (by piggy-backing off of Verisign or another root CA’s cert).   I’ll now show you my world-famous SSL diagram again:

Tony, have you even talked to a girl?

So we all agree, SSL certificates are great (and Tony looks great in a Starfleet uniform), but how do we utilize them?  Well, they’re installed along side your SSL certificate in the SSL device (load balancer or web server).  They deliver your certificate along with the intermediate to the SSL client, that builds a chain in the client’s browser to the root CA.

So what’s the problem? Well, each certificate authority may have several different intermediate certificates.  And you need to pick the right one.  For instance, Verisign has about 18 to choose from. So make sure you get the right one, and test it using a validation tool from your certificate authority vendor, such as this universal one from Digicert.

Privacy comes through the use of digital encryption (RSA, AES, etc.) to keep your web interactions, such as credit card numbers, emails, passwords, confidential documents, etc., safe from prying eyes.

But having private communications with another party is all for naught if you’re talking to the wrong party.  You also need trust.  Trust that someone is who they say they are. For Internet commerce to work on a practical level, you need to able to trust that when you’re typing your username and password into your bank’s website, that you’re actually connecting to a bank, and not someone pretending to be your bank.

Trust is accomplished through the use of SSL certificates, CAs (certificate authorities), intermediate certificates, and certificate chains which combined is known as PKI (Public Key Infrastructure).   To elaborate on the use of these technologies to provide trust, I’m going to forgo the traditional Bob and Alice encryption examples, and go for something a little closer to your heart.  I’m going to drop some Star Trek on you.

Let’s say you’re in the market for a starship.  You’re looking for a sporty model with warp drive, heated seats, and most importantly, a holodeck. You go to your local Starfleet dealer, and you find this guy.

Ensign Tony.

Seriously Tony, how do you get girls to even talk to you?

The problem is, you don’t trust this guy.  It’s nothing personal, but you just don’t know him. He says he’s Ensign Tony, but you have no idea if it’s really him or not.  But there is one Starfleet officer you do know and trust implicitly, even though you never met him.  You trust Captain Jean-Luc Picard.

If there’s a problem a peace negotiation can’t solve, I haven’t met it yet

Captain Picard is the kind of guy you start out automatically trusting.  His reputation precedes him. Your browser is the same way, in that right out of the gate there are several sources (such as Verisign) that your browser trusts implicitly.

But you’re not dealing with Picard directly.  Instead, you’re dealing with Ensign Tony.  So Picard vouches for Ensign Tony, and thus a trust chain is built.   You trust Picard, and Picard trusts Ensign Tony, so by the transitive property, you can now trust Ensign Tony.

Whether it’s Internet Explorer, Firefox, Safari, Chrome, Opera, or other browsers, they come built-in trusting a number of sources.

Intermediate Certificates

One of the lesser understood concepts in the us of SSL certificates is the intermediate certificates.  These are certificates that sit between the CA (Picard) and the site certificate (Ensign Tony).

You see, Picard is an important man.  The Enterprise has over a thousand crew members and he can’t possibly personally know and trust all of them.  (In Ensign Tony’s case, there’s also the little matter of a restraining order.)  So he farms the trust out to his subordinates. And one crew member he does implicitly trust is Chief Engineer Geordi La Forge.

I have not clever caption for this image, as it is perfect.

Ensign Tony works for Geordi, and Geordi trusts Ensign Tony.   Thus Geordi becomes the intermediate certificate.  You can’t trust Ensign Tony directly through Picard because Picard can’t vouch for Tony, but Geordi can vouch fro Tony, and Picard can vouch for Geordi, so we have built a chain of trust.   This is why load balancers and web servers often require you to install an intermediate certificate.

This may be the greatest SSL diagram ever made.

Here’s what happens when you don’t install an intermediate certificate onto your load balancer/ADC/web server:

You’re 33 years old Tony, you’d think you would have made Lieutenant by now

One of the practical issues that comes up with intermediate certificates is which one do you use?  The various SSL certificate vendors such as Thawte, Digicert, and Verisign have several intermediate certificates depending on the type of certificate you purchase. Sometimes it’s not always obvious.  If you have any doubts, use one of the SSL certificate validation tools from the various vendors , including this one by Digicert.  It will tell you if the certificate chain works or not. Do not let a test from your browser determine whether your certificate works.  Browsers handle certs differently, and a validation tool will tell you if it will work with all browsers.

I’m an idiot.

I migrated from one hosting system to another.  I run regular backups of my MySQL database.

Well, apparently there was a field in one of the wordpress databases that the mysql backup application didn’t like.   So as it dumped the contents of the database, it hit this particular record and then stopped.  So the backups are frozen in time.

By the time I figured this out, I had purged all the data off my old server.

Woops.

Ensign Tony, Didn’t Anyone Tell You To Test Those Backups?

I’ll be reconstructing the old articles as best I can.  But for now, enjoy Picard’s epic facepalm.

This post covers network topology, which is how the load balancer fits into the network.  How a device fits into the network is usually a difficult concept to get, and often that’s simply because people make it tougher than it need be. Basically, for a load balancer to be put into a network effectively, two things need to happen.

1. Traffic needs to flow through the load balancer on the way in

1. Traffic needs to flow through the load balancer on the way out

The first part is easy, as there’s only one way.  We direct traffic to the virtual IP (VIP) and port sitting on the load balancer.  This is the IP and port that pretends to be the server.  Getting traffic through the load balancer on the way out is probably one of the toughest concepts to grasp when learning load balancers, as there are several ways to accomplish this.

There’s on method of getting traffic through the load balancer on the way out that’s a quick way to drop a load balancer into an existing infrastructure with minimal changes to the network topology.  This is called one-armed, route-path.

One-armed, route path is not as popular as some of the other methods, although it has the distinct benefit of being a good, quick “drop-in” deployment.  Here’s how it works.

Let’s say you’ve got a network with a couple of servers sitting behind a firewall.  This firewall does NAT from a public address space to private IPs. This is a pretty common scenario for a small to medium sized business.

In the example shown above, the default gateway for the servers is the firewall, at 192.168.1.1.  To network admins, The concept of a default gateway is second nature.  To server folks, keep this in mind:  If you want to send IP traffic to a system not on your local network, you need a router to handle delivery.  That is your default gateway.  Without a default gateway for your servers, you can’t communicate with the Internet.

So now lets say we want to drop a load balancer into the network.  There are several options, and for the most part the advantages of one over another are logistical, not performance related.  For example, to do two-armed, Layer 3 path (arguably the most common topology), you would need to put in a new IP network between the firewall and the servers, and one new Layer 2 network.  This would require re-addressing the IPs on all the servers.

And while adding a new Layer 2 and Layer 3 network would certainly work, we can use one-armed, Layer 3 path without the need to re-IP all the servers or adding new networks.

In the figure above, you see that we’ve changed the default gateway on the servers to that of the administrative IP of the load balancer (if there were two load balancers, they would have a floating administrative IP which you would use as the default gateway).   The default gateway of the load balancer is that of the firewall.

This seems a little odd, as we’ve got two default gateways on the same IP network.  While unusual, it works, and it’s a handy way to drop a load balancer into a network with minimal changes.

The idea is that if everything goes terribly wrong, at least your visitors will see something, instead of nothing.

Which begs the question: How do you like to fail?  Fail fast or fail slow? Would it be better to fail slow, where your site becomes slower and slower, or possibly just unresponsive, or would it be better to put up a quick-serving sorry page if the infrastructure melts?

A wildly successful website can easily become a victim of its own success.  Take the case of two sites that experienced exponential growth in a relatively short period of time:  Twitter.com and Myspace.com.

They took two different paths in the realm of failure.  One failed fast, and one failed slow.

Although Myspace has lost most of its lead to Facebook, it’s still a wildly popular social media site.  They had exponential growth from their start in 2003, and there were many periods of time when Myspace.com was just… slow.  Really really slow. You can’t really blame them.  It’s tough when users come faster than you can install servers and provision bandwidth.  It’s a happy problem to have usually, but it’s still a logistical challenge.

Twitter.com came around a bit later, but it also had exponential growth and problems coping.  But for the most part, they failed in a different way:  Fail Whale. When something went terribly awry, instead of a slow site, you’d get a very quick fail whale image.

Perhaps this is a matter of personal opinion, but I think if you’re going to fail, it’s better to fail quick than fail slow.  That is, have a sorry page or sorry site that comes up quick, rather than a site that is too slow for anyone to use.

The quick sorry page can be done with many of the load balancing/ADC vendors by using the backup/sorry serverfarm feature.  Keeping a group of reserve servers, serving up only a “oops, sorry about that” type of page, your own fail whale, can be better than having a really slow or unresponsive web site.

Of course, you won’t always be able to choose the method of your failure.  If your upstream ISP goes dark, there’s not much you can do (unless you have an offsite fail site).  But I personally think having a fail site is a more “professional” way to fail than having a slow or unresponsive site when things go belly up (and we all know they will).

In the networking realm, there is the concept of PDUs, or Protocol Data Units.  These are the discreet messages for a particular protocol.   Every protocol has them.  In Layer 2, the Ethernet protocol has the Ethernet Frame, and for Layer 3 the IP protocol has the IP packet, just to name a few examples.   Each of theses PDUs have their own format.

For the HTTP protocol, which operates on Layer 7, we also have a PDU: The HTTP message.  HTTP messages are further divided into two categories: HTTP requests and responses.

When your browser wants an object, such as an JPG or HTML file, it must make a properly formatted HTTP request.  The server responds with an HTTP object.  One object equals one request plus one response.  Typically the response includes the object, although sometimes the response says simply “file not found” (HTTP 404), “object moved” (HTTP 301 or 302), or even “access denied” (HTTP 400), among others.

But the thing to keep in mind is that every object, which is to say every JPG, HTML, Flash, CSS, MPEG, every individual file, will require its own HTTP request and will generate an HTTP response from the server.  A web page consisting of 10 objects will require 11 HTTP requests:  One for the HTML page itself, and 10 additional requests to pull all of the objects referenced in the HTML.

In the next post, we’ll discuss some tools you can use to troubleshoot HTTP requests and responses.