• Telnet (Layer 4 TCP testing utility, Layer 7 interface)
• OpenSSL (When Layer 4 is wrapped with encryption)
• HTTP Analyzers (several to choose from, to look at headers)

Telnet is particularly handy, as I can test through Layer 7 (especially if I type “GET /” and hit return twice).  Ping is pretty useless, as is traceroute, but Telnet works wonders.

I’ve documented the tools and how I use them in the latest addition to the brand new lbwiki.com:

Troubleshooting Tools

LBWiki: Troubleshooting Tools

Telnet, OpenSSL, and an HTTP analyzer.  Probably 90% of all the issues I come up against can be solved using those.   Probably 60% of all the problems can be solved by just telnet (used to test Layer 4 and sometimes through Layer 7).

That’s not always the case. Depending on how your load balancer is configured, it could simply mean you’ve made a connection to the load balancer’s proxy server.

Modern load balancers have actually two different types of virtual services: Layer 4, and Layer 7. Clients can’t really tell the difference, but under the hood, they’re substantially different in the way they operate.

A Layer 4 virtual service on a load balancer operates much like a router. It’s just re-writing source and/or destination addresses. Not much more than you’re basic broadband wireless router.

Layer 7 code is a type of application proxy, aware of HTTP and a few other protocols depending upon the vendor (such as FTP, SIPS). The Layer 7 code is what handles cookie persistence. When a connection is made to a Layer 7 virtual service and a request is sent, a separate TCP connection is opened to the server, and the request is forwarded. Because of the way Layer 7 operations occur, the load balancer *can’t* send the request to a server until it sees the request, because the request (and items in the header) will tell the load balancer where it’s sending the request.

The tricky part comes when there’s problem with connectivity between the load balancer and the servers. If the servers are unresponsive, you will still get a connection to the virtual service. This can trick you into thinking there’s something wrong with the servers, when the issue may be elsewhere.

Once this connection is made, what happens next is fairly vendor specific. Once you’re connected and make a request, the connection can remain open and stall. On other vendors, the connection will terminate the second you make a request, with no HTTP error code.

Even within the same vendor, different versions will react differently. In BIG-IP V4, you can open a connection and make a request, and you’ll see no response.  Eventually the connection will terminate, but you will probably wait a while.  In BIG-IP V9 however, when you open a connection and send a request, you’re immediately sent a TCP reset packet.

If you’ve setup SSL termination, which would fall into the Layer 7 camp, you’ll get non-HTTP encoded error (such as read:errno=104, which won’t show up in the browser, but will show up in a raw TCP connection) when connecting and sending a request to a virtual service with no active real servers.

So it’s important to know how your virtual service is configured when you’re troubleshooting an issue.

So what if you’re using SSL? Fortunately, there’s a utility out there that works pretty much the same. It’s part of the freeware OpenSSL package, which is included with just about every Linux, Solaris, BSD, Unix system around, and a Windows version is pretty easy to get.

To specific utility is openssl s_client -host [ip address or hostname] -port [tcp port]. To use, do the following on a command line prompt:

openssl s_client -host 192.168.0.200 -port 443

This will attempt an SSL/TLS connection to the host 192.168.0.200 on port 443. Initially, this spits out a bunch of debugging on the specifics of the connection, including the type of certificate, full certificate chain, cypher, etc.

....

Protocol : TLSv1
Cipher : DHE-RSA-AES256-SHA
Session-ID: 07764E503260A11EBF0D690
Session-ID-ctx:
Master-Key: B108C37807C831EC1895324ED83FDFD4A8DDA0
Key-Arg : None
Krb5 Principal: None
Start Time: 1197668809
Timeout : 300 (sec)
Verify return code: 18 (self signed certificate)
---

You’re then connected to the other end as if you’d used telnet. You can give the other end whatever HTTP requests (or other protocols) you’d like. This utility works not only with HTTPS, but any other SSL/TLS wrapped protocol, including SMTPS, IMAPS, etc.

Enjoy.