One issue that trips up people when diagnosing load balancer problems is they see a connection has been made to a virtual service, so it looks like they’re getting through to the real server.
That’s not always the case. Depending on how your load balancer is configured, it could simply mean you’ve made a connection to the load balancer’s proxy server.
Modern load balancers have actually two different types of virtual services: Layer 4, and Layer 7. Clients can’t really tell the difference, but under the hood, they’re substantially different in the way they operate.
A Layer 4 virtual service on a load balancer operates much like a router. It’s just re-writing source and/or destination addresses. Not much more than you’re basic broadband wireless router.
Layer 7 code is a type of application proxy, aware of HTTP and a few other protocols depending upon the vendor (such as FTP, SIPS). The Layer 7 code is what handles cookie persistence. When a connection is made to a Layer 7 virtual service and a request is sent, a separate TCP connection is opened to the server, and the request is forwarded. Because of the way Layer 7 operations occur, the load balancer *can’t* send the request to a server until it sees the request, because the request (and items in the header) will tell the load balancer where it’s sending the request.
The tricky part comes when there’s problem with connectivity between the load balancer and the servers. If the servers are unresponsive, you will still get a connection to the virtual service. This can trick you into thinking there’s something wrong with the servers, when the issue may be elsewhere.
Once this connection is made, what happens next is fairly vendor specific. Once you’re connected and make a request, the connection can remain open and stall. On other vendors, the connection will terminate the second you make a request, with no HTTP error code.
Even within the same vendor, different versions will react differently. In BIG-IP V4, you can open a connection and make a request, and you’ll see no response. Eventually the connection will terminate, but you will probably wait a while. In BIG-IP V9 however, when you open a connection and send a request, you’re immediately sent a TCP reset packet.
If you’ve setup SSL termination, which would fall into the Layer 7 camp, you’ll get non-HTTP encoded error (such as read:errno=104, which won’t show up in the browser, but will show up in a raw TCP connection) when connecting and sending a request to a virtual service with no active real servers.
So it’s important to know how your virtual service is configured when you’re troubleshooting an issue.
