Load Balancing Digest

One of the most common issues that comes up is something often referred to as the “same subnet” problem.  In certain configurations, you cannot connect to a VIP (Virtual IP address, or Virtual Server) if the client you’re connecting from is on the same subnet as the real server.

For many situations, connecting from the same subnet as your real servers is not a big deal.  However, there are some situations where this is required.  One of the most common reasons I see is that one of the web application servers needs to connect to a VIP that distributes load between other servers on that same subnet.

Most server load balancing happens through NAT (Network Address Translation), with the only exception being DSR (Direct Server Return).  You have the option of two different types of NAT: Half-NAT and Source-NAT (SNAT).  In half-NAT, only the destination address is changed on the way in.  In SNAT, both the source and destination are changed.

If you’re using half-NAT, you cannot connect to a VIP from the same subnet a server resides on.  The reason for this has something to do with the 4-steps required to do server load balancing NAT.

Figure 1: Network Scenario

Take a look at the network scenario depicted in Figure 1.  In this diagram, you see a client with the IP of 10.1.1.1, a VIP on a load balancer with an IP address of 192.168.1.200, and a server with an IP address of 192.168.1.11 as well as some other devices.

Now, the NAT happens in 4-steps, regardless of whether the load balancer is operating in Layer 4 or Layer 7 mode.  Take a look at Table 1 and it’s companion figure, Figure 2.

Table 1: Half-NAT

Figure 2: Half-NAT Path

Because NAT is done on the way in and on the way out, the load balancer needs to be in the path of traffic on the way and on the way out.  With half-NAT, this is done by either being in the Layer 2 path of traffic, or somewhat more commonly, the load balancer is the default gateway.

Now look what happens when we try to connect from the client PC on the same network as the servers.

Note that only three-steps occured.  This is because the server responds directly to the client.  Since everything is on the same Layer 3 network, there’s no reason to go through a default gateway.  The critical 4th step doesn’t occur, so the source address for the server response to the client is invalid.  The client sent a connection to 192.168.1.200, and it got a response back from 192.168.1.11.  When that happens, the client’s IP stack correctly drops all the reponses.

One solution is to do SNAT.  By NATing the source and destination addresses simultaneously, ensure that traffic goes through the load balancer on the way in and on the way out.  Observe what happens when we do Full-NAT in Table 3.  You’ll note we’ve added a new IP address 192.168.1.5, the SNAT address (this can also be a pool of multiple IP addresses).

SNAT makes it possible to connect to the VIP from the same subnet that the servers are on.  But there’s one little problem:  The true source IP address is now hidden from the servers, so the server logs would show all connections as originating from 192.168.1.5.  Many web sites count on the true source IP address of the client showing up in the logs in order to munge the logs.

So you’re caught between a rock and a hard place.  On one had, you have the true source preserved with half-NAT, but you can’t connect to the VIP from the same subnet as the servers.  On the other hand, you can connect to the VIP from the same subnet, but the true source is hidden.

There is a third option if you’re using HTTP or HTTPS.  When you SNAT you can also insert the real source IP address as an HTTP header in the request.  The server, if configured, can then record the HTTP header in its log instead of the Layer 3 source address.  However, this requires configuring both the load balancer and all the servers.  For Apache, it’s a one line config change.  For IIS, it requires an ISAPI filter (such as this one from F5).

Selective SNAT

Instead of deciding between SNAT and half-NAT, some vendors have the ability to use both on a VIP, choosing which based on the incoming source IP address.  A couple of vendors offer this option, but since I happen to have an A10 Networks AX2200 from a recent review, so I’ll use that as an example.

The first step is to build an access list that matches the network that you want to be SNAT’d. This would normally be the subnet that your servers reside on, although there are situations where it would make sense to add in a few more subnets.

A10 uses the standard Cisco IOS-style ACL (including inverse bitmask: 0.0.0.255)  We’re just using it to match the server’s network, 192.168.1.0/24.

Then, in the configuration for the TCP/UDP port of the Virtual Server, we associate ACL 1 with an SNAT pool.

Now, any connection to the VIP originating from the server subnet gets SNAT’d, while everyone else gets half-NAT’d.  Allowing the same subnet to connect while preserving the source IP address for everyone else.  It’s the best of both worlds.