One of the most common issues that comes up in load balancing is summarized in the comment below that showed up in my post: “What Application Developers Need To Know About Load Balancing“.
Hi,
I have two web servers behind a Cisco 11503 device and I wanted to record the clients source IP when they browse my site. However, my challenage is that I am not able to detect the clients source IP because the LB is replacing it with the VIP.
Do you have any idea how I can capture this? Also what techniques are being used to capture this information?
Thank You,
Dharmesh
From the perspective of the server, transparent load balancing will show the true source IP address of the client, and non-transparent (which is what he’s described) does not. In this case, the CSS is replacing both the source and destination IP addresses when it performs load balancing, so the server logs see all connections as coming from the CSS, so it’s non-transparent.
The frustrating part about finding out about this is that these source IP addresses for previous web connections are lost forever (unless you’ve got some time of 24/7 IP sniffing going on).
For the Cisco CSS, non-transparency is the only option if you’re using a one-armed connection, where the Cisco CSS is plugged into the network infrastructure on only one port, and the virtual server IPs and the real server IPs are on the same subnet. (Other load balancers do not have that particular limitation, and can do transparent and non-transparent in one-armed scenarios.)
I don’t have access to a CSS, but I believe there are two possible solutions to getting the true source in your logs. The first is to plug the servers directly into the CSS unit. Because the Cisco CSS is a Layer 2 switch, it doesn’t need to be the default gateway for outbound connections so long as it’s in the Layer 2 path.
A second solution would be to split the subnets of the real servers and the virtual services.  With two subnets, you should get true source.
Another solution may be to set the service up as Layer 4-only. I’m not sure if that’ll work though (again, I don’t have a Cisco CSS on hand).
If you want to try those solutions, be sure to test thoroughly in a controlled environment. In implementing any of the above, you’re bound to hit a bit of downtime.
Another solution would be to tell the load balancer to insert a “X-Forwarded-For” header, which would contain the actual IP address of the client, even in a non-transparent situation. You would use a simple Apache configuration to log the X-Forwarded-For value for the source IP, or you would use an ISAPI filter for IIS (such as this one from F5, but it should work for any vendor). Unfortunately, I do not believe that Cisco CSS supports this header.
The X-Forwarded-For header is by far the best option to use. All of your other suggestions require very specific configuration changes and layer 2 and layer 3 which may be impossible for political reasons if the questioner is not in his networking group, and may be impossible for technical reasons if introducing the load balancer into the path of layer 2 or layer 3 traffic would cause it to tip over due to excessive non-VIP based traffic that it sees. This might be worked around with multiple VLANs per host for VIP and non-VIP traffic, but that complicates the overall networking design and introduces the possibility of errors when non-VIP trafffic is accidentally configured to go through the VIPs (having tried this on the scale of hundreds of VIPs and thousands of servers, I can assure you this is a huge configuration risk, and the more complicated solutions do not scale in terms of configuration management).
I’ve also recently run into issues with layer 3 half-NAT and router interfaces on load balancers leading to networking engineers being unable to configure router interfaces correctly. As a result I hate the idea of running layer 3 through load balancer devices — keeping the router interfaces on the routers is by far the best idea from a network management perspective.
And the best solution to the source IP problem is to solve a layer 7 (the app not getting source IPs) with a layer 7 solution — which is X-Forwarded-For headers.
Unfortunately, I believe you may be correct about the 11503 — in which case I’d suggest pitching that particular piece of equipment in a trash bin (or suitable environmentally-friendly recycling center). The CSS switches are now 2 generations behind (and no longer supported by cisco?) and you should be able to find better switches on ebay (not to mention anything bought new; where you’ll also get a support contract) which support X-Forwarded-For header injection.