For many situations, connecting from the same subnet as your real servers is not a big deal.  However, there are some situations where this is required.  One of the most common reasons I see is that one of the web application servers needs to connect to a VIP that distributes load between other servers on that same subnet.

Most server load balancing happens through NAT (Network Address Translation), with the only exception being DSR (Direct Server Return).  You have the option of two different types of NAT: Half-NAT and Source-NAT (SNAT).  In half-NAT, only the destination address is changed on the way in.  In SNAT, both the source and destination are changed.

If you’re using half-NAT, you cannot connect to a VIP from the same subnet a server resides on.  The reason for this has something to do with the 4-steps required to do server load balancing NAT.

Figure 1: Network Scenario

Take a look at the network scenario depicted in Figure 1.  In this diagram, you see a client with the IP of 10.1.1.1, a VIP on a load balancer with an IP address of 192.168.1.200, and a server with an IP address of 192.168.1.11 as well as some other devices.

Now, the NAT happens in 4-steps, regardless of whether the load balancer is operating in Layer 4 or Layer 7 mode.  Take a look at Table 1 and it’s companion figure, Figure 2.

Table 1: Half-NAT

Figure 2: Half-NAT Path

Because NAT is done on the way in and on the way out, the load balancer needs to be in the path of traffic on the way and on the way out.  With half-NAT, this is done by either being in the Layer 2 path of traffic, or somewhat more commonly, the load balancer is the default gateway.

Now look what happens when we try to connect from the client PC on the same network as the servers.

Note that only three-steps occured.  This is because the server responds directly to the client.  Since everything is on the same Layer 3 network, there’s no reason to go through a default gateway.  The critical 4th step doesn’t occur, so the source address for the server response to the client is invalid.  The client sent a connection to 192.168.1.200, and it got a response back from 192.168.1.11.  When that happens, the client’s IP stack correctly drops all the reponses.

One solution is to do SNAT.  By NATing the source and destination addresses simultaneously, ensure that traffic goes through the load balancer on the way in and on the way out.  Observe what happens when we do Full-NAT in Table 3.  You’ll note we’ve added a new IP address 192.168.1.5, the SNAT address (this can also be a pool of multiple IP addresses).

SNAT makes it possible to connect to the VIP from the same subnet that the servers are on.  But there’s one little problem:  The true source IP address is now hidden from the servers, so the server logs would show all connections as originating from 192.168.1.5.  Many web sites count on the true source IP address of the client showing up in the logs in order to munge the logs.

So you’re caught between a rock and a hard place.  On one had, you have the true source preserved with half-NAT, but you can’t connect to the VIP from the same subnet as the servers.  On the other hand, you can connect to the VIP from the same subnet, but the true source is hidden.

There is a third option if you’re using HTTP or HTTPS.  When you SNAT you can also insert the real source IP address as an HTTP header in the request.  The server, if configured, can then record the HTTP header in its log instead of the Layer 3 source address.  However, this requires configuring both the load balancer and all the servers.  For Apache, it’s a one line config change.  For IIS, it requires an ISAPI filter (such as this one from F5).

Selective SNAT

Instead of deciding between SNAT and half-NAT, some vendors have the ability to use both on a VIP, choosing which based on the incoming source IP address.  A couple of vendors offer this option, but since I happen to have an A10 Networks AX2200 from a recent review, so I’ll use that as an example.

The first step is to build an access list that matches the network that you want to be SNAT’d. This would normally be the subnet that your servers reside on, although there are situations where it would make sense to add in a few more subnets.

A10 uses the standard Cisco IOS-style ACL (including inverse bitmask: 0.0.0.255)  We’re just using it to match the server’s network, 192.168.1.0/24.

Then, in the configuration for the TCP/UDP port of the Virtual Server, we associate ACL 1 with an SNAT pool.

Now, any connection to the VIP originating from the server subnet gets SNAT’d, while everyone else gets half-NAT’d.  Allowing the same subnet to connect while preserving the source IP address for everyone else.  It’s the best of both worlds.

Typically, Apache Bench (ab) is installed with the base Apache install, from at least Apache 1.3 on.  This includes when Apache is installed on Windows.

You can check all of the available options on the ab documentation page, but here’s a (very) quick reference to using it.

Two of the most important options are “-n” for the number of total connections, and “-c” for how many concurrent connections are done at the same time.

For instance, using the option “-n 1000″ will do 1,000 requests, one at a time, to a target URL.

ab -n 1000 http://website.com/

One at a time is rarely an effective test, so it’s best to use the “-c” option to specify a high number of concurrent connections, such as 100.

ab -n 1000 -c 100 http://website.com/

If you use concurrency, ab will split the total number of requests up amongst the concurrent settings.  For instance, using the option “-n 1000″ will do 1,000 connections, but “-n 2000 -c 100″ will only do 20 requests from 100 different connections (2,000 / 100 = 20).  So it’s best to use a much larger number of total connections if you’re doing concurrency.

ab -n 100000 -c 100 http://website.com/

When ab is finished running, it will spit out a performance report, including such info as the time taken for tests, requests per second, wait time, etc.

Finished 1000 requests

Server Software:        Apache/2.2.9
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        45 bytes

Concurrency Level:      10
Time taken for tests:   0.427 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      320640 bytes
HTML transferred:       45090 bytes
Requests per second:    2341.45 [#/sec] (mean)
Time per request:       4.271 [ms] (mean)
Time per request:       0.427 [ms] (mean, across all concurrent requests)
Transfer rate:          733.17 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   0.4      2       3
Processing:     0    2   0.5      2       7
Waiting:        0    2   0.5      2       6
Total:          0    4   0.7      4       8

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      5
  80%      5
  90%      5
  95%      5
  98%      5
  99%      5
 100%      8 (longest request)

The ab utility defaults to one request per TCP connection (KeepAlive turned off).  If you want to use KeepAlive, where multiple requests are made through a TCP connection, use the “-K” option, open up as many TCP connections as you specify in concurrency (“-c”) and make the total number of quests through those few open TCP connections.

The utility is a simple but power tool for testing load balancers and web servers.  It doesn’t tend to reflect real-world usage, but it can be useful for baseline testing and troubleshooting.  I’ve found it quite useful over the years.

The driving force behind Layer 7 persistence (keeping an individual user tied to a specific server in a server group based on HTTP headers instead of IP address) was the dreaded AOL Megaproxy issue.  AOL had the nasty little tendancy of routing all web traffic through a couple of mega proxies located throughout the US and Canada.

This caused a problem with the previous method of persistence, which was to base it on source IP address. Typically, one IP address equaled a single user.  However, with AOL, you could have 20,000 users coming from a single IP address.  The load balancer would think it’s a single user, and if you had 300 servers ready to take orders, all 20,000 users would go to one.  That situation has happened a few times, and it’s hillarious, so long as you aren’t the company with the 300 servers.

I still teach that mega proxy problem, mostly out of muscle memory.  But I stopped to think about it, do we really have a problem with megaproxies anymore?  Does AOL even do this practice, and even if they did, is AOL represent a significant amount of traffic?

The answer to the later question is almost certainly no.  AOL has seen a dramatic drop in subscribers, and most people connect directly to the Internet through their cable modem or DSL provider.  And I don’t know of any major Internet provider that utilizes proxies for their users Internet requests.

Layer 7 persistence is still applicable to situations where you may have multiple users coming from a single IP address (such as a small client base coming from a handful of offices, with each office using on public IP address), but I wonder what doing Layer 4 persistence would do to a major site these days.  I’m thinking, not much.

What do you think?

So what’s a part-time regex’r to do? Like any good professional, I cheat. I use a great little tool called Regex Coach.

It’s great little graphical tool that lets you take an example string and test (in real time) your regex statement.

It handles splits, fields, and all of the other regex madness that you might need help with.

Update: Reader Jason Williams pointed out another great little Regex tool called Kodos.  Similar in function, it’s got some neat features like automatic code generation.

Number 1: Get Your Sniffing In Order

Take some time to prepare your infrastructure to make sure you can sniff the points where traffic enters and leaves your load balancing infrastructure. If you’re lucky, your load balancer has the tcpdump utility built in. If that’s the case, you’re done. If not, you’ll need a way to sniff traffic on all LANs that your load balancer operates on.

Without TCPDump on your load balancer, you’ll need to setup some type of span/mirror port on your switching infrastructure, or put a hub near the ingress/egress point. This can be somewhat of a pain to setup, which is why it’s best to do it when there isn’t a problem that needs diagnosing.

What works great is some type of Unix or Windows box that you can log into remotely with a spare Gigabit Ethernet port. Have a couple of ports on you Layer 2 infrastructure switches ready to be turned into span/mirror ports, and all you need to do is plug the spare Ethernet port to do the sniffing. You can of course move the cable around to different span ports as needed.

If you’re really in a pinch, you can plug a hub in between the load balancer and the servers or the load balancer and the Internet and sniff traffic that way, although you’ll probably degrade network performance some with the hub (and thus, collisionable) network.

Number 2: Run MRTG/RRDTool

Management guru Peter Drucker said “What gets measured, gets managed”, and that’s certainly true for networks. Installing MRTG/RRDTool on your network (if you haven’t already) and pulling stats from your load balancer will definitely help you manage your infrastructure.

Getting Interface bandwidth stats is trivial with any load balancer. Most support extended objects, which will get you even more detailed metrics on your load balancing infrastructure.

Number 3: Check For Ethernet Errors

It’s always a good idea to check your Ethernet interfaces for errors. If you’ve got a managed switch, such as a Cisco Catalyst, the command is something along the lines of “show port counters [port]“.

Port  Align-Err  FCS-Err    Xmit-Err   Rcv-Err    UnderSize

----- ---------- ---------- ---------- ---------- ---------
  3/2       1511        213          0          0         0

Port  Single-Col Multi-Coll Late-Coll  Excess-Col Carri-Sen Runts     Giants

----- ---------- ---------- ---------- ---------- --------- --------- ---------

 3/2           0          0          0          0         0      5511         0

On a Linux, FreeBSD, or other Unix-type system, “ifconfig -a” usually does it.

RX packets:157721935 errors:5869 dropped:6008 overruns:5869 frame:0
TX packets:172114172 errors:0 dropped:0 overruns:15 carrier:0
collisions:0 txqueuelen:1000

In both cases, there are errors on the interface. These types of errors can be caused by a couple of things, (including faulty wiring), but the usual suspect is a duplex mismatch.

Duplex mismatches are insidious because when they occur, you can still pass traffic. With light traffic, you won’t even notice a difference. With moderate to heavy traffic, things will get slow, connections dropped, but you’ll still pass traffic. Other failures completely block traffic, but a duplex mis-match isn’t always obvious, and it’ll make traffic crawl.

100 Mbps duplex autodetect doesn’t really work that well. With Gigabit, the protocol is much more refined, and I’ve yet to see a duplex mismatch even on auto when Gigabit is used. But with 100 Mbps, you can’t really trust auto-detect.

The solution: Always hard-code 100 Mbps links, and check your interfaces occasionally for errors.

Number 4: Check For Software Updates

Even if there’s no immediate need to update your code, it’s a good idea to keep current. It’s a lot easier to schedule and update code in regular intervals than it is to find yourself in a situation where you need to be on the latest code, and you’re several major versions back. So it’s a good idea to schedule code updates on a regular basis.

The swiss army knife of diagnostic tools, and the one most essential in the entire networking realm, is tcpdump. Most load balancers these days have this installed, from the big-budget F5 BIG-IP to the value boxes, tcpdump is included.

It’s one of the benefits of the proliferation of x86-based load balancers, and an advantage they have over some of their switch/ASIC-based counterparts.  While it’s possible for the switch-based system to just set up a mirror/span port and use a sniffer (or Linux box with tcpdump installed), it’s far more convenient when tcpdump is installed on the device itself.

If the virtual server is setup as “transparent”, where the client IP address is preserved (i.e., the load balancer doesn’t act like proxy), then the clients cannot respond correctly.

The load balancer performs network address translation on the way in, and on the way out. This is necessary for the connection to work correctly. However, if the client is on the same subnet as the real server, the server will respond to the client directly, without hitting the load balancer on the way out. The source IP address will appear to be the server, and not the virtual service. Because the client sent the connection to the virtual service, it will drop all incoming packets from the server.

The client can be on the same subnet as the virtual service and browsing will work, provided that the virtual service is on a separate subnet from the real servers.

There are a few solutions for this:

The first is to disable transparency, and have the load balancer act like a proxy. The downside to this is that you lose the source IP address for your server logs (all connections appear to be coming from the load balancer).

Another solution is to use DSR (Direct Server Return). The downside to this is that you cannot do cookie persistence, or any other fancy Layer 7 features.

Depending on your situation, a compromise might be to setup two virtual IPs, each with identical real servers sitting behind them.  One would be for the public, which would preserve the IP address, and the other would be for client systems that sit on the same subnet, and that one would be set for non-transparent/proxy mode.

Steve Lothspeich was kind enough to post a fix for the DST problem.