Building a Highly-Available Load Balancer with Nginx and Keepalived on CentOS

In this post I will show how to build a highly-available load balancer with Nginx and keepalived. There are issues running keepalived on KVM VMs (multicast over the bridged interface) so I suggest you don’t do that. Here, we’re running on physical nodes, but VMware machines work fine too. The end result will be a high performance and scalable load balancing solution which can be further extended (for example, to add SSL support).

First, a diagram indicating the proposed topology. All hosts are running CentOS 6.5 x86_64.

nginx_load_balancerAs you can see, there are four hosts. lb01 and lb02 will be running Nginx and keepalived and will form the highly-available load balancer. app01 and app02 will be simply running an Apache webserver for the purposes of this demonstration. www01 is the failover virtual IP address that will be used for accessing the web application on port 80. My local domain name is .local.

Backend Server Configuration

First off, we need to configure the basic Apache servers on app01 and app02. These servers will serve a simple HTML page allowing us to verify which backend our load balancer is accessing. If you already have backend servers serving up your web application, you can skip these steps.

Start by installing Apache.

Configure a virtual host on each server. The following configuration is for app01 – substitute accordingly for app02.

Create the document root:

Echo the hostname into index.html in each document root:

Start Apache and set to start automatically in the appropriate runlevels:

Browse to app01.local and app02.local and verify that the hostname is served.

Next, we need to reconfigure Apache so that the logging will reflect the IP address of the user, rather than that of the load balancer. We will do that by inserting a header (X-Forwarded-For) in the next section, but for now, create another LogFormat format directive in httpd.conf as follows:

Modify the CustomLog directive in 00-default.conf as follows:

to use the forwarded LogFormat instead of combined. As you can see, this logs the contents of the X-Forwarded-For header (as defined by %{X-Forwarded-For}i) instead of the remote host (%h).

Once the log formatting has been updated on both nodes, reload the Apache configuration:

We’re now ready to move onto the load balancer configuration.

Load Balancer Configuration

All steps should be performed on both lb01 and lb02 unless otherwise indicated. Before we start, add the following iptables rules (if you’re using iptables). This presumes you want to prepend the rules to the start of your ruleset:

These rules are required so that multicast and VRRP (and thus keepalived) will work correctly. Be sure to write this configuration out (depending on your OS):

To allow Nginx to bind to the non-local shared IP, update the net.ipv4.ip_nonlocal_bind kernel parameter and reload the parameters with sysctl -p:

Install the latest EPEL release from an appropriate mirror:

Start by installing the nginx and keepalived packages, and their dependencies, with yum:

Next, begin the Nginx configuration. Open nginx.conf and add the following to the http block:

Here we set two headers for proxied traffic (including the X-Forwarded-For header that our Apache configuration will log), and define the actual upstream proxy itself – which by default will use a round-robin load balancing algorithm.

Next, create a file containing the proxy configuration. The EPEL release of Nginx is configured to include any configuration files under /etc/nginx/conf.d/*.conf. I created proxy.conf as follows:

And finally remove the default server configuration file default.conf:

With that, we can start Nginx on both nodes, and configure it to start automatically in the appropriate runlevels:

Now we can move onto keepalived configuration. Modify /etc/keepalived/keepalived.conf as follows:

Ensure that the priority is set to 101 on the master, 100 on the slave. Here you can see that we do a simple check with killall -0 which is less expensive than pidof to verify that nginx is running.

Start and enable keepalived:

Our load balancer is now complete! You will note on the master server that the virtual IP address is active:

Whereas on the slave it is not:

Shutdown Nginx on the master (or shutdown the node), and you should see the virtual IP address failover. Start it back up, and the virtual IP address should fail back to the master.

Health Checks

As per, the reverse proxy implementation in nginx includes in-band (or passive) server health checks. If the response from a particular server fails with an error, Nginx will mark this server as failed, and will try to avoid selecting this server for subsequent inbound requests for a while (configurable). For more elaborate health checks, the commercially supported and paid version of Nginx (Nginx Plus) is required (see for details).