In this post I will show how to build a highly-available load balancer with Nginx and keepalived. There are issues running keepalived on KVM VMs (multicast over the bridged interface) so I suggest you don’t do that. Here, we’re running on physical nodes, but VMware machines work fine too. The end result will be a high performance and scalable load balancing solution which can be further extended (for example, to add SSL support).
First, a diagram indicating the proposed topology. All hosts are running CentOS 6.5 x86_64.
As you can see, there are four hosts. lb01 and lb02 will be running Nginx and keepalived and will form the highly-available load balancer. app01 and app02 will be simply running an Apache webserver for the purposes of this demonstration. www01 is the failover virtual IP address that will be used for accessing the web application on port 80. My local domain name is .local.
Backend Server Configuration
First off, we need to configure the basic Apache servers on app01 and app02. These servers will serve a simple HTML page allowing us to verify which backend our load balancer is accessing. If you already have backend servers serving up your web application, you can skip these steps.
Start by installing Apache.
|
1 |
# yum -y install httpd |
Configure a virtual host on each server. The following configuration is for app01 - substitute accordingly for app02.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# vi /etc/httpd/conf.d/00-default.conf NameVirtualHost *:80 <VirtualHost *:80> ServerAdmin toki@tokiwinter.com ServerName app01.local DocumentRoot /var/www/virtualhosts/app01.local/htdocs ErrorLog logs/app01.local-error_log CustomLog logs/app01.local-access_log combined <Directory /> AllowOverride all Order allow,deny Allow from all </Directory> DirectoryIndex index.html </VirtualHost> |
Create the document root:
|
1 |
# mkdir -p /var/www/virtualhosts/app01.local/htdocs |
Echo the hostname into index.html in each document root:
|
1 |
# echo "app01" > /var/www/virtualhosts/app01.local/htdocs/index.html |
Start Apache and set to start automatically in the appropriate runlevels:
|
1 2 |
# service httpd start # chkconfig httpd on |
Browse to app01.local and app02.local and verify that the hostname is served.
Next, we need to reconfigure Apache so that the logging will reflect the IP address of the user, rather than that of the load balancer. We will do that by inserting a header (X-Forwarded-For) in the next section, but for now, create another LogFormat format directive in httpd.conf as follows:
|
1 2 |
# vi /etc/httpd/conf/httpd.conf LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" forwarded |
Modify the CustomLog directive in 00-default.conf as follows:
|
1 |
CustomLog logs/app01.local-access_log forwarded |
to use the forwarded LogFormat instead of combined. As you can see, this logs the contents of the X-Forwarded-For header (as defined by %{X-Forwarded-For}i) instead of the remote host (%h).
Once the log formatting has been updated on both nodes, reload the Apache configuration:
|
1 |
# service httpd reload |
We’re now ready to move onto the load balancer configuration.
Load Balancer Configuration
All steps should be performed on both lb01 and lb02 unless otherwise indicated. Before we start, add the following iptables rules (if you’re using iptables). This presumes you want to prepend the rules to the start of your ruleset:
|
1 2 |
# iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT # iptables -I INPUT -p vrrp -j ACCEPT |
These rules are required so that multicast and VRRP (and thus keepalived) will work correctly. Be sure to write this configuration out (depending on your OS):
|
1 |
# service iptables save |
To allow Nginx to bind to the non-local shared IP, update the net.ipv4.ip_nonlocal_bind kernel parameter and reload the parameters with sysctl -p:
|
1 2 3 |
# vi /etc/sysctl.conf net.ipv4.ip_nonlocal_bind=1 # sysctl -p |
Install the latest EPEL release from an appropriate mirror:
|
1 |
# rpm -ivh http://mirror.rackcentral.com.au/epel/6/i386/epel-release-6-8.noarch.rpm |
Start by installing the nginx and keepalived packages, and their dependencies, with yum:
|
1 |
# yum -y install nginx keepalived |
Next, begin the Nginx configuration. Open nginx.conf and add the following to the http block:
|
1 2 3 4 5 6 7 |
# vi /etc/nginx/nginx.conf proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; upstream wwwapp01 { server app01.local; server app02.local; } |
Here we set two headers for proxied traffic (including the X-Forwarded-For header that our Apache configuration will log), and define the actual upstream proxy itself - which by default will use a round-robin load balancing algorithm.
Next, create a file containing the proxy configuration. The EPEL release of Nginx is configured to include any configuration files under /etc/nginx/conf.d/*.conf. I created proxy.conf as follows:
|
1 2 3 4 5 6 7 8 9 |
# vi /etc/nginx/conf.d/proxy.conf server { listen 80; server_name www01.local; location / { proxy_pass http://wwwapp01; } } |
And finally remove the default server configuration file default.conf:
|
1 |
# rm /etc/nginx/conf.d/default.conf |
With that, we can start Nginx on both nodes, and configure it to start automatically in the appropriate runlevels:
|
1 2 |
# service nginx start # chkconfig nginx on |
Now we can move onto keepalived configuration. Modify /etc/keepalived/keepalived.conf as follows:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# vi /etc/keepalived/keepalived.conf vrrp_script chk_nginx { script "killall -0 nginx" interval 2 weight 2 } vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 1 priority 101 # 101 on master, 100 on backup virtual_ipaddress { 192.168.122.61 } track_script { chk_nginx } } |
Ensure that the priority is set to 101 on the master, 100 on the slave. Here you can see that we do a simple check with killall -0 which is less expensive than pidof to verify that nginx is running.
Start and enable keepalived:
|
1 2 |
# service keepalived start # chkconfig keepalived on |
Our load balancer is now complete! You will note on the master server that the virtual IP address is active:
|
1 2 3 |
# ip addr sh eth0 | grep 'inet ' inet 192.168.122.55/24 brd 192.168.122.255 scope global eth0 inet 192.168.122.61/32 scope global eth0 |
Whereas on the slave it is not:
|
1 2 |
# ip addr sh eth0 | grep 'inet ' inet 192.168.122.56/24 brd 192.168.122.255 scope global eth0 |
Shutdown Nginx on the master (or shutdown the node), and you should see the virtual IP address failover. Start it back up, and the virtual IP address should fail back to the master.
Health Checks
As per http://nginx.org/en/docs/http/load_balancing.html, the reverse proxy implementation in nginx includes in-band (or passive) server health checks. If the response from a particular server fails with an error, Nginx will mark this server as failed, and will try to avoid selecting this server for subsequent inbound requests for a while (configurable). For more elaborate health checks, the commercially supported and paid version of Nginx (Nginx Plus) is required (see http://nginx.com/blog/load-balancing-with-nginx-plus/ for details).