Introduction

This article will cover the build of a two-node high-availability cluster using DRBD (RAID1 over TCP/IP), the Corosync cluster engine, and the Pacemaker resource manager on CentOS 6.4. There are many applications for this type of cluster - as a free alternative to RHCS for example. However, this example does have a couple of caveats. As this is being built in a lab environment on KVM guests, there will be no STONITH (Shoot The Other Node In The Head) (a type of fencing). If this cluster goes split-brain, there may be manual recovery required to intervene, tell DRBD who is primary and who is secondary, and so on. In a Production environment, we’d use STONITH to connect to ILOMs (for example) and power off or reboot a misbehaving node. Quorum will also need to be disabled, as this stack doesn’t yet support the use of quorum disks - if you want that go with RHCS (and use cman with the two_node parameter, with or without qdiskd).

This article, as always, presumes that you know what you are doing. The nodes used in this article are as follows:

• 192.168.122.30 - rhcs-node01.local - first cluster node - running CentOS 6.4
• 192.168.122.31 - rhcs-node02.local - second cluster node - running CentOS 6.4
• 192.168.122.33 - failover IP address

DRBD will be used to replicate a volume between the two nodes (in a Master/Slave fashion), and the hosts will eventually run the nginx webserver in a failover topology, with this example having documents being served from the replicated volume.

Ideally, four network interfaces per host should be used (1 for “standard” node communications, 1 for DRBD replication, 2 for Corosync), but for a lab environment a single interface per node is fine.

Let’s start the build …

Initial Node Preparation

For the sake of simplicity, disable iptables and SELinux on each node:

Ensure that NTP is correctly configured (I do this in my kickstart configuration anyway):

If you are not using DNS, add entries to each of your host’s /etc/hosts files for all nodes in the cluster - or do so anyway as a safeguard against DNS failure:

At this point, I configured LVM. I presented a disk to each node (themselves running as KVM guests, as discussed earlier), at /dev/vdb - verified with fdisk -l:

Next, I partitioned the entire disk as type 8e (Linux LVM), created a new physical volume using the new partition (/dev/vdb1), a new volume group using that PV (vg_data) and then a 1GB logical volume within the new volume group - lv_data_01:

This 1GB volume will be used for the DRBD replicated storage volume on each node. We can now move on to software package installation.

Package Installation

I installed pacemaker and corosync from the standard CentOS repositories:

However, between CentOS 6.3 and 6.4 being released, pacemaker has been updated from 1.1.7 to 1.1.8, which means that the CRM shell is no longer included. This is now maintained in its own project - crmsh - and CentOS 6 RPMs are available via the OpenSUSE package repositories.

So, I installed crmsh by way of its dependencies:

Next, I installed DRBD 8.4 from the ELrepo repository:

I then ensured that it wasn’t set to start automatically at boot (as the cluster will manage this):

I left nginx package installation for later in the procedure.

Corosync Configuration

As I’m using KVM guests, and multicast is particularly troublesome over the bridged interface (and I didn’t want to hassle with smcroute, or similar, to set up multicast routing) I used unicast for cluster communications. A sample unicast configuration file is included at /etc/corosync/corosync.conf.example.udpu. I took that file and edited as follows:

Next, I created a service definition for the Pacemaker resource manager that we’ll be using:

As I’d changed secauth to on (it was off in the example), I then needed to create a key to encrypt cluster communications and distribute that around the cluster. On the first node:

This command will sit waiting for entropy to be generated. A quick way of doing this is to ssh in to the node via another session and run

a few times. Once the key is generated, it’ll be available at /etc/corosync/authkey. Copy it to the other cluster node, and secure it:

Now, we can start our bare-bones cluster:

And verify the status:

Awesome. The cluster is running. A quick check via the crm shell shows that there isn’t much to the configuration at this point:

DRBD Configuration

The next piece of the puzzle is to configure the replicated storage volume. On both nodes, take a backup of /etc/drbd.conf (which itself includes files under /etc/drbd.d - I’ll just use /etc/drbd.conf) and edit as follows. As you can see, I have commented the configuration file directives very well, so will not explain them in this text apart from saying that the LVM logical volume at /dev/vg_data/lv_data_01 created earlier is used as our backing store device, and the new DRBD volume will be available at /dev/drbd0:

On both nodes, create the DRBD metadata:

You can use a separate volume for metadata, but if you’ve not yet created a filesystem you can use internal metadata as is the case above. You may also opt to use a separate volume for metadata for performance reasons. You can read the DRBD Internals page for more information on that. For most simple applications, internal metadata will suffice.

Start the DRBD service on both nodes (remembering that we will be shutting it down again as it will be cluster managed):

On the MASTER NODE only, promote the volume to the primary role:

Note that in DRBD versions <8.4 (e.g. 8.3) you will need to use a different command:

You can cat /proc/drbd to watch the synchronisation operation in progress:

Or, run service drbd status:

On both nodes, create the mountpoint for the DRBD-replicated volume, which in our case will be /data. It is needless to say (but I’ll say it anyway) that this volume will be only ever mounted from a single node at a time.

From the MASTER NODE, create an ext4 filesystem and mount/umount the new filesystem to test. Ensure that you use /dev/drbd0 - i.e. the DRBD device - not the backing store device:

Now, on both nodes, stop DRBD:

We have our storage. Next, the murky world of the cluster resource manager.

Cluster Resource Configuration

Whilst there is lots of documentation available (both in articles like mine, and the excellent (and definitive) documentation at clusterlabs.org such as Clusters from Scratch), the CRM shell can still be a daunting place to be. Whilst there are GUI cluster resource configuration tools available, I much prefer to use the CRM shell due to the flexibility it offers. It is worth taking some time to learn the shell - you will be rewarded for doing so as you will have full control over your cluster.

First off, as we’re only running a demo cluster and do not have any real fencing methods available to us, let’s disable STONITH:

and as this is a two-node cluster, ignore a loss of quorum:

Let’s create our first resource - the failover IP address. Using the ocf:heartbeat:IPaddr2 resource agent:

Here, you can see that the IP address resource (or primitive) of agent type ocf:heartbeat:IPaddr2 has been created with a name of failover_ip. There are parameters passed (an ip of 192.168.122.33 and a cidr_netmask of 32) and operations (here, a monitor of interval 20s). If you are ever unsure of which parameters and operations a resource agent supports, you can view extensive documentation via the crm ra meta <ra-name> command:

Once the failover_ip resource has been created, you can view the status of it with crm resource status:

Running ip addr will also show you that the new IP address is online:

You can now use crm move resource <resource> to move the resource to the second node:

Oh noes! What does this mean? Essentially - you, the sysadmin, know what you are doing, and have manually forced a move of a cluster-managed resource. The cluster manager therefore deems the current node unfit for purpose and creates a location constraint with a score of -INFINITY (negative-infinity) - the resource will never run there again. You can remove the constraint via the crm shell, or just fail it back with crm unmove resource <resource> which will also deem the node fit for purpose once again, and remove the constraint:

You can also test failover by making the current node standby:

Remembering to bring it online again once done:

You can view the current status of the cluster and any resources with crm status, or crm_mon -1 which will do the same thing - more on that later:

You now have the first clustered resource working, and failing over as needed - the “floating” IP address. Next, nginx.

nginx Installation and Configuration

At this point, I’ll install nginx as it’ll be the next clustered resource to be configured:

Again, ensure that it doesn’t start automatically on boot, as the cluster will manage the nginx service:

Reading the output of crm ra meta nginx, we learn that the agent works by polling /nginx_status. This is not present in the configuration provided by the nginx RPMs just installed, so we will need to add it manually. I configured it as per the HttpStubStatusModule documentation, with appropriate ACLs:

Once done, start nginx and test that this URL is reachable:

Next, configure the nginx resource via the crm shell:

The WARNING I’ll ignore - a 10s timeout for a response from /nginx_status is still quite conservative. Quite a bit happened above. First, I configured another primitive (i.e. resource) called nginx_res, using the ocf:heartbeat:nginx resource agent. I specified a path to the configfile and the nginx binary (httpd=) and then specified various operations to define monitor, start and stop timeouts.

Next, I created a colocation called nginx_ip_colo. Colocations tell the cluster that resources need to be colocated on the same node. An affinity of INFINITY is specified - i.e. nginx_res and failover_ip MUST be on the same node, no exceptions. -INFINITY (i.e. negative infinity) would have the two resources repel and run on different nodes, no exceptions - not what we want.

Finally a mandatory (does the same thing as INFINITY) order directive, nginx_after_ip, was created, forcing nginx_res to start after failover_ip.

crm configure show now shows the current status of our cluster configuration:

A “one-shot” view of our cluster status with crm_mon -1 shows the cluster operating as expected, with the failover_ip and nginx_res resources being served from the same node (rhcs-node02.local):

We can again move the resources around with crm move/unmove resource:

Next, DRBD integration into the cluster.

DRBD Cluster Configuration

All this time, we’ve been issuing commands and making direct changes to the cluster configuration, e.g. crm configure <some-configuration> - the changes are applied immediately. However, if we try to configure the multi-part DRBD resources like this, some of the configuration will be applied before the entire resource has been configured, which will not give us the desired result. Therefore, I’ll take a copy of the current CIB configuration and make changes there on a “shadow instance”, verify, then commit back into the main cluster configuration. This is a much safer way of working, and should be used when making any significant change on a Production cluster.

Let’s start by creating the shadow configuration instance:

You can see that the crm prompt has changed from crm(live) to crm(drbd) indicating that we are working with the shadow instance.

Next, I’ll create the DRBD primitive and a new type of resource - ms:

Again, much has happened. The drbd_res primitive has been created of type ocf:linbit:drbd, with the drbd_resource parameter set to data - the same resource name we defined for our DRBD replicated resource in /etc/drbd.conf. Then, the monitor intervals for both master and slave are defined. Next, the drbd_master_slave ms resource is created - ms resources are used where master/slave relationships are required. Meta data is passed during the creation of this resource ensuring that we only ever have a single master online at any one time. These parameters are taken from the DRBD on Pacemaker documentation.

Run a configure show and verify that all is well - if you’re happy, commit your changes to the live cluster configuration:

Next, another shadow instance will be created to configure the filesystem resource that runs atop our DRBD resource:

Create the filesystem resource and appropriate colocation and ordering definitions:

So - another primitive, fs_res, of type ocf:heartbeat:Filesystem is created on device /dev/drbd0, to be mounted on directory /data, of fstype ext4.

Colocations are then created so that the fs_res is colocated on whichever node is running drbd_master_slave:Master, and that nginx_res runs wherever fs_res is running. Ordering is configured so that drbd_master_slave:promote occurs before fs_res:start (i.e. the DRBD resource is promoted to master before the filesystem resource is started and any mount occurs), and that nginx_res is only started after fs_res is online. The configuration is then checked, and committed. I’ve substituted “INFINITY:“, “inf:” and “mandatory:” above to show that they all do exactly the same thing, and as you’ll see below, all get converted to “inf:” in the committed configuration.

If this were a production cluster, I’d just need to configure STONITH here as an additional step. However, for our purposes, the cluster configuration is now complete - and we have a replicated storage volume that is promoted/demoted appropriately across the master/slave, and a failover nginx instance. One final step is to have nginx serve files from the shared storage.

Final Steps

Let’s take a look at the completed cluster configuration:

As you can see, it’s not really that daunting at all once you’ve worked through the confirguation steps.

Let’s move the nginx document root onto the replicated storage - make the following configuration change on both nodes:

and populate index.html from the master node:

Fail the services over to the other node, and ensure that the new content is visible. The best way to do this at this point is by taking the current master node offline, then bringing online again (including an appropriate sleep so that the failure is detected). If you leave the node offline, the DRBD slave will not run. As no preference configuration has taken place, the services will not automatically fail back when the node is brought back online. Or, of course, just reboot the master node and ensure failover occurs as expected:

Excellent - failover works correctly, and the content is correctly served from the replicated volume.

Conclusion

This has been a bit of a whirlwind tour of a complicated stack of software. A very viable alternative to “enterprise” cluster solutions such as RHCS or VCS, this approach can be adopted to cluster a wide range of applications.

There is a vast amount of documentation available, and a very active user base (i.e. mailing lists, etc.). If you need to implement a high-availability solution on Linux, then this is probably for you. I’ve never been much of a fan of RHCS, although in my opinion it still doesn’t come close to the ease of administration and general rock-solid stability of Solaris Cluster.

One place I’ve used this is a highly-available MySQL master server, with several standalone replicated slaves.