SCSI-3 Persistent Reservations on Fedora Core 20 with targetcli over iSCSI and Red Hat Cluster

In this article, I’ll show how to set up SCSI-3 Persistent Reservations on Fedora Core 20 using targetcli, serving a pair of iSCSI LUNs to a simple Red Hat Cluster that will host a failover filesystem for the purposes of testing the iSCSI implementation. The Linux IO target (LIO) ( has been the Linux SCSI target since kernel version 2.6.38. It supports a rapidly growing number of fabric modules, and all existing Linux block devices as backstores. For the purposes of our demonstration, the important fact is that it supports operating as an iSCSI target. targetcli is the tool used to perform the LIO configuration. SCSI-3 persistent reservations are required for a number of cluster storage configurations for I/O fencing and failover/retakeover. Therefore, LIO can be used as the foundation for high-end clustering solutions such as Red Hat Cluster Suite. You can read more about persistent reservations here.

The nodes in the lab are as follows:

  • – Red Hat Cluster node 1
  • – Red Hat Cluster node 2
  • – Fedora Core 20 storage node


I’ll start by installing targetcli onto fedora01:

Let’s check that it has been installed correctly:

Make sure that, before proceeding, any existing configuration is removed:

Continue reading

Building a Highly-Available Load Balancer with Nginx and Keepalived on CentOS

In this post I will show how to build a highly-available load balancer with Nginx and keepalived. There are issues running keepalived on KVM VMs (multicast over the bridged interface) so I suggest you don’t do that. Here, we’re running on physical nodes, but VMware machines work fine too. The end result will be a high performance and scalable load balancing solution which can be further extended (for example, to add SSL support).

First, a diagram indicating the proposed topology. All hosts are running CentOS 6.5 x86_64.

nginx_load_balancerAs you can see, there are four hosts. lb01 and lb02 will be running Nginx and keepalived and will form the highly-available load balancer. app01 and app02 will be simply running an Apache webserver for the purposes of this demonstration. www01 is the failover virtual IP address that will be used for accessing the web application on port 80. My local domain name is .local.

Continue reading

Installing Nagios under Nginx on Ubuntu 14.04 LTS

Nagios is an excellent open source monitoring solution that can be configured to monitor pretty much anything. In this article, I’ll describe how to install Nagios under Nginx on Ubuntu 14.04 LTS.

First of all, check that the system is fully up to date:

Next, install the build-essential package so that we can build Nagios and its plugins from source:

Install Nginx, and verify that it has started:

Install libgd2-xpm-dev, php5-fpm, spawn-fcgi and fcgiwrap:

Next, create a nagios user:

Issue the following commands to create a nagcmd group, and add it as a secondary group to both the nagios and www-data users:

Download the latest Nagios core distribution from – at the time of writing this was version 4.0.7.

Continue reading

Implementing Git Dynamic Workflows with Puppet

Puppet is the obvious choice for centralised configuration management and deployment, but what happens when things go wrong (or you have the need to test changes)? A typo in a manifest or module, or an accidental deletion, and all hell could break loose (and be distributed to hundreds of servers). What’s needed is integration with a version control system.

I thought about using Subversion, but instead I decided to get with the times, and look at implementing a git repository for the version of my Puppet manifests and modules. Whilst I was at it, I decided to make use of Puppet’s dynamic environment functionality. The end goal was to be able to take a branch of the master Puppet configuration, and have that environment immediately available for use using the --environment=<environment> option to the Puppet agent.

An example will help clarify. Suppose I’m working on a new set of functionality, and don’t want to touch the current set of Puppet modules and inadvertently cause change in production. I could do this:

and then run my Puppet agent against this new testing code:

It would be a pain to have to update /etc/puppet/puppet.conf each time I create a new environment, so it is much easier to use dynamic environments, where a variable ($environment) is used in the configuration instead of static configuration. See the Puppet Labs documentation for more clarity.

First, edit /etc/puppet/puppet.conf - mine looks like this after editing – yours may be different:

As you can see, I set a default environment of production, and then specify paths to the manifest and modulepath directories, using the $environment variable to dynamically populate the path. Production manifest and modulepath paths will end up being $confdir/environments/production/manifests/site.pp and $confdir/environments/production/modules respectively. As new environments are dynamically created, the $environment variable will be substituted as appropriate.

Next, I moved my existing Puppet module and manifest structure around to suit the new configuration:

And restarted Apache (as I run my puppetmaster under Apache HTTPD/Passenger):

I then ran a couple of agents to ensure everything was still working:

They defaulted, as expected, to the Production environment.

Next, I installed git on my puppetmaster:

After this I created a root directory for my git repository:

/opt is on a separate logical volume in my setup. Next, create a local git repository from the existing Puppet configuration:

And clone a bare repository from this commit:

This cloned repository is where people will clone their own copies of the code, make changes, and push them back to – this is our remote repository.

All of the people making changes are in the wheel group, so set appropriate positions across the repository:

We can now clone the repository, make changes, and push them back up to the remote repository. But we still need to add the real functionality. Two git hooks need to be added – one to occur on update (the update hook) to perform some basic syntax checking of the Puppet code being updated and rejecting the update if syntax is bad, and a post-receive hook to check the code out into the appropriate place under /etc/puppet/environments, taking into account whether this is an update, a new branch, or a deletion of an existing branch. I took the update script from and made a slight alteration (as it was failing on import statements), and took the Ruby from here and the shell script from here, plus some of my own sudo shenanigans, to come up with a working post-receive script.

Here is /opt/git/puppet.git/hooks/update:

And here is /opt/git/puppet.git/hooks/post-receive:

As previously discussed, all admins working with Puppet are members of the wheel group, so I made sure they could run commands as puppet so that the sudo commands in the post-receive hook would work:

I also removed my Puppet account from lockdown for this:

With all these changes in place, I can now work as expected, and dynamically create environments with all the benefits of version control for my Puppet configuration.

GFS2 Implementation Under RHEL

This article will demonstrate setting up a simple RHCS (Red Hat Cluster Suite) two-node cluster, with an end goal of having a 50GB LUN shared between two servers, thus providing clustered shared storage to both nodes. This will enable applications running on the nodes to write to a shared filesystem, perform correct locking, and ensure filesystem integrity.

This type of configuration is central to many active-active application setups, where both nodes share a central content or configuration repository.

For this article, two RHEL 6.1 nodes, running on physical hardware (IBM blades) were used. Each node has multiple paths back to the 50GB SAN LUN presented, and multipathd will be used to manage path failover and rebuild in the event of interruption.

Continue reading

User, Group and Password Management on Linux and Solaris

This article will cover the user, group and password management tools available on the Linux and Solaris Operating Systems. The specific versions covered here are CentOS 6.4 and Solaris 11.1, though the commands will transfer to many other distributions without modifications (especially RHEL and its clones), or with slight alterations to command options. Check your system documentation and manual pages for further information.

Knowing how to manage users effectively and securely is a requirement of financial standards such as PCI-DSS, and information security management systems such as ISO 27001.

In this article, I will consider local users and groups – coverage of naming services such as NIS and LDAP is beyond its scope but may be covered in a future article. This article also presumes some prior basic system administration exposure with a UNIX-like operating system. 

Continue reading

Securing the Oracle Solaris 11 Operating System

Solaris 11 is the latest Operating System in the Solaris server OS range from Oracle, previously Sun Microsystems. It incorporates many features from Solaris 10 such as the Service Management Framework, but also pulls a lot from the now defunct OpenSolaris project including a new packaging system and a whole new suite of *adm administrative commands to configure the operating system.

This article will cover security configuration of the OS after a standard text-based installation. I’m using Solaris 11.1 x86_64 running as a VMware Fusion guest, but almost all of the steps will be applicable for the SPARC architecture too.

Continue reading

Configuring Interface Bonding on Debian (Wheezy)

This is a brief article detailing the steps to configure network interface bonding on Debian Wheezy (7.0 stable). The procedure is very different from RHEL/CentOS. I will be configuring active-backup (i.e. failover) mode bonding – there are other modes available, including round-robin load-balanced, LACP aggregation, etc. Read /usr/share/doc/ifenslave-2.6/README.Debian or for further information.

First, verify via ifconfig that your two slave interfaces are available – I’ll be bonding eth0 and eth1 into a bond called bond0:

Install the ifenslave package:

Next, stop networking. As you’re stopping networking, ensure that you’re connected to your host via a console of some form:

Modify /etc/network/interfaces. Remove (or comment out) any existing configuration for your slave interfaces (eth0 and eth1), and configure your new bond0 interface appropriately:

bond_miimon is the MII link monitoring frequency in milliseconds, and bond_{down,up}delay are the time, in milliseconds, to wait before disabling or enabling an interface in the bond (to safeguard against flapping), and should be a multiple of the bond_miimon value. You can adjust these values to suit your needs. These bond_<parameter> directives correlate to the <parameter> directives passed to the bonding module itself.

Once configured, start networking:

There is no need to fiddle with module loading (editing /etc/modules, creating a file under /etc/modprobe.d, etc.) – the ifenslave-2.6 package deposits scripts to do this for us. Take a look at /etc/network/if-pre-up.d/ifenslave to see this being done.

You can see the other scripts installed by ifenslave-2.6 with a dpkg -L:

Running ifconfig -a should now show the correct network configuration:

You can also view the contents of /proc/net/bonding/bond0 to check the status of the bond:



How to Fix Perl Make Errors Under Solaris

When you build Perl modules under Solaris, they are optimised for Sun Studio, which of course, we all use :/ So if you build with gcc, the build will likely fail. To avoid this Perl make error, you can use the following magical one-liner to fix this brain damage, and your modules will build correctly.

BIND 10: Configuring DHCP for DHCPv4 Clients

The fourth article in my series covering BIND 10 has nothing to do (directly) with DNS. This one describes how to configure the DHCP4 component of BIND 10 to provided DHCP service for IPv4 clients in my lab network. I used bind10-1.0.0-rc.

There is quite an important caveat to note from the RC release notes:

As you can see, DHCPv4 clients must be connected to the DHCP server via a relay. As I have no Cisco devices on my home network running ip-helper or similar, I used dhcrelay to provide DHCP relay services. I configured my network as per the following diagram, utilising three CentOS 6.3 x86_64 VMs:


gooby will provide DHCP service to the network via its eth1 interface. spooderman‘s dhclient will come online and make a multicast discovery request. dhcrelay on gooby will relay that message out of eth0 to dolan and b10-dhcp4. b10-dhcp4 will assign an IP address based upon configuration and current leases, and hand the offer back to gooby. dhcrelay will receive this offer, and relay it back out over eth1 via broadcast. spooderman will pick up this offer, and acknowledge it. Once b10-dhcp4 receives the acknowledgment, it commits the lease to the MySQL database. dhclient configures spooderman‘s eth0 interface, and the exercise is complete.

As is my NATted network out of my test workstation, VMware provides its own DHCP service on the host OS – ensure this is shut down prior to commencing work. If your networks do not already run a DHCP service, you don’t need to worry about that – if they do, shut them down before proceeding (as long as this isn’t production :/) …

With the network planned and any existing DHCP services disabled, we can continue.

Continue reading