Secure MySQL Replication over SSL

MySQL is a popular open-source relational-database management system. One of its core features is replication, and in this article I will be showing how to configure a master and slave MySQL instance, and then configure replication from master to slave over SSL. Encryption will help protect the replication from snooping. This type of replication has many uses, for example: disaster-recovery scenarios whereby the slave can be switched to a master role in the case of a master outage, for performance where all reads can take place on the slave with writes and updates occurring on the master, and so on. Replication can be configured without encryption, but encrypting with SSL is preferred as part of a defence-in-depth strategy - it’s an extra layer of security.

This article already presumes a good working knowledge of MySQL. The master server is centosa with IP address 10.1.1.150, and is running a minimal installation of CentOS 6.4 x86_64. The slave, centosb, is running the same OS and has IP address 10.1.1.151. MySQL will be installed from the latest current stable RPMs available at dev.mysql.com, rather than using the upstream versions. The latest stable version available at the time of writing is 5.6.14.

This article will cover the configuration of an SSL-encrypted replicated environment from scratch - it does not cover the migration of an existing replicated configuration to an SSL-encrypted replicated configuration, or the migration of any existing data to a new slave.

Continue reading →

Using rsync to Synchronise Two Directories Securely

rsync is a utility that provides fast incremental file transfer. This allows us to keep directories and files synchronised between servers whilst only transferring the minimum required data. This is often employed during code deployment scenarios from one environment to another.

It is preferable to use the -e option to specify a remote shell to use, namely ssh. This will ensure that the file transfers are encrypted and performed over SSH.

In our deployment scenario there are two servers, venus and earth. Both are webservers, and have content at /var/www/example.com/htdocs. venus is our test server, and earth is production. We’d like to deploy our content from venus to earth. This deployment could be automated (think continuous integration), or even scripted for simple deployments, but for the purposes of this tutorial we’ll just manually issue the commands as required. This is being performed on CentOS 6.4 with SELinux enabled.

I’ll start off by creating some simple “content” on venus:

On earth, I’ll just create the empty content directory:

Create a user/group on both servers that will be used for deployment. deploy@venus will allow interactive logins to push content to deploy@earth. deploy@earth will only allow the specific remote-side rsync command to run. This will be defined later in ~/.ssh/authorized_keys. For now, just create the users and groups:

Set ownership on our deployment directories. Do this on both servers:

It is obvious that any permissions set here should not interfere with the way your web server actually serves its content. You may need to set up ample group permissions for the deploy group, then add the webserver user to that group.

The first step in setting up our file syncronisation is to configure password-less SSH login from deploy@venus to deploy@earth. Having no passphrase on the SSH key we’re about to create is the reason for limiting command access via ~/.ssh/authorized_keys.

Start with key generation as deploy@venus:

As you can see, a 2048-bit RSA key has been generated.

Next, create ~/.ssh on earth, and copy the public key to deploy@earth:.ssh/authorized_keys:

On earth, we now need to fix SELinux contexts and permissions on deploy@earth:

We can now check that password-less SSH is working:

Next, we can use rsync to perform a dry-run of our deployment with the --dry-run option. Specifying the -v option twice shows the remote command that will be executed:

The remote command can be seen above - rsync --server -vvnlogDtprCze.iLs . /var/www/example.com/htdocs. We can populate deploy@earth:.ssh/authorized_keys with this command against deploy@venus‘s public key in the file, limiting deploy@venus to ONLY be able to execute the remote rsync command. We need to drop one of the -v‘s and -n (--dry-run) from the remote command too. Modify deploy@earth:.ssh/authorized_keys as appropriate:

Note that I also add the from= configuration parameter so that only connections from venus are permitted. Now, the password-less SSH configuration is locked down so that any connections from deploy@venus can only run the remote rsync command.

Test from deploy@venus:

Running the command again should not transfer any changes (as there haven’t been any):

And we can now verify that earth has the files:

 

Knot DNS Server

Introduction

Knot DNS server is a high-performance authoritative-only DNS server which supports all of the key features of the domain name system including zone transfers and DNSSEC. It was developed by the CZ.NIC. Knot DNS is open-source and multi-threaded. Current features include:

  • Full and incremental zone transfers (AXFR/IXFR)
  • Dynamic DNS updates
  • EDNS0 and DNSSEC extensions, including NSEC3
  • Response Rate Limiting
  • NSID
  • TSIG
  • and all of the standard features you’d expect with an authoritative-only nameserver implementation

Knot used to require zone files to be compiled (like NSD), but this requirement was removed as of Knot 1.3.0.

For this article, I’ll be running the latest stable version of Knot, 1.3.3, on CentOS 6.4. The test environment will comprise two hosts - venus (master) on 192.168.122.12 and earth (slave) on 192.168.122.13.

Knot 1.4.0-rc1 was also available at the time of writing which supports experimental DNSSEC auto-signing, so that may be worth checking out if relevant to your needs.

Continue reading →

How to Network Solaris 11 Zones Under VMware or VirtualBox

Whilst playing around with the changes in zone virtualisation technology between Solaris 10 and Solaris 11, I found that all zones now use exclusive IP, not shared. There is a new anet interface type configured via zonecfg that handles this.

This will all be covered in detail in a future article, but for now take a look at this:

I boot my new testzone, and the vnic is automatically created over net0. But net0 is itself a virtual NIC (i.e. VMware or VirtualBox is virtualising this for us in the first place), whilst Solaris obviously sees it as a physical interface.

Inside the new zone, I was unable to ping anything. The zone was not on the network. The fix? Place net0 into promiscuous mode using snoop inside the global zone. This makes sense when you think about it, and will fix your zone networking allowing you to virtualise within your VM:

OpenVMS: How to Fix %NCP-W-MIRCTF and %SYSTEM-F-NOSUCHNODE No Such Node errors

Whilst configuring an OpenVMS 7.3 (VAX) 2-node Cluster using DECnet Phase IV, I was unable to use commands such as SET HOST as nodename lookup was failing. Each node could “see itself”, as it were, but that was about it.

To resolve this, I first started by using NCP and ran a couple of LOOP NODE commands. The first to my local node TOM (which silently returns - i.e. loop is verified) and the second to remote node JERRY (which fails):

OK - let’s check the NODE database:

So, we can only see the EXECUTOR itself, TOM (with DECnet address 1.151). Using the SET NODE and DEFINE NODE commands, I added an entry for JERRY to the database on TOM:

A SHOW NODE now looks more promising:

Next, on JERRY, run NCP and copy the known nodes database down from TOM (1.151):

Great - we can see both nodes from both nodes. LOOP NODE will confirm this, but let’s try something cooler, a remote login with SET HOST:

From JERRY:

Excellent. Check the other direction with a SET HOST JERRY from TOM, and verify that all is well. The PHONE utility should now work too - bet you’re happy about that ;)

Installing chroot()ed BIND - Ubuntu 12.04 Server

This HOWTO will cover how to configure chroot()ed BIND under Ubuntu 12.04 Server. We want to chroot() BIND for security reasons - having software operate out of chroot() jails closes an avenue for exploit. I will be configuring a chroot() of /var/chroot/named.

First, install the bind9 and dnsutils packages. The former will provide the BIND software itself, whilst dnsutils provides utilities such as dig and nslookup that can be used to perform queries or troubleshoot your installation:

This will give us BIND 9.8.1-P1 at the time of writing this article:

It will also start BIND - so shut it down before proceeding:

Modify /etc/default/bind9. This file contains an OPTIONS variable whose contents are passed as options to named when it’s started. Add the -t option - which specifies the path to the chroot() jail:

Create the appropriate directories for the chroot() tree:

Move the default BIND configuration files to the chroot(), maintaining a symlink to the original location:

Create the null and random devices under the chroot(), and fix permissions:

Change ownership of the chroot() to the appropriate user and group, in our case bind:bind:

libgost.so from openssl-1.0.0 needs to be available under the chroot(), so create a bind mount for it. Add the folllowing to /etc/fstab:

Create the mountpoint under the chroot(), and mount:

We’ll need to create a syslog socket inside the chroot() so that BIND can find /dev/log where it expects it to be. Create /etc/rsyslog.d/60-bind-chroot.conf as follows:

The final step is to update our AppArmor configuration to allow reads and mmap calls on libgost.so, and reads on the BIND configuration files. Update the local AppArmor configuration as follows:

Reload the AppArmor profiles:

Now, we can start our chroot()ed BIND

and test:

 

Reconfiguring KVM-based networking DHCP range with virsh

Whilst building a KVM-based virtualisation server on CentOS 6.4, I noticed that the default network created during the installation of the KVM/QEMU packages (192.168.122.0/24) allocated the entire address space to DHCP, as can be seen below in virt-manager (click to enlarge):

virtualnetwork

All of the configuration fields are read-only, and I wanted to reduce the DHCP scope so that I could build statically-addressed virtual servers without the hassle of creating static DHCP reservations for them. This scope can also be confirmed by looking at the current running dnsmasq configuration (dnsmasq is used by KVM/QEMU to manage DHCP):

The powerful virsh shell can be used to perform this modification and directly alter the XML configuration that backs the virtualisation platform.

To start, run virtsh net-edit <network-name> to fire up your $EDITOR:

Prior to making any modifications, my configuration was as follows:

Next, reduce the DHCP range as required - my modified configuration is as follows:

Once you quit your editor, you’ll see a message as follows, indicating that the change has been made:

If at this stage you dump the configuration with virsh net-dumpxml <network-name> you will see that the changes have NOT yet been applied:

To apply the changes, restart the network with virsh net-destroy <network-name> and virsh net-start <network-name>:

Now, a dump of the XML shows that the changes have been applied

as does another ps on the dnsmasq process:

Now, launching virt-manager shows that the desired DHCP scope is in effect:

virtnetwork-updated

Another case of command line winning over the GUI. virsh is a VERY powerful command, and there will be more articles on its usage as I dabble more with KVM-based virtualisation.

Oracle Products: Enterprise Grade Or Just Enterprise Pricing?

I recently worked on a major project for a large organisation implementing over a hundred servers. These servers comprised the entire back-office infrastructure and would provide billing and financial services, order processing and CRM. The software stack chosen was a suite of Oracle products: Oracle eBusiness Suite, Oracle Billing and Revenue Management, Oracle Siebel, Oracle SOA Suite, Oracle OBIEE and Oracle DB Enterprise Edition. All very expensive software to both license and support.

I’ve worked heavily with Oracle DB before, and found it to be very stable and robust when sitting on Sun Clusters - however this would be the first time I got to install the software on our new “Enterprise” Operating System - Oracle Enterprise Linux. The reasoning behind the sea of Oracle was that we’d have complete end-to-end support for the entire stack. We’d log one ticket about an issue and someone at Oracle would be able to help. That was the theory, anyway.

Continue reading →

SELinux: Allowing httpd to Listen on Non-Standard Ports

If you attempt to have httpd listen on a non-standard port (we’ll get to what’s “standard” and what isn’t in a moment) on an SELinux enabled host, SELinux will deny the request. Let’s try it out (on a CentOS 6.4 box):

Kaboom! As you can see, the error message is quite clear “(13)Permission denied: make_sock: could not bind to address”. Let’s take a look at the audit logs and see what’s going on:

There are two sets of messages there, one for the IPv4 bind, the other for the IPv6 bind. The AVC message actually makes it clear what’s going on:

type=AVC msg=audit(1371477046.382:20195): avc: denied { name_bind } for pid=16429 comm="httpd" src=8585 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket

You can see that the violation is from source context (scontext) unconfined_u:system_r:httpd_t which is the httpd process, and the target context (tcontext) is system_u:object_r:port_t with target class (tclass) tcp_socket. The denied port (8585 in this case) is also logged in the src field.

SELinux will only allow httpd to bind to ports of type http_port_t. Here, we can see that httpd is attempting to bind to a port of type port_t - which is going to be denied by the policy. We need to define port 8585 as a port of type http_port_t. semanage is the tool for that - and you’ll need the policycoreutils-python package installed:

First, let’s look at which ports are currently defined as type http_port_t:

And add our desired listen port, 8585/tcp to the list:

Looks good. Fire up httpd:

And test with nc or similar:

 

Script: add_users.sh

Script Name

add_users.sh

Script Summary

Add users to a system en-masse.

Script Description

This script will read a list of space separated Firstname Lastname entries, one per line, from a file specified. It will create an appropriate group (specified within the script) to add users to if it doesn’t already exist. If the group does exist, the users will be created via useradd and a random password set, and those passwords outputted to STDOUT.

Script Limitations

  • Only tested on RHEL/OEL/CentOS 5.x and 6.x.
  • Will handle duplicate usernames semi-gracefully by exiting with an error, rather than doing anything fancy - you’ll need to add those users manually

Continue reading →