Disclaimer: This post was originally posted in 2008 as an article on the now-defunct website zazzybob.com. While the software version and actual commands used may vary, the concepts are still similar and give a general idea of how to approach a given problem.

This article describes the process of replacing the venerable but limited syslog daemon with a versatile, flexible and customisable replacement, syslog-ng. We will be using our syslog-ng enabled host as a central logging server, so we’ll use LVM to create a log volume that can easily be expanded as disks are added to the system.

I am using a Red Hat Enterprise Linux AS 4 Update 2 host for my logging server. This has four SCSI disks attached, one 10Gb (for the / filesystem, plus /boot, /var, /var/log, swap, /tmp, /home). Three 5Gb disks are attached that will be used to create the volume upon which our logs will be stored. For the purpose of illustration, I will use two disks initially, and then extend the volume to encompass the third disk.

First, prepare the disks that will form our log volume with fdisk. I will just use two disks initially, then extend the volume onto the third disk to illustrate the process. As you can see, we set the partition type to 8e, for Linux LVM. I’ll show all command input, and will also show command output where it’s useful/unique/interesting.

We do the same for /dev/sdc

As this is the first time we’re using LVM on the host, we’ll run vgscan to initialise the LVM facility

Now, we can designate our physical disk partitions as physical volumes, and make them available for LVM use.

We can now create our volume group, which will will define as vg1

I will now create a 9Gb logical volume on the volume group

The next step is to create a filesystem on our logical volume. The standard mkfs tools can be used for this.

Now we can test the volume by mounting it

All good. Now, let’s extend our volume across the third disk. First, we must create a partition using fdisk

Next, use pvcreate to designate the partition for use by LVM, and vgextend to add the partiton to our vg1 volume group

Let’s use vgdisplay to check that things are looking good for our vg1 volume group

Now we can resize our logical volume, log_lv.

Next we check the filesystem with fsck, and then resize our ext3 filesystem across this newly allocated space.

Now, let’s mount the filesystem and ensure that all is well.

We now have a 14G volume ready to use as our log volume. I’ll now add a permanent mountpoint (/var/syslog-ng) and add an entry to /etc/fstab so that the filesystem is mounted persistently

Downloading the Required Packages

Now that our log volume is ready for use, we can start downloading the required packages for our logging server. I am compiling syslog-ng from source (http://www.balabit.com/downloads/syslog-ng/1.6/src/syslog-ng-1.6.11.tar.gz) and this has a prerequisite dependency on libol, so I’ll download that too (http://www.balabit.com/downloads/libol/0.3/libol-0.3.18.tar.gz). For viewing the (eventual) logfiles, I recommend multitail - an amazingly feature rich tail replacement and colorizer (http://dag.wieers.com/packages/multitail/multitail-4.0.5-1.el4.rf.i386.rpm)

The Build Process

Before we can build syslog-ng, we must first build libol on which syslog-ng is dependent. The usual build process applies…

If no errors are encountered, we can proceed to building syslog-ng. I’m happy with everything ending up in /usr/local so no need to modify --prefix during configure. Make sure you have flex installed too … configure will complete, but the make will barf without it.

While we’re at it, I’ll now install the multitail RPM.

Completing the syslog-ng Installation and Configuration

Now that we have syslog-ng installed (our binary is at /usr/local/sbin/syslog-ng) we need to do a few things before we can start using it. First, we must install a start/stop script under /etc/init.d for syslog-ng. A script comes with the syslog-ng bundle that will suffice for our purposes.

Before we can chkconfig this service, we must first modify the start/stop script slightly to point to the correct syslog-ng binary.

OK, next, we need to create our configuration file at /usr/local/etc/syslog-ng/syslog-ng.conf

Much of the configuration for this file is straightforward, and a detailed discussion is beyond the scope of this article - man syslog-ng.conf will give you the required information. I have created my own configuration file to mimick the functionality in the standard RHEL4 /etc/syslog.conf. As well as this, I have tailored the file to capture remote (and local) messages and direct them to the appropriate file under /var/syslog-ng.

OK, we’re almost there. Next, we need to stop syslogd and chkconfig it to disable the service. Then we can chkconfig syslog-ng and start it up!

We now have syslog-ng running! Let’s run logger, and check that the message appears in the traditional /var/log/messages file, as well as our new logfile under /var/syslog-ng/$HOSTNAME/$YEAR/$MONTH/$DAY/


I’m happy with that, so after a few more tests with logger, remove the old syslog service from chkconfig.

Getting Remote Hosts to log to our Logging Server

Let’s get a couple of our hosts logging to our new logging server. I’ll show two ways of doing this (both using /etc/syslog.conf on “traditional” syslogd-based boxes ) - one on another RHEL4 host, the other Solaris 10.

Of course, ensure that mphost04 (our logging server) is in /etc/hosts or DNS, otherwise just specify the IP address.

On our Solaris 10 box (which is actually a sparse root zone, but that’s a story for another article ), something along the lines of the following does the trick:

We should now see the logs pouring in from our remote hosts!

Using multitail to View Log Output

multitail has a *plethora* of options available; take a look at man multitail to get an idea. There are plenty of examples of usage on the multitail homepage (http://www.vanheusden.com/multitail), but I like to have a terminal window open at all times, tailing the files and using the default syslog colour scheme.

A sample of the output from multitail can be seen here.

Shortcomings

One problem with this is that if at the start of the day you fire up multitail, and a particular host has not yet logged a message for a particular day, its directory will not exist within the /var/syslog-ng filesystem when you initiate the tail, as the wildcards are expanded by the shell before multitail is invoked. Therefore the easiest workaround for this is to create a simple cronjob that calls logger on each host to generate the log file each morning - something like

This will ensure that when you fire up multitail at the start of the day, all relevant log files are already created.

Conclusion

This has just touched the surface of what syslog-ng can do, but has introduced a few other things along the way, including LVM and the very cool multitail package. Read the syslog-ng manual pages, and have a play with the available functionality. You can use FIFOs to match information in other log files, for example, and redirect that information elsewhere (or mail it, etc). Now is a good time to implement NTP on your network too if you’re not already using it, otherwise your timestamps within the logfiles will be out of whack.