Puppet is system administration automation software from Puppet Labs (http://puppetlabs.com). It has gained a lot of popularity, and rivals other automation/orchestration software such as Chef and Ansible.

In this article, I will detail how security can be managed on CentOS 6.x and Solaris 11.1 hosts with Puppet 3.x. Some familiarity with Puppet or some other automation software, as well as a Linux/UNIX system administrator audience, is assumed.

The topology being used for the examples given in this article is shown in Figure 1.

Figure 1. Example Puppet topology

As you can see, centosa is the Puppet master. Four hosts will contact it for configuration, including itself. There are three CentOS hosts in total (centos[a-c]) and a single Solaris host (sol11test). We will start with server and agent installation, then move on to cover various Puppet configuration tasks, and develop our own security module to deploy a set of security configuration to the hosts.

Whilst this article has been written with CentOS 6.x and Solaris 11.1 in mind, the techniques utilised should translate to RHEL/OEL 6.x and Solaris 10 without many changes. In case of doubt, consult the relevant security guide for your operating system at http://cisecurity.org.

Server Installation

The Puppet server is installed on host centosa. Start by installing the latest repository RPM from http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html#for-red-hat-enterprise-linux-and-derivatives. At the time of writing, this was puppetlabs-release-6-7.noarch.rpm.

Let’s see what was just installed:

The appropriate repositories are enabled by default (check /etc/yum.repos.d/puppetlabs.repo for details):

Next, install the Puppet server and agent packages, and their dependencies. This will install various required packages such as ruby, facter, hiera, and others.

The packages installed on a minimal CentOS installation are as follows:

Once the packages are installed, the Puppet master can be started. We will use the init scripts supplied with the Puppet master to control the daemon, and chkconfig to have it run at the appropriate runlevels.

First, start the Puppet master service:

Next, use chkconfig to enable the service:

Confirm that the service is configured to start as intended:

Now we need to update the local firewall (iptables is enabled by default on a minimal CentOS install). The default ruleset is as follows:

All of my test hosts are on the 10.1.1.0/24 network, and the Puppet master listens on port 8140. I therefore insert the rule as follows:

And verify:

If the ruleset looks good, save it:

From another host on the network, try using the openssl s_client to connect to the Puppet master:

All Puppet traffic is encrypted over SSL.

The Puppet server is now configured. We will use the common name shown above, centosa.local, in the server values in the [agent] section of puppet.conf, so take a note of yours. To avoid issues later on, this should match the FQDN of your host.

CentOS Agent Installation

On each server you wish to install the agent on (in our case, centos{a,b,c}.local, perform the following steps. Note that you don’t need to reinstall the packages on your Puppet master as you already did them during the Server Installation phase above.

Install the repository RPM:

Next, install Puppet and all dependencies. There is no need to install the puppet-server package on the other client nodes.

Once complete, on all client nodes, update /etc/puppet/puppet.conf. Edit (or add if it doesn’t exist) the [agent] section and add the following:

Change the value of the server variable to suit your environment. This should be the FQDN of the host, and should match the CN of your SSL certificate.

Next, a client certificate needs to be generated and signed by the Puppet master to authorise the addition of each node to the orchestration topology. Issue the puppet agent command with the -test option. -test includes many options useful for testing, including -debug and -no-daemonize and -show_diff.

We didn’t use -waitforcert (another option to puppet agent) so the agent will terminate after sending its CSR to the Puppet master for signing.

On the Puppet master, sign the outstanding request:

And verify:

Note above that a signed certificate already exists for centosa.local. It’s also noteworthy that the certificate has alternate DNS names of puppet and puppet.local - so we could reference centosa.local by a CNAME of puppet.local, and update the server variable in puppet.conf appropriately.

Repeat the process for all nodes. You should get a clean run once the certificate has been signed:

Solaris Agent Installation

The easiest way to install Puppet on Solaris is to obtain the packages from http://OpenCSW.org. OpenCSW uses a tool called pkgutil on top of the existing Solaris toolset to obtain, install and maintain OpenCSW packages.

Start by installing the latest version of CSWpkgutil:

The first step is to configure pkgutil to use PGP cryptographic verification. Issue the following command to install the CSWpki package via pkgutil:

Next, import the keys with cswpki:

The current fingerprint is available at http://www.opencsw.org/manual/for-administrators/getting-started.html, and currently looks like this:

With the key imported, edit /etc/opt/csw/pkgutil.conf and uncomment the following values, thus setting them to true from their defaults of false:

Now, run a pkgutil catalog update. You should see the GPG verification taking place:

Now, we can search for the appropriate Puppet package using pkgutil -a:

As this is only a client, we will need the puppet3 package, and any dependencies. pkgutil takes care of dependency resolution for us with respect to other OpenCSW.org packages.

For the sake of convenience, at this point you should update your $PATH accordingly to find binaries under /opt/csw/bin:

Install the puppet3 package and its dependencies:

By default, an SMF service is created to run the Puppet agent daemonised. This is not something that we want - the updates will be run out of cron for more control and granularity (more on this later). For now, check the status of the service:

Disable it, thus stopping it also:

Copy the supplied sample puppet.conf into place:

Update the puppet.conf server variable in the [agent] section as appropriate:

Try a test run; the certificate request will be sent to the Puppet master, and can be signed as shown in the CentOS instructions above.

Once the certificate is signed, a clean run should be observed:

A quick check of some of the facter variables on each type of host confirms that things are ready to go:

You can run facter –p to get a listing of all facts known to Puppet.

Puppet Configuration

The bulk of this article will now highlight some of the features of the Puppet configuration language, and how Puppet can used to deploy security configuration to hosts. Within an article there is an obvious limit to what can be covered, so the official Puppet documentation at http://docs.puppetlabs.com/references/latest should be consulted for authoritative information, as well as the appropriate security benchmarks for your operating system.

Before starting, it’s worth looking at the directory structure of the Puppet master installation. There are two subdirectories under /etc/puppet of note - manifests which contains Puppet manifests, and modules which contains Puppet modules. Each module is within its own subdirectory, e.g. /etc/puppet/modules/foomodule. Beneath the module subdirectory are three more directories - files (contains files you wish to serve to clients), manifests (manifests that comprise the module) and templates (any ERB templates your module uses). The entry point manifest is, by default, /etc/puppet/manifests/site.pp. Other subdirectories may exist under /etc/puppet if other features are being used (for example, hiera).

Start by configuring site.pp. In this file, include any site-wide defaults. /etc/puppet/manifests/site.pp is shown in Listing 1.

Listing 1. /etc/puppet/manifests/site.pp

A few things to notice about this example. Centralised backups to the Puppet master are configured using the filebucket type with name “main” and a server of “centosa.local” - our Puppet master. A default File object is then created causing all file modifications across all manifests and modules to be backed up to the filebucket “main“. .svn and .git files are ignored. We then go on to include nodes.pp via an import statement. The node declarations could go into site.pp, but we want to break things down for manageability and maintainability.

nodes.pp is shown in Listing 2 below:

Listing 2. /etc/puppet/manifests/nodes.pp

Using this format, you could import nodes-www.pp, nodes-mysql.pp, nodes-oracle.pp and so on. Each of these files will contain actual node definitions.

Listing 3 presents nodes-test.pp, the node definition file for our test hosts.

Listing 3. /etc/puppet/manifests/nodes-test.pp

As you can see, three node definitions are present. Within these are include statements which will call the classes of configuration that will be applied. The first matches hostname centosa.local and includes two classes - the security base class (/etc/puppet/modules/security/manifests/init.pp - which we will meet again later), and the security::logging::server class. The second definition matches two CentOS nodes - centosb.local and centosc.local. We could also have used a regular expression to match these. The third declaration is for the Solaris 11 node. Both of these definitions include the base security class and the security::logging::client class.

The various classes that comprise the configuration of the nodes is defined in a Puppet module called security. Puppet looks for modules in /etc/puppet/modules by default. Creating a module first requires the appropriate directory structure to be in place.

When the bare module name is included, as is the case in our example above (include security), there should be an init.pp file at /etc/puppet/modules/<modulename>/manifests/init.pp containing the corresponding class (in our case, this would be a definition of the security class).

Let’s take a look at init.pp for the security module in Listing 4.

Listing 4. /etc/puppet/modules/security/manifests/init.pp

There is a lot going on here. init.pp is merely a wrapper class in this case, farming off the work to various worker classes. The first six calls (include security::base::files through to include security::sshd) are applied to all hosts. Then, a case statement evaluates the operatingsystem facter variable. Facter is installed as a Puppet prerequisite and enables Puppet to query the host for “facts” about its configuration. One such variable is operatingsystem, and this and other top-level variables can be accessed via the $::<variable_name> syntax.

Depending on whether $::operatingsystem evaluates to CentOS or Solaris will dictate what further action is taken. You can see that in the CentOS case, there are four calls to the defined type security::base::filesystem, and the inclusion of the security::selinux class. For Solaris, three additional classes are included. The default case would be evaluated should the $::operatingsystem variable contain some other value.

The first included class is security::base::files. Let’s take a look at the class in Listing 5. It can be found at /etc/puppet/modules/security/manifests/base/files.pp. Note that the subdirectory base has been created and this matches the class name (using :: as a path separator).

Listing 5. /etc/puppet/modules/security/manifests/base/files.pp

Again, the content of the $::operatingsystem variable is evaluated and the appropriate file types are defined. Let’s break down a couple of the entries. First, /etc/issue. On a CentOS host, the configuration applied is as follows:

Here, the file /etc/issue will be created if it doesn’t exist (we “ensure” that it’s “present“), and the owner will be set to root, the group to root, and the permissions to 0644. The source of the file is on the Puppet master at puppet:///modules/security/etc/issue (which corresponds to a physical file path of /etc/puppet/modules/security/files/etc/issue). There are many more configuration attributes for the file type; consult the type reference (http://docs.puppetlabs.com/references/latest/type.html) for further detail.

Another type of entry is shown below, making use of ERB templates, here for Solaris hosts:

As you can see, the security/motd.erb template is being used to populate the file (the content attribute). The physical path to the file is /etc/puppet/modules/security/templates/motd.erb. Here is the ERB file:

fqdn is a facter variable, but we could equally reference any variable that’s defined in the appropriate scope, for example in the calling class.

A more complete example using ERB templates to configure Apache VirtualHosts can be found on my website (http://www.tokiwinter.com/puppet-module-apache2-virtualhost-templates).

We can now look at the next class - security::services - in Listing 6.

Listing 6. /etc/puppet/modules/security/manifests/services.pp

This class takes care of stopping any unnecessary services, and disabling them via whatever OS-specific mechanism is required (which Puppet hides from us, whether the provider is init.d scripts or SMF, we have a consistent interface via the service type).

The next class, security::tcpwrappers, is quite complex. It uses Hiera to look up variable values in a hierarchical database, in our case stored as plain text in YAML format. Hiera used to be an additional tool, but has been integrated fully with Puppet since version 3.0.

Let’s start with the class definition - take note of the positional parameters $hostsallow and $hostsdeny. These two variables will be populated via a Hiera lookup.

Listing 7. /etc/puppet/modules/security/manifests/tcpwrappers.pp

Since version 3.x, automatic parameter lookup in Hiera is enabled by default (see http://docs.puppetlabs.com/hiera/1/puppet.html#automatic-parameter-lookup for more details). Therefore, the values of security::tcpwrappers::hostsallow and security::tcpwrappers::hostsdeny will be looked up in Hiera.

The Hiera configuration is defined on the Puppet master at /etc/puppet/hiera.yaml as a plain text YAML file. Its contents are shown in Listing 8.

Listing 8. /etc/puppet/hiera.yaml

So that the hiera command-line utility works as expected, remove the installed /etc/hiera.yaml and symlink:

The Hiera configuration above does several important things. Firstly, it defines the available :backends: - here we use the yaml backend. The :datadir: of /etc/puppet/hieradata for the :yaml: files is defined next, followed by the :hierarchy: we wish to use. Our :hierarchy: is as follows:

First, the clientcert facter variable is checked, which will return the common name of the client certificate - generally the fully-qualified domain name of the host. If the file /etc/puppet/hieradata/%{::clientcert}.yaml exists, the security::tcpwrappers::hostsallow and security::tcpwrappers::hostsdeny variables looked up within them. If that file doesn’t exist, the operatingsystem variable is checked. If /etc/puppet/hieradata/%{::operatingsystem}.yaml exists, the variables are looked up there, and finally the catch all - if the granular tests fail, common.yaml will be used.

The contents of the YAML files are as follows:

Host centosa.local would use centosa.local.yaml (due to %{::clientcert} in the hierarchy) and pull the values in for security::tcpwrappers::hostsallow and security::tcpwrappers::hostsdeny from that file. Host centosb.local would fall through to common.yaml (unless there was a %{::clientcert}.yaml or CentOS.yaml), and a Solaris host would use Solaris.yaml.

Looking back at the security::tcpwrappers class, you can see that this substitution occurs during class instantiation as positional parameters:

These variables are then referenced in the file type definitions:

These variables will contain the output of the hiera lookup and thus the source attribute will reference the correct version of the file for the node we are deploying to.

After the files are deployed as appropriate, Solaris has some additional checks and/or configuration performed. First, inetadm -p is checked for tcp_wrappers=TRUE to verify whether inetd-controlled services are configured to use TCP Wrappers. If not set to TRUE, inetadm -M is used to update the configuration. Next, TCP Wrappers is enabled for the RPC portmapping service if it isn’t already via a call to svccfg. The service type definition is required to give us something to notify from the exec.

Defined types are pieces of code you want to call repeatedly with different parameters, akin to functions. security::ipadm is a defined type to check and set properties on Solaris hosts with ipadm. Its contents are shown in Listing 9.

Listing 9. /etc/puppet/modules/security/manifests/ipadm.pp

This can then be called from classes. The commented documentation in the listing explains each of the parameters. They default to empty, which would cause the commands to syntax error. You should add checks that sanitise input via conditionals. We will call this defined type in later manifests.

Next, we deploy kernel tuning changes - again the file is well commented. On Solaris, a change to /etc/system requires a reboot - obviously we don’t orchestrate that …

For suggested values for these files, see my previous articles published in PenTest Magazine on Securing the Linux and Solaris 11 Operating Systems.

Listing 10. /etc/puppet/modules/security/manifests/kerneltuning.pp

All is very straightforward in the above class. Files are copied for both hosts, and on CentOS, the sysctl -p command is run only if the /etc/sysctl.conf file changes (refreshonly=true makes the exec respond to events, and for that we use a file subscription to /etc/sysctl.conf).

security::networktuning is the next class, and it shows how the security::ipadm defined type can be called:

Listing 11. /etc/puppet/modules/security/manifests/networktuning.pp

security::sshd is shown in Listing 12. On CentOS hosts, it checks that the appropriate packages are installed and up-to-date. For both OSes, it then copies an appropriate sshd_config into place, before enabling and starting the service. If the configuration file is changed, the service is notified and refreshed.

Listing 12. /etc/puppet/modules/security/manifests/sshd.pp

Going back to init.pp, we have now discussed all of the OS-generic classes. Next, the OS-specific classes are considered:

You’ll note another defined type - security::base::filesystem. It is shown in Listing 13.

Listing 13. /etc/puppet/modules/security/manifests/base/filesystem.pp

This code will ensure that appropriate secure mount options are used in both /etc/fstab, and for any current live mounts via a call to mount -o remount where necessary.

The final CentOS specific class is security::selinux, which checks that SELinux is configured and enforcing. It is shown in Listing 14.

Listing 14. /etc/puppet/modules/security/manifests/selinux.pp

There are three final classes to discuss - the Solaris-specific classes. The first two, security::coreadm (Listing 15) and security::routeadm (Listing 16) are very straightforward. They check whether core dumps and routing are enabled, respectively, and disable the functionality if it is. You may need to adjust this to suit your purposes and site policy.

Listing 15. /etc/puppet/modules/security/manifests/coreadm.pp

Listing 16. /etc/puppet/modules/security/manifests/coreadm.pp

The final class, security::strong_tss is shown in Listing 17. You’ll note it calls the security::ipadm defined type:

Listing 17. /etc/puppet/modules/security/manifests/strong_iss.pp

This has only scratched the surface of what Puppet can achieve and I encourage you to read the documentation as it really is very good.

Running the Puppet Agent

Now that all the configuration is in place, the Puppet agent can be run:

You’ll see a lot of changes take place. For the sake of brevity, here is a brief excerpt from a run:

Run again and you should have a clean run as the configuration is up-to-date.

You can daemonise the Puppet agent, however I prefer to run out of cron. You can provide granular timing via cron, and control which hosts fetch their configuration at which times to balance load. I prefer to run puppet agent -test and have that output to a log file which is then managed by logrotate so that we have logs of verbose agent output. You may need to adjust this to suit your needs.

Add a cron job for root such as the following:

This would run the agent at 06 and 36 past the hour, outputting both STDOUT and STDERR to /var/log/puppet.log. You can then manage this log via logrotate. Read the manual page for logrotate.conffor details.

As a final noteworthy point, you can view a log file of all HTTP requests made to the server by checking the masterhttp log, which is located at /var/log/puppet/masterhttp.log by default.

Conclusion

This has only scratched the surface of what Puppet can do. There are further examples of Puppet modules and configuration on my website, as well as articles on adding robustness to your Puppet master (replacing the built-in webserver with Apache and Passenger), and integrating Puppet with a git workflow.

This article does not cover all security aspects of hardening CentOS and Solaris hosts. It serves as a guide to show you the power of Puppet, and set you writing your own modules and custom defined types.