Puppet is the obvious choice for centralised configuration management and deployment, but what happens when things go wrong (or you have the need to test changes)? A typo in a manifest or module, or an accidental deletion, and all hell could break loose (and be distributed to hundreds of servers). What’s needed is integration with a version control system.
I thought about using Subversion, but instead I decided to get with the times, and look at implementing a git repository for the version of my Puppet manifests and modules. Whilst I was at it, I decided to make use of Puppet’s dynamic environment functionality. The end goal was to be able to take a branch of the master Puppet configuration, and have that environment immediately available for use using the --environment=<environment> option to the Puppet agent.
An example will help clarify. Suppose I’m working on a new set of functionality, and don’t want to touch the current set of Puppet modules and inadvertently cause change in production. I could do this:
|
1 2 3 4 5 6 7 8 |
$ git clone me@git.server:/opt/git/puppet.git $ cd puppet $ git branch testing $ git checkout testing ... make some edits $ git add * $ git commit -m "made some edits" $ git push origin testing |
and then run my Puppet agent against this new testing code:
|
1 2 3 4 |
# puppet agent --test --environment=testing Info: Retrieving plugin Info: Caching catalog for sun.local Info: Applying configuration version '1371740640' |
It would be a pain to have to update /etc/puppet/puppet.conf each time I create a new environment, so it is much easier to use dynamic environments, where a variable ($environment) is used in the configuration instead of static configuration. See the Puppet Labs documentation for more clarity.
First, edit /etc/puppet/puppet.conf - mine looks like this after editing - yours may be different:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
[main] logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl confdir = /etc/puppet environment = production [agent] classfile = $vardir/classes.txt localconfig = $vardir/localconfig server = sun.local [master] storeconfigs = true storeconfigs_backend = puppetdb environment = production manifest = $confdir/environments/$environment/manifests/site.pp modulepath = $confdir/environments/$environment/modules |
As you can see, I set a default environment of production, and then specify paths to the manifest and modulepath directories, using the $environment variable to dynamically populate the path. Production manifest and modulepath paths will end up being $confdir/environments/production/manifests/site.pp and $confdir/environments/production/modules respectively. As new environments are dynamically created, the $environment variable will be substituted as appropriate.
Next, I moved my existing Puppet module and manifest structure around to suit the new configuration:
|
1 2 |
# mkdir -p /etc/puppet/environments/production # mv /etc/puppet/manifests /etc/puppet/modules /etc/puppet/environments/production |
And restarted Apache (as I run my puppetmaster under Apache HTTPD/Passenger):
|
1 |
# service httpd restart |
I then ran a couple of agents to ensure everything was still working:
|
1 |
# puppet agent --test |
They defaulted, as expected, to the Production environment.
Next, I installed git on my puppetmaster:
|
1 |
# yum install git |
After this I created a root directory for my git repository:
|
1 |
# mkdir /opt/git |
/opt is on a separate logical volume in my setup. Next, create a local git repository from the existing Puppet configuration:
|
1 2 3 4 |
# cd /etc/puppet/environments/production # git init # git add * # git commit -m "Initial import of Production Puppet repository" |
And clone a bare repository from this commit:
|
1 2 |
# cd /opt/git # git clone --bare /etc/puppet/environments/production puppet.git |
This cloned repository is where people will clone their own copies of the code, make changes, and push them back to - this is our remote repository.
All of the people making changes are in the wheel group, so set appropriate positions across the repository:
|
1 2 |
# chgrp -R wheel /opt/git/puppet.git # chmod -R g+w /opt/git/puppet.git |
We can now clone the repository, make changes, and push them back up to the remote repository. But we still need to add the real functionality. Two git hooks need to be added - one to occur on update (the update hook) to perform some basic syntax checking of the Puppet code being updated and rejecting the update if syntax is bad, and a post-receive hook to check the code out into the appropriate place under /etc/puppet/environments, taking into account whether this is an update, a new branch, or a deletion of an existing branch. I took the update script from projects.puppetlabs.com and made a slight alteration (as it was failing on import statements), and took the Ruby from here and the shell script from here, plus some of my own sudo shenanigans, to come up with a working post-receive script.
Here is /opt/git/puppet.git/hooks/update:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
#!/bin/bash # from: http://projects.puppetlabs.com/projects/puppet/wiki/Puppet%20Version%20Control NOBOLD="\033[0m" BOLD="\033[1m" BLACK="\033[30m" GREY="\033[0m" RED="\033[31m" GREEN="\033[32m" YELLOW="\033[33m" BLUE="\033[34m" MAGENTA="\033[35m" CYAN="\033[36m" WHITE="\033[37m" # V +1007 # Peff helped: # http://thread.gmane.org/gmane.comp.version-control.git/118626 # For Puppet 0.25.x: # syntax_check="puppet --color=false --confdir=/tmp --vardir=/tmp --parseonly --ignoreimport" # # For Puppet 2.7.x: # syntax_check="puppet parser validate --ignoreimport" # # NOTE: There is an outstanding bug against `puppet parser` which causes # the --ignoreimport option to turn the syntax check into a no-op. Until # the bug is resolved, the syntax check hook should not include the # --ignoreimport option and will only work correctly on manifests which # do not contain "import" lines. # See http://projects.puppetlabs.com/issues/9670 # syntax_check="puppet parser validate" tmp=$(mktemp /tmp/git.update.XXXXXX) log=$(mktemp /tmp/git.update.log.XXXXXX) tree=$(mktemp /tmp/git.diff-tree.XXXXXX) git diff-tree -r "$2" "$3" > $tree echo echo diff-tree: cat $tree exit_status=0 while read old_mode new_mode old_sha1 new_sha1 status name do # skip lines showing parent commit test -z "$new_sha1" && continue # skip deletions [ "$new_sha1" = "0000000000000000000000000000000000000000" ] && continue # Only test .pp files if [[ $name =~ [.]pp$ ]] then git cat-file blob $new_sha1 > $tmp set -o pipefail # added 21/06/2013 - Toki Winter grep -qs "import" $tmp if [ "$?" -eq "0" ]; then # blows up when it can't import a file, so just continue continue fi $syntax_check $tmp 2>&1 | sed "s|$tmp|$name|"> $log if [[ $? != 0 ]] then echo echo -e "$(cat $log | sed 's|JOJOMOJO|'\\${RED}${name}\\${NOBOLD}'|')" >&2 echo -e "For more details run this: ${CYAN} git diff $old_sha1 $new_sha1 ${NOBOLD}" >&2 echo exit_status=1 fi fi done < $tree rm -f $log $tmp $tree exit $exit_status |
And here is /opt/git/puppet.git/hooks/post-receive:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
#!/bin/bash read oldrev newrev refname REPOSITORY="/opt/git/puppet.git" BRANCH=$( echo "${refname}" | sed -n 's!^refs/heads/!!p' ) ENVIRONMENT_BASE="/etc/puppet/environments" # master branch, as defined by git, is production if [[ "${BRANCH}" == "master" ]]; then ACTUAL_BRANCH="production" else ACTUAL_BRANCH=${BRANCH} fi # newrev is a bunch of 0s echo "${newrev}" | grep -qs '^0*$' if [ "$?" -eq "0" ]; then # branch is marked for deletion if [ "${ACTUAL_BRANCH}" = "production" ]; then echo "No way!" exit 1 fi echo "Deleting remote branch ${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}" sudo su - puppet -c "cd ${ENVIRONMENT_BASE}; rm -rf ${ACTUAL_BRANCH}" else echo "Updating remote branch ${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}" if [ -d "${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}" ]; then sudo su - puppet -c "cd ${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}; git fetch --all; git reset --hard origin/${BRANCH}" else sudo su - puppet -c "cd ${ENVIRONMENT_BASE}; git clone ${REPOSITORY} ${ACTUAL_BRANCH} --branch ${BRANCH}" fi fi exit 0 |
As previously discussed, all admins working with Puppet are members of the wheel group, so I made sure they could run commands as puppet so that the sudo commands in the post-receive hook would work:
|
1 2 3 4 5 |
# visudo ... %wheel ALL=(ALL) NOPASSWD: /bin/su - puppet * ... |
I also removed my Puppet account from lockdown for this:
|
1 2 3 |
# usermod -s /bin/bash puppet # passwd -u puppet # passwd puppet |
With all these changes in place, I can now work as expected, and dynamically create environments with all the benefits of version control for my Puppet configuration.