As a System Administrator, there are many things that you need to know; you can’t just point and click your way around problems. While the number of skills is vast, I’ve narrowed them down to my list of the top 5 practices of a good UNIX System Administrator.

1. Be efficient

Writing one-liners on the UNIX command line is a sure way to increase your productivity, and is a very satisfying experience too. They key is to remember Einstein’s (paraphrased) approach: “Everything should be made as simple as possible, but no simpler.” In reducing keystrokes, don’t reduce readability of the code.

Take a look at the following code snippet:

Whilst command line editing can make fairly short work of this, I always find it better to write a quick on-the-fly loop, for example:

As well as saving you some typing, you can be guaranteed that this one-liner will be error free. That said, a system administrator should take a great deal of caution before doing something potentially destructive, such as removing files.

2. Security by Default

Always practice security by default. What do I mean by this? Be paranoid at all times. When you build and configure a system, take time to secure it properly and ensure that it is fully patched and tuned. All systems that leave your build lab (or dedicated build VLAN) should be hardened and completely locked down. Some of the fundamental concepts that are paramount are:

• Always choose strong passwords, enforce password rotation and implement strict access controls.
• Deny all access, and then grant specific access as required. This theory can be applied almost anywhere - firewalls, filesystem ACLs, Web Server ACLs, BIND ACLs, tcp_wrappers, etc.
• Ensure that all relevant Operating System patches and errata are applied before a system enters Production, and are vigilantly applied throughout the systems lifetime.
• Ensure that all systems are fully hardened. Subscribe to various security focused mailing lists, and respond to alerts before they become a problem. Always use an encrypted transport between systems. For example, ensure that site-to-site transfers take place over an encrypted VPN, and via an encryped scp (or other encrypted copy mechanism). Only use ssh for shell access to remote machines - never use rlogin or rsh. There are simple methods to allow ssh to provide the convenience of password-less authentication without the security risks.
• Physically secure your machine. There is no point ensuring host and network security if somebody can just walk over to a server and pull the power cord(s) out and steal the hard disks.
• Use a central authentication mechanism such as Kerberos or NIS. This will make user management far more efficient. Once an employee leaves an organisation, their accounts can be terminated system-wide very simply. That said, always maintain a local root account (and local System Administrator accounts).
• BE PARANOID!

This has, of course, barely scraped the surface of good security practices, and as such it is a topic that will be covered in more detail in later articles.

3. Document EVERYTHING!

No matter how small you think the change is that you’re making to a system, ensure that you document it fully. I normally keep shell session transcripts of any changes I make, so that I can refer to them later.

A good system administrator always maintains thorough documentation for all system builds. I recommend a wiki for this purpose as they are perfect for technical documentation; they harness a powerful but simple to use markup language, and allow for producing searchable documentation very quickly. It takes a couple of seconds to fire up a new browser window, and paste a command session into a new wiki page. Wrap it in some <pre> tags, and you’re done. Of course, we’d really need to sanitise this (write a script) and replace certain characters with their corresponding HTML entities (e.g. & for &, > for >, etc.), but this illustrates the point. Use it to document any common procedures, system information, site-specific documentation or anything else that will assist you in your daily administration.

4. Think laterally

Do not disregard the simple things. I’ve seen people waste hours diagnosing intermittent packet loss issues, only to find that faulty CAT5 was to blame. That said, trust your instincts.

5. Be consistent

Create server build templates and checklists, so that all server builds are completed consistently. There is nothing worse than logging into webserver-01 and finding the DocumentRoot under /var/www/somehost/htdocs, then jumping onto webserver-02 and finding it under /www/somehost/htdocs. Consistency is the key to automation and simple, effective administration. If you make a minor (and permanent) configuration file change to Apache on webserver-01, ensure that the changes are made on all other webserver-nn nodes too. Stumbling around trying to figure out where things are and why things behave differently will cost you time and cause a great deal of stress when you need to fix something in a hurry.

There are, of course, many more - and everybody has their own list. These are the five practices that I feel are the most important in a good system administrator - an excellent one has many more.