Any application is only as secure as the operating system hosting it. Whilst most servers are protected behind firewalls with the main avenue of exploit being the application, if an application is exploited and the user did manage to obtain server access, we need to minimise the damage that can be done to the exploited machine and others. Running hardened systems is a requirement of certain data security standards such as PCI-DSS (PCI Application Data Security Standard), as well as mandated by many organisations. Plus, it is just good practice. Out-of-the-box, most modern OSes are configured to be secure. There is still a great deal we can do, however, to further harden the operating system to various vectors of attack.

This article will only cover the core aspects of Linux OS lockdown, and will focus on CentOS 6.4. Most of the concepts here should be able to be ported to other flavours, using the appropriate commands for your OS.

For further detail, consult the security benchmarks available at http://cisecurity.org. You should employ multiple layers of security within your architecture – for example, using network-based firewalls as well as host firewalls, writing secure code, ensuring that all aspects of the application stack are patched and up-to-date, etc. Hardening the OS is just one more layer of security.

Operating System Installation

The OS needs to be installed in such a way that we can easily harden it. For this, choose a minimal package set, and then onto this only install the packages and services required. Partitions should also be created so we can separate and isolate logs and temporary files from critical system filesystems to prevent Denial-of-service by filling a filesystem. An excellent mechanism for implementing this is the Linux Logical Volume Manager (LVM). This allows us to easily create new volumes as required, and increase the size of existing volumes and their hosted filesystems should the need arise. The OS should also be patched up-to-date.

Let’s consider our test system, earth.local. This is the system that will be hardened during this article. The system is a KVM guest, and has two disk devices attached. Of those, /dev/vda has two partitions, /dev/vda1 for the /boot filesystem, and /dev/vda2 to provide a backing store for the vg_sys LVM volume group. Within this volume group, a number of logical volumes have been created:

As you can see, /tmp, /var/log and /var/tmp are filesystems on their own logical volumes, as is /home. This will prevent the root filesystem being filled should a simple DoS attack produce a large volume of logs, for example. If you have custom binaries running out of /usr/local, it is worth placing that filesystem on an isolated volume too, as well as any chroot jails or other application-specific filesystems. If you are installing a new system, you can specify this partitioning scheme. However, if you’re hardening an existing system, you may need to move filesystems onto logical volumes. Below, I’ll describe the process for moving /usr/local onto its own logical volume. A similar procedure can be utilised for moving other filesystems.

First, create a logical volume for the new filesystem:

Here, I create a 1GB logical volume named lv_usr_local in volume group vg_sys. Create a filesystem on the device. Instead of using the indirect block scheme for storing the location of data blocks in an inode, use extents instead. This encoding is much more efficient and speeds up filesystem access, especially for large files.

At this point, shut down any applications or other services that have open file handles within the target filesystem. Tools such as lsof and fuser can be used for this, e.g.:

Move the existing data sideways:

Create the new mountpoint:

Update /etc/fstab:

Mount the new filesystem and verify with df:

The data can now be copied to the new filesystem:

Start any applications, and verify correct operation. Once you’ve verified that all required data is under your new /usr/local filesystem, you can remove the copy:

The OS should now be patched. On CentOS 6.4 as well as other modern RHEL-derivatives, yum is the command to use for package management. Check if any packages are marked for update:

Then, apply the updates:

Once all updates have been applied, if the kernel has been updated as a result of the yum update, reboot the server:

Filesystem Security

There are several useful filesystem mount options that we can apply to certain filesystems in order to increase security. We can use the noexec option to prevent users running executable code, the nodev option to prevent users from creating block and character devices on a filesystem, and the nosuid option, to prevent users from creating SETUID and SETGID files. We’ll apply nodev, noexec and nosuid to /tmp, /var/tmp, and /dev/shm, and nodev to /home.

To see if the options are already set, run the following commands:

As you can see, out-of-the-box the mount options are not applied. This can be verified with the mount command:

First, I’ll add the required mount options to /etc/fstab, and verify:

Next, the filesystems can be remounted, and the new options verified with another mount command:

SELinux

SELinux is enabled and enforcing by default on CentOS 6.x. It should be left in this state and will provide mandatory access control to the system. More information on SELinux can be found here: http://www.nsa.gov/research/selinux. You can verify the state with the following commands:

If required, update the SELinux status by updating /etc/selinux/config as required, and setting the live SELinux status with setenforce:

Packages

Perform an audit of currently installed packages with the rpm command:

Review the output, and remove any packages that are not required with the yum erase command:

Running Services

Use the chkconfig command to list daemons currently enabled to start in the various runlevels:

If anything can be disabled, it should. Running a minimal set of services is paramount to providing security as it minimises the attack vectors available on a server. You can disable services with the following command:

We can also obtain a list of currently listening ports and their owning process with the netstat command:

Here we can see that the only TCP ports open are 22 (SSH: sshd) and 25 (SMTP: Postfix), as well as an established SSH session.

Checking Permissions

Several core system files should be checked for appropriate permissions and ownership.

If any of the permissions are not correct, fix them now. With /etc/shadow and /etc/gshadow, you may notice that it has 000 permissions. Any process acting upon these files must have root ownership so that it can read these files regarding of their permissions – this is fine and expected.

/boot/grub/grub.conf should be equally secure:

Boot-time Security

If you didn’t specify a bootloader password during operating system installation, you should do so now. You can verify if a password is set by grepping the grub configuration:

If no output is produced, then no password is set. To set the password, first generate an encrypted password with grub-md5-crypt:

Then, add the following to grub.conf:

By default, when the user enters single-user mode by passing the “single” option to the kernel during startup, the user will not be prompted for a password before being presented with a single-user root shell, and will also have the ability to perform an interactive startup:

To secure this, change SINGLE to /sbin/sulogin, and PROMPT to “no” to disable interactive startup with the following commands:

Verify the changes:

Additional Process Hardening

There are a great number of things we can do to fine-tune how the operating system handles errors, manages processes and resources, and so on. Core dumps should be disabled if they are not required. These memory-dumps can be used to glean confidential information.

Add the following to /etc/security/limits.conf:

Also, prevent SETUID programs from dumping core with the fs.suid_dumpable kernel parameter:

Execshield is made up of a number of kernel features to provide mitigation against buffer overflow attacks. Ensure that it is enabled with the following command:

If a value other than 1 is returned, update /etc/sysctl.conf appropriately.

Virtual memory regions should be randomly placed to make it hard to exploit via memory page predictability. The kernel.randomize_va_space parameter should be set to 2:

Network Stack Hardening

There are a myriad of kernel parameters that can tune the network stack. Some of the main ones are detailed below, where quick security wins can be gleaned. If the server is not acting as a router, set the net.ipv4.ip_forward flag to 0:

We can also check that ICMP redirects are disabled. Again, as the host is not acting as a router, there is no need to send these redirects as they could be exploited to send invalid ICMP redirects to other devices in an attempt to corrupt routing. By default, redirects are enabled:

Add the following lines to /etc/sysctl.conf:

Reload the parameters, and verify:

The appropriate accept_redirects variables should also be set to 0 and updated via the same mechanism, to prevent an attacker from updating the system’s routing table.

To provide an audit ability, we can log packets with an unroutable source address to the kernel log. This is done by setting net.ipv4.conf.all.log_martians and net.ipv4.conf.default.log_martians both to 1 and reloading the sysctl parameters.

ICMP echo and timestamp requests to broadcast and multicast addresses should also be ignored, by setting net.ipv4.icmp_echo_ignore_broadcasts to 1. Set net.ipv4.icmp_ignore_bogus_error_responses to 1 to ignore RFC-1122 non-compliant responses. RFC-recommended source route validation should be enabled by setting net.ipv4.conf.all.rp_filter and net.ipv4.conf.default.rp_filter to 1. Ensure that TCP SYN cookies are enabled to mitigate against SYN flood attacks:

If IPv6 is being used on the server, the appropriate net.ipv6 variables should be set too.

TCP Wrappers

TCP Wrappers provides access lists and standardised logging for services and is installed by default on a minimal CentOS 6.x build:

Services called from xinetd can make use of tcp_wrappers, as can any services linked against libwrap.so. For example, we can check that sshd supports tcp_wrappers by checking how it’s linked with ldd:

This shows that sshd is indeed linked against libwrap.so. Next, the /etc/hosts.allow and /etc/hosts.deny files should be populated with access lists defined by your requirements. For example, to allow all hosts in the 192.168.100.0/24 network access to sshd on the server, but disallow all other access to tcp_wrappers-enabled services, create /etc/hosts.allow with the following contents:

and /etc/hosts.deny with the following contents:

See ‘man 5 hosts_options‘ and ‘man 5 hosts_access‘ for further examples of rule syntax.

Host-based Firewall

Whilst a full discussion of the iptables firewall is outside the scope of this article, you should ensure that the iptables service is enabled:

If it isn’t, enable it and set it to automatically start in the appropriate runlevels:

If your server uses IPv6, then the ip6tables service should also be enabled. The default ruleset applied during a minimal installation of CentOS 6.x allows access to port 22/tcp from any host.

You can view the current iptables configuration with the iptables -L command:

Logging

Logging is one of the core aspects of a well-secured system. Without a good level of logging it is impossible to deal with suspicious behaviour, or even check for system events and errors. In CentOS 6.x, rsyslog is used for syslogging.

Verify that rsyslog is configured to start automatically:

The default logging configuration does a good job of capturing important system events. You should ensure that logs are also sent to a remote syslog server. The following configuration in /etc/rsyslog.conf sends all messages from all facilities at all priorities to the remote syslog server mars.local:

Logs should also be rotated and archived via logrotate (again, the default configuration does a good job of this but you may need to customise it for your needs). Logs should be monitored frequently to ensure that they do not consume too much space and that they are capturing relevant information. Permissions over log files will depend on how logs are managed at your site.

By default, the system audit daemon auditd will audit SELinux AVC denials, system logins, account modifications, and authentication events to /var/log/audit/audit.log. Configuring the system audit daemon is beyond the scope of this article, but is covered in great detail in the RHEL 6 Benchmark document available from http://cisecurity.org.

Access to cron/at

Cron and at access should be limited only to users that require it by populating the /etc/cron.allow and /etc/at.allow files as required, with one user per line that requires access. Or, /etc/cron.deny and /etc/at.deny can be used to specify who can’t access these facilities, with /etc/cron.allow and /etc/at.allow remaining empty. It is easier to manage the *.allow lists than *.deny, as each time a new user is added to the system, the *.deny files would need to be updated and this could potentially be forgotten.

SSH Configuration

You should ensure that sshd only offers SSH v2 service with the following command:

If “Protocol 2” is not returned, then update sshd_config and reload sshd with:

Verify that sshd is logging at an appropriate level for your site:

DEBUG may generate too much logging for your requirements, in which case you may want to drop back to LogLevel INFO. Also ensure that permissions on /etc/ssh/sshd_config are suitably restricted:

By default, X11 forwarding is enabled:

If there is no operational requirement to use X11 applications, this can safely be set to “no”.

The MaxAuthTries parameter should be set to 4 or less – this parameter specifies the maximum number of authentication attempts permitted per connection. When the login failure count reaches half the defined number, error messages will be written to the system logs detailing the failure (depending on your syslog configuration). Ensure that IgnoreRhosts is set to yes to force users to enter a password even if .rhosts and .shosts files are present. HostbasedAuthentication should also be set to no, to ignore .rhosts and /etc/hosts.equiv. These are slightly redundant if .rhosts support is already disabled in your PAM configuration, but is worth configuring in SSH as an additional layer of security.

By default, root can log in directly via SSH:

This parameter should be uncommented and its value changed to “no” to disallow root login via ssh. This requires administrators to authenticate using their own accounts, and then escalating to the root account via sudo or su. This provides an audit trail, and stops remote attackers attempting to log in directly via root and running password cracking tools against the root account. PermitEmptyPasswords should also be set to “no” to prevent the server allowing logins to accounts with empty password strings. There are many ways to further restrict SSH access to the system, and the sshd_config manual page should be consulted as the authoritative text on this matter.

Other Security Matters

You should review /etc/login.defs for adequate password parameters dependant upon your needs. The defaults specified in this file will only be applied if the user{add,mod} commands are used. If you use a different mechanism to add/modify users on your system, you will need to use the chage command to effect changes to those users.

Ensure that no users have empty passwords:

If any accounts to have an empty password, lock them with passwd -l until it can be determined why the account didn’t have a password configured.

You should verify that only the root account has UID zero:

If any accounts other than root are returned they should be investigated urgently as they will also have root privileges. You should also check for other duplicate UIDs and GIDs (scripts can be found in the CIS benchmarks).

There are a plethora of authentication-related options that can be defined via PAM (Pluggable Authentication Modules). Again, consult the CIS benchmark for a detailed discussion. Such functionality offered includes locking out users after a certain number of consecutive failed login attempts, enforcing strong passwords, limiting password reuse, and so on.

The system should be regularly monitored for world-writable files. World-writable files and directories are open to exploit as they can be modified and compromised by any user on the system. A simple DoS could involve writing to a world-writable file until it fills the filesystem on which it resides. A find command such as the following will display a list of such files, excluding links:

Review the resulting output and remove write access for the “other” category at your discretion with:

Be sure to consult any relevant documentation before doing this as changes to file permissions may break an application. Also be wary of changing permissions on device or socket files.

You should regularly review the SETUID programs that are installed on the system and investigate any that have been introduced into the system. Again, find is the command to use:

A similar command can be used to obtain a list of SETGID programs:

You should also ensure that /etc/motd and /etc/issue contain a suitably ominous message – something along the lines of “Authorised users only – all access may be monitored and logged” normally suffices.

Validate that the root PATH environment variable does not contain the current directory (.):

If an empty path (::), a trailing colon or current directory (.) is found, take steps to remove the offending configuration.

Sensible system administration and regular auditing will take care of a lot of the security issues surrounding file and directory ownership/permissions. For example, ensure that users home directories are only writable by the owner, unless required by some site-specific policy (a shared group home directory for some project, for example). Regular scripted find runs over the filesystems with appropriate monitoring should be applied to the system and acted upon.

Further Considerations

Whilst the host has now had a plethora of security options applied, it’s time to reboot and make sure all is well. You can also run nmap against the host to get a profile of ports open (and much more) and perform any service hardening as appropriate. Nessus can be used to provide vulnerability scans and is well renowned for being one of the best vulnerability scanners available. SAINT and SATAN are other security testing and reporting tools that can be used to test hosts for known vulnerabilities.

Host Based Intrusion Detection Systems (HIDS) are an excellent way of maintaining host operating system security. OSSEC is an open source HIDS that provides log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response, and can be run on most operating systems.

Linux Security Auditing Tool (LSAT) is a post-install security auditing tool for Linux. It checks many system configuration files and local network settings on the system for common security/configuration errors.

Conclusion

Hardening the Linux operating system is a complex task, and there are many aspects of the operating system that can be tuned and secured. This article has described many of the “quick-wins” from a security perspective, and provided pointers on where to find further information to take the security hardening further.

Only once the operating system has been correctly hardened should any applications be deployed to it. A secure operating system should be deployed alongside secure network infrastructure, well-written code, and so on, for it to be ultimately secure. That said, providing multiple layers of security will slow an attacker down, and prevent most attacks from occurring in the first place.

Further reading around SELinux, Pluggable Authentication Modules (PAM) and the Linux auditing framework is definitely recommended.

The CISecurity.org benchmarks provide a great deal of further information on these and other hardening topics, and provided a lot of the hardening guidelines I’ve used in this article.

Whilst it’s slightly older, the NSA Guide to the Secure Configuration of Red Hat Enterprise Linux 5 (http://www.nsa.gov/ia/_files/os/redhat/NSA_RHEL_5_GUIDE_v4.2.pdf) is still a very good read.

This article was previously published in Pentest Magazine.