Category Archives: UNIX & Linux How Tos and Tutorials

SELinux: Allowing httpd to Listen on Non-Standard Ports

If you attempt to have httpd listen on a non-standard port (we’ll get to what’s “standard” and what isn’t in a moment) on an SELinux enabled host, SELinux will deny the request. Let’s try it out (on a CentOS 6.4 box):

Kaboom! As you can see, the error message is quite clear “(13)Permission denied: make_sock: could not bind to address”. Let’s take a look at the audit logs and see what’s going on:

There are two sets of messages there, one for the IPv4 bind, the other for the IPv6 bind. The AVC message actually makes it clear what’s going on:

type=AVC msg=audit(1371477046.382:20195): avc: denied { name_bind } for pid=16429 comm="httpd" src=8585 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket

You can see that the violation is from source context (scontext) unconfined_u:system_r:httpd_t which is the httpd process, and the target context (tcontext) is system_u:object_r:port_t with target class (tclass) tcp_socket. The denied port (8585 in this case) is also logged in the src field.

SELinux will only allow httpd to bind to ports of type http_port_t. Here, we can see that httpd is attempting to bind to a port of type port_t - which is going to be denied by the policy. We need to define port 8585 as a port of type http_port_t. semanage is the tool for that - and you’ll need the policycoreutils-python package installed:

First, let’s look at which ports are currently defined as type http_port_t:

And add our desired listen port, 8585/tcp to the list:

Looks good. Fire up httpd:

And test with nc or similar:

 

rsyslog - “Could not open dynamic file - discarding message” errors on CentOS 6

Whilst configuring a central syslog server using rsyslog on CentOS 6, I was experiencing issues with dynamic log filenames being created. I had created a new filesystem, mounted at /var/syslog, for my logs, and configured rsyslog.conf with the following:

However, the dynamic logs were not being created, and instead the following error message was observed in the local /var/log/messages file (mars being the hostname):

A quick check of the SELinux context found the issue:

chcon to the rescue, referencing /var/log:

A restart of rsyslog later, and we were in business:

Note: it’s worth noting here that I updated my dynamically created file rule to be as follows (including the date too) as one huge file per host is not very useful:

 

Running Puppet Master under Apache and Passenger - CentOS 6.4

I have been running my puppetmaster using the embedded WEBrick server for a while. I decided it was time to migrate to something a little more robust - namely Apache and Passenger. I loosely followed the documentation available on the Puppet site, although that covers Passenger 3.0.x and I’m using 4.0.x, and the supplied Apache configuration does not work. There were also a few other changes I had to make along the way to suit my configuration requirements. My puppetmaster is running CentOS 6.4.

Continue reading →

MongoDB: First Steps: Installation, Configuration and Authentication

MongoDB is one of the more popular NoSQL databases available. NoSQL databases do not use typical relational database structure, nor do they typically rely on SQL for queries. MongoDB stores data in JSON-style documents (which themselves can be stored within collections) and allows for dynamic storage of data. For example, we do not need to create a schema, configure relations between tables, and so on. We just create our database and populate it with data on the fly. The data can be as complex or as simple (just key/value pairs) as our requirements dictate. MongoDB is also highly-scalable via the use of shard-based clustering.

I am a strict wearer of an RDBMS hat, and this is my first foray into the world of NoSQL. This first article on MongoDB will cover initial installation, configuration and testing. I’ll also dabble with authentication. Further articles will cover the configuration of replica sets and other administrative features. I’m a sysadmin, not a developer, so I’ll be looking more at how to administer MongoDB rather than how to actually use it.

Let’s get on with the installation.

Continue reading →

Adding Puppet DB to a CentOS 6.4-based puppetmaster

After configuring my puppetmaster to run on Passenger on Apache, the next step was to add Puppet DB so that I could take advantage of exported resources, as well as have a central store of facts/catalogs that I could query. I am configuring Puppet DB to run on the same host as my puppetmaster, on CentOS 6.4.

Whilst you can use the embedded HSQLDB database, I opted for the scale-out offered by a backend PostgreSQL database (which is recommended by Puppet Labs for deployments of more than 100 nodes). MySQL is not supported, as it doesn’t support recursive queries.

I installed PostgreSQL using my postgresql::server class. The class is shown below (you can see it includes postgres, which does nothing more than install the PostgreSQL client package). Essentially, it covers the installation of the postgresql-server package on the host, initialisation tasks, ensuring an appropriate pg_hba.conf is installed (for example, replacing localhost ident authentication methods with password), registering the service, and starting PostgreSQL:

To install, it was as simple as:

Next, I created a PostgreSQL user and database:

The user is created so that it cannot create databases (-D), or roles (-R) and doesn’t have superuser privileges (-S) - it’ll prompt for a password (-P). Let’s assume a password of “s3cr3tp@ss” has been used. The database is created and owned (-O) by the puppetdb user.

Access to the database can then be tested:

Next, install the puppetdb and puppetdb-terminus packages. I installed from the Puppet Labs yum repository:

Configure /etc/puppetdb/conf.d/database.ini as appropriate (it ships configured for HSQLDB):

Create /etc/puppet/puppetdb.conf, as appropriate for the Jetty configuration on the host. You can check the Jetty configuration under /etc/puppetdb/conf.d/jetty.ini. Here’s puppetdb.conf (my puppetmaster runs on sun.local):

Create /etc/puppet/routes.yaml:

Finally, update /etc/puppet/puppet.conf:

Once all steps are complete, restart your puppetmaster. As I’m running under Passenger:

Run a puppet agent --test from once of your nodes (or wait for your scheduled runs):

As you can see, I ran from venus.local. Let’s check that the data is being stored in puppetdb - by the time I got there three more nodes had already had an agent run:

Awesome! Let’s get something meaningful. Suppose we want to know what OS is running on all of our nodes:

An important note: ensure that all of your puppet nodes can connect to the puppetdb server on the appropriate SSL-enabled Jetty port (by default, 8081) so they can report in.

With Puppet DB now configured, I can start writing some reports, and using exported resources.

WordPress: Avoiding Infinite Recursion with mod_rewrite and mod_fastcgi

Whilst converting tokiwinter.com away from mod_php to mod_fastcgi/PHP-FPM, I experienced the following error:

Enabling LogLevel debug for the VirtualHost showed the following extra detail:

We can see that the rewrites required for WordPress in .htaccess are interfering with the correct operation of mod_fastcgi. The net result - HTTP 500 Internal Server Error for all our clients.

The fix is easy enough - add the following additional rewrite to .htaccess for the WordPress installation:

i.e. if the request URI is a mod_fastcgi request, do not apply any rewrites. My complete .htaccess file in my WordPress VirtualHost is now:

This works for me with the following VirtualHost configuration:

Your mileage may well vary.

How to Load Balance Tomcat with Apache HTTPD and mod_jk

In the following article, I’ll demonstrate how we can use Apache HTTPD to load balance across two Apache Tomcat instances. Whilst in this example the Apache HTTPD load balancer is a single point of failure, we could implement (although outside the scope of this article) a failover HTTPD instance clustered using one of the many available clustering stacks (RHCS, or something more lightweight like keepalived). Then, we’d have a highly-available load balancer.

There are a few ways we can use HTTPD to load balance, via the use of loadable modules. mod_proxy_http is the simplest, and can be used to load balance any service that “speaks” plain HTTP. mod_proxy_ajp is a simple AJP balancer module, with a slight performance increase of mod_proxy_http. However, mod_proxy_ajp is purported to be rather buggy when compared with other similar modules. Hence, we’ll use mod_jk. This is a very active module developed alongside Tomcat, and in many years of working with it I’ve never experienced a major bug or a configuration requirement it couldn’t handle.

Continue reading →

SELinux: Allowing HTTPD to Connect to PHP-FPM

When running PHP-FPM (PHP FastCGI Process Manager), it can be configured to listen on a UNIX socket, or a TCP port. When using the latter on an SELinux enabled system, you will receive HTTP 500 Internal Server Errors if SELinux is not configured correctly.

For example, on my system, I’m using the following directive (in my example.com VirtualHost):

However, the default value of the httpd_can_network_connect SELinux boolean is false, or off. Therefore, httpd is unable to connect to the PHP-FPM pool listening on 127.0.0.1:9000.

You will see AVC denial messages in /var/log/audit/audit.log such as:

To fix this issue, set the httpd_can_network_connect SELinux boolean to true, or on, remembering the -P option so that this change persists across system reboots.

 

MySQL Cluster: Adding New Data Nodes Online

MySQL Cluster has a pretty cool feature that allows you to add new data nodes whilst the cluster is online, thus avoiding any downtime. This is incredibly useful for scaling out the data nodes and adding additional node groups. In this article, I’ll show how to add two new data nodes to an existing cluster that has two data nodes defined. I’ll also explain what needs to happen after the configuration change to ensure that any existing data is correctly partitioned across the new nodes.

Continue reading →

Solaris Cluster 4.1 Part Four: Highly Available Containers

Introduction

The previous article covered the configuration of two resource groups, each containing a failover zpool for use as the zonepath to a highly-available zone, and a failover IP address to be assigned to each zone. The two zones were also configured and installed, and we verified that they could be booted on either node of the cluster, provided that the storage had been failed over appropriately and was available on the node where the zone was being booted.

This final part in the series will cover the incorporation of the zone boot/shutdown/failover into the cluster framework, as well as the configuration of two iPlanet resources to illustrate how Solaris Cluster can manage SMF services deployed within a highly-available Solaris zone.

Highly-Available Zones

First, install the ha-zones data service, if you haven’t done so already. I installed the full cluster package suite, so already have all data services at my disposal:

Register the SUNW.gds resource type:

This is the Generic Data Service that is utilised by SUNWsczone (HA for Solaris Containers) for deploying highly-available zones. SUNWsczone supplies three highly-available mechanisms for zone deployment - sczbt (zone boot - used to start/stop/failover zones), sczsh (zone script resource - used for deploying highly-available services within zones, with start/stop scripts to control them) and sczsmf (zone SMF resource, used for deploying highly-available services within zones, with SMF services to control them). We’ll be using both sczbt and sczsmf.

Continue reading →