rsyslog

Whilst configuring a central syslog server using rsyslog on CentOS 6, I was experiencing issues with dynamic log filenames being created. I had created a new filesystem, mounted at /var/syslog, for my logs, and configured rsyslog.conf with the following:

...
$template FILENAME,"/var/syslog/%HOSTNAME%/syslog.log"
*.* ?FILENAME
...

...

$template FILENAME,"/var/syslog/%HOSTNAME%/syslog.log"

*.* ?FILENAME

...

However, the dynamic logs were not being created, and instead the following error message was observed in the local /var/log/messages file (mars being the hostname):

Jun 13 23:30:01 mars rsyslogd-3000: Could not open dynamic file '/var/syslog/mars/syslog.log' [state -3000] - discarding message [try http://www.rsyslog.com/e/3000 ]

1	Jun 13 23:30:01 mars rsyslogd-3000: Could not open dynamic file '/var/syslog/mars/syslog.log' [state -3000] - discarding message [try http://www.rsyslog.com/e/3000 ]

A quick check of the SELinux context found the issue:

# ls -Zd /var/syslog
drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 /var/syslog
# ls -Zd /var/log
drwxr-xr-x. root root system_u:object_r:var_log_t:s0   /var/log

# ls -Zd /var/syslog

drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 /var/syslog

# ls -Zd /var/log

drwxr-xr-x. root root system_u:object_r:var_log_t:s0 /var/log

chcon to the rescue, referencing /var/log:

# chcon --reference /var/log /var/syslog

1	# chcon --reference /var/log /var/syslog

A restart of rsyslog later, and we were in business:

# ls -l /var/syslog/mars
total 4
-rw-------. 1 root root 445 Jun 13 23:33 syslog.log

# ls -l /var/syslog/mars

total 4

-rw-------. 1 root root 445 Jun 13 23:33 syslog.log

Note: it’s worth noting here that I updated my dynamically created file rule to be as follows (including the date too) as one huge file per host is not very useful:

$template FILENAME,"/var/syslog/%FROMHOST%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"

1	$template FILENAME,"/var/syslog/%FROMHOST%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"

Module: logging-server

Purpose: This module is used to configure my centralised rsyslog server, on CentOS 6.

File: logging-server/manifests/init.pp

Notes: This will refresh the rsyslog service if the configuration file is updated, and will also run restorecon to fix the SELinux context on the configuration file.

# logging-server/manifests/init.pp - configure syslog server
# 
# Essentially, this class only allows for rsyslog server configuration
# on CentOS 6
class logging-server {
  case $::operatingsystem {
    'CentOS': {
      case $::operatingsystemmajrelease {
        '6': {
          package { "rsyslog": ensure => "latest" }
          file { "/etc/rsyslog.conf":
               owner => 'root', group => 'root', mode => '0644',
               source => 'puppet:///modules/logging-server/rsyslog.conf',
               before => Service["rsyslog"],
               notify => Service["rsyslog"]
          }
          exec { "/sbin/restorecon /etc/rsyslog.conf":
            subscribe => File["/etc/rsyslog.conf"],
            refreshonly => true
          }
          service { "rsyslog":
            enable  => true,
            ensure  => running,
            require => Package["rsyslog"]
          } 
        }
        default: {
          fail('Operating system is unsupported by logging class')
        }
      }
    }
    default:  {
      fail('Operating system is unsupported by logging class')
    }
  }
}

# logging-server/manifests/init.pp - configure syslog server

# Essentially, this class only allows for rsyslog server configuration

# on CentOS 6

class logging-server {

case $::operatingsystem {

'CentOS': {

case $::operatingsystemmajrelease {

'6': {

package { "rsyslog": ensure => "latest" }

file { "/etc/rsyslog.conf":

owner => 'root', group => 'root', mode => '0644',

source => 'puppet:///modules/logging-server/rsyslog.conf',

before => Service["rsyslog"],

notify => Service["rsyslog"]

}

exec { "/sbin/restorecon /etc/rsyslog.conf":

subscribe => File["/etc/rsyslog.conf"],

refreshonly => true

}

service { "rsyslog":

enable => true,

ensure => running,

require => Package["rsyslog"]

}

default: {

fail('Operating system is unsupported by logging class')

}

default: {

fail('Operating system is unsupported by logging class')

}

File: logging-server/files/rsyslog.conf

Notes: This configuration allows rsyslog to receive messages over both UDP and TCP.

###########################################
# WARNING - THIS FILE IS MANAGED BY PUPPET
# ANY LOCAL CHANGES WILL BE LOST
###########################################
#
#
# rsyslog v5 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
#### RULES ####
$template FILENAME,"/var/syslog/%FROMHOST%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"
*.* ?FILENAME
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure
# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog
# Log cron stuff
cron.*                                                  /var/log/cron
# Everybody gets emergency messages
*.emerg                                                 *
# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler
# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

###########################################

# WARNING - THIS FILE IS MANAGED BY PUPPET

# ANY LOCAL CHANGES WILL BE LOST

###########################################

# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html

# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)

$ModLoad imklog # provides kernel logging support (previously done by rklogd)

#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception

$ModLoad imudp

$UDPServerRun 514

# Provides TCP syslog reception

$ModLoad imtcp

$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Use default timestamp format

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,

# not useful and an extreme performance hit

#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/

$IncludeConfig /etc/rsyslog.d/*.conf

#### RULES ####

$template FILENAME,"/var/syslog/%FROMHOST%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"

*.* ?FILENAME

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.* /dev/console

# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.

authpriv.* /var/log/secure

# Log all the mail messages in one place.

mail.* -/var/log/maillog

# Log cron stuff

cron.* /var/log/cron

# Everybody gets emergency messages

*.emerg *

# Save news errors of level crit and higher in a special file.

uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log

local7.* /var/log/boot.log

# ### begin forwarding rule ###

# The statement between the begin ... end define a SINGLE forwarding

# rule. They belong together, do NOT split them. If you create multiple

# forwarding rules, duplicate the whole block!

# Remote Logging (we use TCP for reliable delivery)

# An on-disk queue is created for this action. If the remote host is

# down, messages are spooled to disk and sent when it is up again.

#$WorkDirectory /var/lib/rsyslog # where to place spool files

#$ActionQueueFileName fwdRule1 # unique name prefix for spool files

#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)

#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown

#$ActionQueueType LinkedList # run asynchronously

#$ActionResumeRetryCount -1 # infinite retries if host is down

# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional

#*.* @@remote-host:514

# ### end of the forwarding rule ###

Toki Winter

Advanced UNIX for the experienced system administrator

Tag Archives: rsyslog

rsyslog - “Could not open dynamic file - discarding message” errors on CentOS 6

Puppet Module: logging-server: Centralised rsyslog Server