Debugging logcheck rule files

logcheck is a neat little log file monitoring tool I use on all of my machines.

I recently noticed however that I hadn't received any logcheck messages in a while from one of my servers. Either that was a sign that things were going really well or, more likely, that logcheck wasn't producing any output anymore.

Manually logging an error to syslog

Here's what I did to force a message to be printed to the logs:

logger -p kern.error This is a test

Which I would expect to produce this logcheck notice:

Dec 20 15:34:08 hostname username: This is a test

Unfortunately, that didn't happen on the next scheduled run.

Forcing a logcheck run

To rule out the following:

• mail not getting through
• cron not running

I ran logcheck manually:

sudo -u logcheck /usr/sbin/logcheck -o -d >& logcheck.out

Looking at the output file however, my test message still wasn't there. Either logcheck was broken or one of my rule files was swallowing everything.

Finding the broken rule file

To find the broken rule, I started by ignoring rules defined in /etc/logcheck/ignore.d.server/ and /etc/logcheck/ignore.d.workstation/ by running logcheck in paranoid mode:

logger -p kern.error This is a test
sudo -u logcheck /usr/sbin/logcheck -o -d -p >& logcheck.out

This worked, so I then ran logcheck in server mode:

logger -p kern.error This is a test
sudo -u logcheck /usr/sbin/logcheck -o -d -s >& logcheck.out

Given that this also worked, it meant that the offending rule file was in /etc/logcheck/ignore.d.workstation/. So I moved all of my custom local-* rule files out of the way and ran logcheck in workstation mode:

logger -p kern.error This is a test
sudo -u logcheck /usr/sbin/logcheck -o -d -w >& logcheck.out

Once I verified that this worked, I started to put my local files back one by one until it broke again. Then slowly removed lines from the offending file until it worked.

Solution to my problem

It turns out that one of my rule files had a line like this:

path != NULL || column != NULL

Escaping the pipe symbols with backslashes solved the problem:

path != NULL \|\| column != NULL

Maybe I should periodically print a message to syslog to make sure that logcheck is still working...