Debugging logcheck rule files

logcheck is a neat little log file monitoring tool I use on all of my machines.

I recently noticed however that I hadn't received any logcheck messages in a while from one of my servers. Either that was a sign that things were going really well or, more likely, that logcheck wasn't producing any output anymore.

Manually logging an error to syslog

Here's what I did to force a message to be printed to the logs:

logger -p kern.error This is a test

Which I would expect to produce this logcheck notice:

Dec 20 15:34:08 hostname username: This is a test

Unfortunately, that didn't happen on the next scheduled run.

Forcing a logcheck run

To rule out the following:

  • mail not getting through
  • cron not running

I ran logcheck manually:

sudo -u logcheck /usr/sbin/logcheck -o -d >& logcheck.out

Looking at the output file however, my test message still wasn't there. Either logcheck was broken or one of my rule files was swallowing everything.

Finding the broken rule file

To find the broken rule, I started by ignoring rules defined in /etc/logcheck/ignore.d.server/ and /etc/logcheck/ignore.d.workstation/ by running logcheck in paranoid mode:

logger -p kern.error This is a test  
sudo -u logcheck /usr/sbin/logcheck -o -d -p >& logcheck.out

This worked, so I then ran logcheck in server mode:

logger -p kern.error This is a test  
sudo -u logcheck /usr/sbin/logcheck -o -d -s >& logcheck.out

Given that this also worked, it meant that the offending rule file was in /etc/logcheck/ignore.d.workstation/. So I moved all of my custom local-* rule files out of the way and ran logcheck in workstation mode:

logger -p kern.error This is a test  
sudo -u logcheck /usr/sbin/logcheck -o -d -w >& logcheck.out

Once I verified that this worked, I started to put my local files back one by one until it broke again. Then slowly removed lines from the offending file until it worked.

Solution to my problem

It turns out that one of my rule files had a line like this:

path != NULL || column != NULL

Escaping the pipe symbols with backslashes solved the problem:

path != NULL \|\| column != NULL

Maybe I should periodically print a message to syslog to make sure that logcheck is still working...

Ignoring files in git repositories

According to the man page, there are three ways to exclude files from being tracked by git.

Shared list of files to ignore

The most well-known way of preventing files from being part of a git branch is to add such files in .gitignore. (This is analogous to CVS' .cvsignore files.)

Here's an example:

*.generated.html  
/config.php

The above ignore list will prevent automatically generated HTML files from being committed by mistake to the repository. Because this is useful to all developers on the project, .gitignore is a good place for this.

The next line prevents the local configuration file from being tracked by git, something else that all developers will want to have.

One thing to note here is the use of a leading slash character with config.php. This is to specifically match the config file in the same directory as the .gitignore file (in this case, the root directory of the repository) but no other. Without this slash, the following files would also be ignored by git:

/app/config.php  
/plugins/address/config.php  
/module/config.php

Local list (specific to one project)

For those custom files that you don't want version controlled but that others probably don't have or don't want to automatically ignore, git provides a second facility: .git/info/exclude

It works the same way as .gitignore but be aware that this list is only stored locally and only applies to the repository in which it lives.

(I can't think of a good example for when you'd want to use this one because I don't really use it. Feel free to leave a comment if you do use it though, I'm curious to know what others do with it.)

Local list (common to all projects)

Should you wish to automatically ignore file patterns in all of your projects, you will need to use the third gitignore method: core.excludesfile

Put this line in your ~/.gitconfig:

[core]  
excludesfile = /home/username/.gitexcludes

(you need to put the absolute path to your home directory, ~/ will not work here unless you use git 1.6.6 or later)

and then put the patterns to ignore in ~/.gitexcludes. For example, this will ignore the automatic backups made by emacs when you save a file:

*~

This is the ideal place to put anything that is generated by your development tools and that doesn't need to appear in your project repositories.