Excluding files from git archive exports using gitattributes

git archive provides an easy way of producing a tarball directly from a project's git branch.

For example, this is what we use to build the Mahara tarballs:

git archive --format=tar --prefix=mahara-${VERSION}/ ${RELEASETAG} | bzip2 -9 > ${CURRENTDIR}/mahara-${RELEASE}.tar.bz2

If you do this however, you end up with the entire contents of the git branch, including potentially undesirable files like .gitignore.

There is an easy, though not very well-documented, way of specifying files to exclude from such exports: gitattributes.

This is what the Mahara .gitattributes file looks like:

/test export-ignore
.gitattributes export-ignore
.gitignore export-ignore

With this file in the root directory of our repository, tarballs we generate using git archive no longer contain the selenium tests or the git config files.

If you start playing with this feature however, make sure you commit the .gitattributes file to your repository before running git archive. Otherwise the settings will not be picked up by git archive.

Restoring lost RT tickets

Request Tracker (also know as RT), the defacto Open Source ticket tracking software, has a special status value to get rid of spam tickets: deleted. It does exactly what it's supposed to do, that is get rid of these tickets quickly.

Unfortunately, it's very easy to accidentally delete a bunch of non-spam tickets via the bulk update interface. Restoring these deleted tickets is not straightforward.

Good news: none of the lost tickets were removed from the database.
Bad news: you can't search for deleted tickets.

Finding the lost tickets

If you have a copy of the page before you did the bulk update (perhaps in your browser cache?) then you've got all you need. Otherwise, you'll need to access the database directly to get a list of ticket IDs.

First of all, find the user ID of the person who marked of all these tickets as deleted (42 in this example), then issue the following query:

 SELECT id, lastupdated, subject
  FROM tickets
 WHERE status = 'deleted' AND
       lastupdatedby = 42
ORDER BY lastupdated;

You may want to filter some more using the lastupdated timestamp if you know roughly when the deletion occurred.

Restoring deleted tickets

Since all of the tickets are still in the database, all you need to do is to go to each of them directly (e.g. http://www.example.com/Ticket/Display.html?id=12345 for Ticket #12345) and change the status back to "open".

Ignoring files in git repositories

According to the man page, there are three ways to exclude files from being tracked by git.

Shared list of files to ignore

The most well-known way of preventing files from being part of a git branch is to add such files in .gitignore. (This is analogous to CVS' .cvsignore files.)

Here's an example:

*.generated.html
/config.php

The above ignore list will prevent automatically generated HTML files from being committed by mistake to the repository. Because this is useful to all developers on the project, .gitignore is a good place for this.

The next line prevents the local configuration file from being tracked by git, something else that all developers will want to have.

One thing to note here is the use of a leading slash character with config.php. This is to specifically match the config file in the same directory as the .gitignore file (in this case, the root directory of the repository) but no other. Without this slash, the following files would also be ignored by git:

/app/config.php
/plugins/address/config.php
/module/config.php

Local list (specific to one project)

For those custom files that you don't want version controlled but that others probably don't have or don't want to automatically ignore, git provides a second facility: .git/info/exclude

It works the same way as .gitignore but be aware that this list is only stored locally and only applies to the repository in which it lives.

(I can't think of a good example for when you'd want to use this one because I don't really use it. Feel free to leave a comment if you do use it though, I'm curious to know what others do with it.)

Local list (common to all projects)

Should you wish to automatically ignore file patterns in all of your projects, you will need to use the third gitignore method: core.excludesfile

Put this line in your ~/.gitconfig:

[core]
excludesfile = /home/username/.gitexcludes

(you need to put the absolute path to your home directory, ~/ will not work here unless you use git 1.6.6 or later)

and then put the patterns to ignore in ~/.gitexcludes. For example, this will ignore the automatic backups made by emacs when you save a file:

*~

This is the ideal place to put anything that is generated by your development tools and that doesn't need to appear in your project repositories.

Debugging logcheck rule files

logcheck is a neat little log file monitoring tool I use on all of my machines.

I recently noticed however that I hadn't received any logcheck messages in a while from one of my servers. Either that was a sign that things were going really well or, more likely, that logcheck wasn't producing any output anymore.

Manually logging an error to syslog

Here's what I did to force a message to be printed to the logs:

logger -p kern.error This is a test

Which I would expect to produce this logcheck notice:

Dec 20 15:34:08 hostname username: This is a test

Unfortunately, that didn't happen on the next scheduled run.

Forcing a logcheck run

To rule out the following:

• mail not getting through
• cron not running

I ran logcheck manually:

sudo -u logcheck /usr/sbin/logcheck -o -d >& logcheck.out

Looking at the output file however, my test message still wasn't there. Either logcheck was broken or one of my rule files was swallowing everything.

Finding the broken rule file

To find the broken rule, I started by ignoring rules defined in /etc/logcheck/ignore.d.server/ and /etc/logcheck/ignore.d.workstation/ by running logcheck in paranoid mode:

logger -p kern.error This is a test
sudo -u logcheck /usr/sbin/logcheck -o -d -p >& logcheck.out

This worked, so I then ran logcheck in server mode:

logger -p kern.error This is a test
sudo -u logcheck /usr/sbin/logcheck -o -d -s >& logcheck.out

Given that this also worked, it meant that the offending rule file was in /etc/logcheck/ignore.d.workstation/. So I moved all of my custom local-* rule files out of the way and ran logcheck in workstation mode:

logger -p kern.error This is a test
sudo -u logcheck /usr/sbin/logcheck -o -d -w >& logcheck.out

Once I verified that this worked, I started to put my local files back one by one until it broke again. Then slowly removed lines from the offending file until it worked.

Solution to my problem

It turns out that one of my rule files had a line like this:

path != NULL || column != NULL

Escaping the pipe symbols with backslashes solved the problem:

path != NULL \|\| column != NULL

Maybe I should periodically print a message to syslog to make sure that logcheck is still working...

Reducing website bandwidth usage

There are lots of excellent tools to help web developers optimise their websites.

Here are two simple things you have no excuse for overlooking on your next project.

HTML, XML, Javascript and CSS files

One of the easiest ways to speed up a website (often to a surprising degree) is to turn on compression of plaintext content through facilities like mod_deflate or the Gzip Module.

Here's the Apache configuration file I normally use:

AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript text/xml application/x-javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch "MSIE 6" no-gzip dont-vary
BrowserMatch ^Mozilla/4\.0[678] no-gzip

Images

As far as images go, the following tools will reduce file sizes through lossless compression (i.e. with no visual changes at all):

• gifsicle -O2 -b image.gif
• jpegoptim -p --strip-all image.jpg
• optipng -o7 -q image.png

(An alternative to optipng is pngcrush.)

Note that the --strip-all argument to jpegoptim will remove any EXIF/comments tags that may be present.