Feeding the Cloud

Integrating Launchpad and Gerrit Code Review

2011-05-30T18:30:00.002+12:00

While talking about how Mahara uses Gerrit and Gitorious for code reviews, I forgot to include details on how to link bugs in the commit logs to our bug tracker (Launchpad).

Links to Launchpad in the Gerrit review UI

Turning the bug numbers in git commit messages to links was just a matter of adding this to our etc/gerrit.config file.

[commentlink "launchpad"]
  match = "([Bb]ug\\s+#?)(\\d+)"
  link = https://bugs.launchpad.net/mahara/+bug/$2

All mentions of a bug number like "(bug #1234)" or "bug 5678" in a commit message or Gerrit review will be converted into links to the appropriate page on Launchpad.

Updating Launchpad bugs after Gerrit merges

Linking the original bug reports in Launchpad to committed code (and its review in Gerrit) was a little more complicated. It makes use of the hooks available in Gerrit.

The first step was to create a new Launchpad account for these automated updates and to confirm the email address it will be sending emails from. In our case, the account is MaharaBot.

Then, we wrote a custom hook script (inspired by one for JIRA) which we placed in ~/mahara_reviews/etc/gerrit.config.

For an example of what it sends to the bug tracker, see bug 788457.

Code reviews with Gerrit and Gitorious

2011-04-17T17:00:00.009+12:00

The Mahara project has just moved to mandatory code reviews for every commit that gets applied to core code.

Here is a description of how Gerrit Code Review, the peer-review system used by Android, was retrofitted into our existing git repository on Gitorious.

(If you want to know more about Gerrit, listen to this FLOSS Weekly interview.)

Replacing existing Gitorious committers with a robot

The first thing to do was to log into Gitorious and remove commit rights from everyone in the main repository. Then I created a new maharabot account with a password-less SSH key (stored under /home/gerrit/.ssh/) and made that new account the sole committer.

This is to ensure that nobody pushes to the repository by mistake since all of these changes would be overwritten by Gerrit.

Basic Gerrit installation

After going through the installation instructions, I logged into the Gerrit admin interface and created a new "mahara" project.

I picked the "merge if necessary" submit action because "cherry-pick" would disable dependency tracking which is quite a handy feature.

Reverse proxy using Nginx

Since we wanted to offer Gerrit over HTTPS, I decided to run it behind an Nginx proxy. This is the Nginx configuration I ended up with:

server {
     listen 443;
     server_name reviews.mahara.org;
     add_header Strict-Transport-Security max-age=15768000;

     ssl on;
     ssl_certificate  /etc/ssl/certs/reviews.mahara.org.crt;
     ssl_certificate_key  /etc/ssl/certs/reviews.mahara.org.pem;

     ssl_session_timeout  5m;
     ssl_session_cache shared:SSL:1m;

     ssl_protocols  TLSv1;
     ssl_ciphers  HIGH:!ADH;
     ssl_prefer_server_ciphers   on;

     location / {
             proxy_pass   http://127.0.0.1:8081;
             proxy_set_header X-Forwarded-For $remote_addr;
             proxy_set_header Host $host;
     }
}

Things to note:

An HTTP to HTTPS redirection is not provided.

The HSTS headers indicates to modern browsers that this URL should only be accessed via HTTPS.

Only strong SSL ciphers are enabled.

Before proxying requests to Gerrit, Nginx adds a few headers to identify the origin of the request. The Host header in particular must be present otherwise built-in Gerrit redirections will fail.

Mail setup

To enable Gerrit to email reviewers and committers, I installed Postfix and used "reviews.mahara.org" as the "System mail name".

Then I added the following to /home/gerrit/mahara_reviews/etc/gerrit.config:

[user]
 email = "gerrit@reviews.mahara.org"

to fix the From address in outgoing emails.

Init script and cron

Following the installation instructions, I created these symlinks:

ln -s /home/gerrit/mahara_reviews/bin/gerrit.sh /etc/init.d/gerrit
cd /etc/rc2.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc3.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc4.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc5.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc0.d && ln -s ../init.d/gerrit K21gerrit
cd /etc/rc1.d && ln -s ../init.d/gerrit K21gerrit
cd /etc/rc6.d && ln -s ../init.d/gerrit K21gerrit

and put the following settings into /etc/default/gerritcodereview:

GERRIT_SITE=/home/gerrit/mahara_reviews
GERRIT_USER=gerrit
GERRIT_WAR=/home/gerrit/gerrit.war

to automatically start and stop Gerrit.

I also added a cron job in /etc/cron.d/gitcleanup to ensure that the built-in git repository doesn't get bloated:

MAILTO=admin@example.com
20 4 * * * gerrit GIT_DIR=/home/gerrit/mahara_reviews/git/mahara.git git gc --quiet

Configuration enhancements

To allow images in change requests to be displayed inside the browser, I marked them as safe in /home/gerrit/mahara_reviews/etc/gerrit.config:

[mimetype "image/*"]
 safe = true

Another thing I did to enhance the review experience was to enable the gitweb repository browser:

apt-get install gitweb

and to make checkouts faster by enabling anonymous Git access:

[gerrit]
  canonicalGitUrl = git://reviews.mahara.org/git/
[download]
  scheme = ssh
  scheme = anon_http
  scheme = anon_git

which requires that you have a git daemon running and listening on port 9418:

apt-get install git-daemon-run
ln -s /home/gerrit/mahara_reviews/git/mahara.git /var/cache/git/
touch /home/gerrit/mahara_reviews/git/mahara.git/git-daemon-export-ok

Finally, I included the Mahara branding in the header and footer of each page by providing valid XHTML fragments in /home/gerrit/mahara_reviews/etc/GerritSiteHeader.html and GerritSiteFooter.html.

Initial import and replication

Once Gerrit was fully working, I performed the initial code import by using my administrator account to push the exiting Gitorious branches to the internal git repository:

git remote add gerrit ssh://username@reviews.mahara.org:29418/mahara
git push gerrit 1.2_STABLE
git push gerrit 1.3_STABLE
git push gerrit master

Note that I had to temporarily disable "Require Change IDs" in the project settings in order to import the old commits which didn't have these.

To replicate the internal Gerrit repository back to Gitorious, I created a new /home/gerrit/mahara_reviews/etc/replication.config file:

[remote "gitorious"]
 url = gitorious.org:mahara/${name}.git
 push = +refs/heads/*:refs/heads/*
 push = +refs/tags/*:refs/tags/*

(The ${name} variable is required even when you have a single project.)

Contributor instructions

This is how developers can get a working checkout of our code now:

git clone git://gitorious.org/mahara/mahara.git
cd mahara
git remote add gerrit ssh://username@reviews.mahara.org:29418/mahara
git fetch gerrit
scp -p -P 29418 reviews.mahara.org:hooks/commit-msg .git/hooks/

and this is how they can submit local changes to Gerrit:

git push gerrit HEAD:refs/for/master

Anybody can submit change requests or comment on them but make sure you do not have the Cookie Pie Firefox extension installed or you will be unable to log into Gerrit.

Taking the max/min of two columns in PostgreSQL

2010-10-11T14:00:00.002+13:00

As part of a database view, I found myself wanting to get Postgres to display values from one of two columns, whichever was the largest.

My first attempt was:

SELECT id, MAX(column1, column2) FROM table1;

which of course didn't work because MAX() is an aggregate function which only takes a single parameter.

What I was looking for instead, is something that was introduced in 8.1:

SELECT id, GREATEST(column1, column2) FROM table1;

where GREATEST() (and its opposite: LEAST()) is a conditional expression.

Cherry-picking a range of git commits

2010-07-20T23:30:00.002+12:00

The cherry-pick command in git allows you to copy commits from one branch to another, one commit at a time. In order to copy more than one commit at once, you need a different approach.

Cherry-picking a single commit

Say we have the following repository composed of three branches (master, feature1 and stable):

$ git tree --all
* d9484311 (HEAD, master) Delete test file
* 4d4a0da8 Add a test file
| * 5753515c (stable) Add a license
| * 4b95278e Add readme file
|/
| * a37658bd (feature1) Add fourth file
| * a7785c10 Add lines to 3rd file
| * 7f545188 Add third file
| * 2bca593b Add line to second file
| * 0c13e436 Add second file
|/
* d3199755 Add a line
* b58d925c Initial commit

The "git tree" command is an alias I defined in my ~/.gitconfig:

[alias]
tree = log --oneline --decorate --graph

To copy the license file (commit 5753515c) to the master branch then we simply need to run:

$ git checkout master
$ git cherry-pick 5753515c
Finished one cherry-pick.
[master 08ff7d4] Add a license
1 files changed, 676 insertions(+), 0 deletions(-)
create mode 100644 COPYING

and the repository now looks like this:

$ git tree --all
* 08ff7d4a4 (HEAD, master) Add a license
* d94843113 Delete test file
* 4d4a0da88 Add a test file
| * 5753515c (stable) Add a license
| * 4b95278e Add readme file
|/
| * a37658bd (feature1) Add fourth file
| * a7785c10 Add lines to 3rd file
| * 7f545188 Add third file
| * 2bca593b Add line to second file
| * 0c13e436 Add second file
|/
* d3199755 Add a line
* b58d925c Initial commit

Cherry-picking a range of commits

In order to only take the third file (commits a7785c10 and 7f545188) from the feature1 branch and add it to the stable branch, I could cherry-pick each commit separately, but there is a faster way if you need to cherry-pick a large range of commits.

First of all, let's create a new branch which ends on the last commit we want to cherry-pick:

$ git branch tempbranch a7785c10
$ git tree --all
* 08ff7d4a (HEAD, master) Add a license
* d9484311 Delete test file
* 4d4a0da8 Add a test file
| * 5753515c (stable) Add a license
| * 4b95278e Add readme file
|/
| * a37658bd (feature1) Add fourth file
| * a7785c10 (tempbranch) Add lines to 3rd file
| * 7f545188 Add third file
| * 2bca593b Add line to second file
| * 0c13e436 Add second file
|/
* d3199755 Add a line
* b58d925c Initial commit

Now we'll rebase that temporary branch on top of the stable branch:

$ git rebase --onto stable 7f545188^ tempbranch
First, rewinding head to replay your work on top of it...
Applying: Add third file
Applying: Add lines to 3rd file
$ git tree --all
* ec488677 (HEAD, tempbranch) Add lines to 3rd file
* a85e5281 Add third file
* 5753515c (stable) Add a license
* 4b95278e Add readme file
| * 08ff7d4a (master) Add a license
| * d9484311 Delete test file
| * 4d4a0da8 Add a test file
|/
| * a37658bd (feature1) Add fourth file
| * a7785c10 Add lines to 3rd file
| * 7f545188 Add third file
| * 2bca593b Add line to second file
| * 0c13e436 Add second file
|/
* d3199755 Add a line
* b58d925c Initial commit

All that's left to do is to make stable point to the top commit of tempbranch and delete the old branch:

$ git checkout stable
Switched to branch 'stable'
$ git reset --hard tempbranch
HEAD is now at ec48867 Add lines to 3rd file
$ git tree --all
* ec488677 (HEAD, tempbranch, stable) Add lines to 3rd file
* a85e5281 Add third file
* 5753515c Add a license
* 4b95278e Add readme file
| * 08ff7d4a (master) Add a license
| * d9484311 Delete test file
| * 4d4a0da8 Add a test file
|/
| * a37658bd (feature1) Add fourth file
| * a7785c10 Add lines to 3rd file
| * 7f545188 Add third file
| * 2bca593b Add line to second file
| * 0c13e436 Add second file
|/
* d3199755 Add a line
* b58d925c Initial commit
$ git branch -d tempbranch
Deleted branch tempbranch (was ec48867).

It would be nice to be able to do it without having to use a temporary branch, but it still beats cherry-picking everything manually.

Another approach

Another way to achieve this is to use the format-patch command to output patches for the commits you are interested in copying to another branch and then using the am command to apply them all to the target branch:

$ git format-patch 7f545188^..a7785c10
0001-Add-third-file.patch
0002-Add-lines-to-3rd-file.patch
$ git am *.patch

Update: looking forward to git 1.7.2

According to a few people who were nice to point this out in a comment, version 1.7.2 of git, which is going to be released soon, will have support for this in cherry-pick:

git cherry-pick 7f545188^..a7785c10

Handling email-based events safely

2010-06-14T17:20:00.000+12:00

Mahara.org users have recently witnessed the unfortunate effects of a bug in the Mahara event handling: the Mahara cron job got stuck in an email loop and kept on sending the same forum post over and over again.

Here is what the affected code used to look like (edited for clarity):

db_begin();
$activities = get_records('activity_queue');
foreach ($activities as $activity) {
  handle_activity($activity);
}
delete_records('activity_queue');
db_commit();

One of the problems with this code was that if the handle_activity() function threw an exception, it would interfere with the processing of the entire queue. So it got fixed in the following way:

db_begin();
$activities = get_records('activity_queue');
foreach ($activities as $activity) {
  try {
      handle_activity($activity);
  }
  catch (MaharaException $e) {
      log_debug($e->getMessage());
  }
}
delete_records('activity_queue');
db_commit();

Much better. However, there was still a problem with the code: the whole queue processing is contained within a transaction. This means that should any of the SQL statements fail at any point, the exception would be caught but the SQL statements would all be rolled back at the end by the database.

Now the idea of a transaction is good: we want activity handling to either succeed entirely or be rolled back. But the fact that some activities cannot be handled should not interfere with other activities. So this has been fixed by moving the transaction to the inside of the loop:

$activities = get_records('activity_queue');
foreach ($activities as $activity) {
  db_begin();
  try {
      handle_activity($activity);
  }
  catch (MaharaException $e) {
      log_debug($e->getMessage());
  }
  db_commit();
}
delete_records('activity_queue');

So individual activites are allowed to fail and get rolled back, but they will not affect other activites. But there was still one remaining problem: what if we encounter an error we cannot catch? For example, what would happen if PHP were to segfault or run out of memory while the activity queue is being processed?

Well, in that case, it turns out that Mahara will never reach the delete_records() call and the activity queue will never be cleared. Which means that on the next cron run, all of the activities will be handled again, even the ones that were successfully handled already (i.e. emails will be sent over and over again).

The way we fixed this problem was by moving the delete_records() from the end of the function to the beginning of the loop:

$activities = get_records('activity_queue');
foreach ($activities as $activity) {
  delete_records('activity_queue', 'id', $activity->id);

  db_begin();
  try {
      handle_activity($activity);
  }
  catch (MaharaException $e) {
      log_debug($e->getMessage());
  }
  db_commit();
}

Each activity is removed from the queue before it is processed.

Unfortunately, there is a downside to this modification: should an activity handler fail for whatever reason, no further attempts will be made. This means that some notifications could be lost if an unexpected error occurs.

However, given that some of the activity handlers send emails out into the world and that it is not possible to "un-send" them, this is the only way we can guarantee that no duplicate emails will be sent. Of course, if you notice that certain notifications are lost because of a bug in Mahara, let us know and we'll fix it!

"Abusing" git storage

2010-03-31T13:36:00.001+13:00

A git repository is primarily intended to store multiple branches of a single program or component. The underlying system however is much more flexible. Here are two ways to add files which are related to the project but outside the normal history.

Storing a single file all by itself

This intermediate trick allows you to store a single file into a git repository without the file being in any of the branches.

First of all, let's create a new file:

echo "Hi" > message.txt

and store it in git:

git hash-object -w message.txt b14df6442ea5a1b382985a6549b85d435376c351

At this stage, the file is stored within the git repository but there is no other way to get to it other than using the hash:

git cat-file blob b14df6442ea5a1b382985a6549b85d435376c351 Hi

A good way to point to the stored contents is to use a tag:

git tag message.txt b14df6442ea5a1b382985a6549b85d435376c351

Now you can access your file this way:

git cat-file blob message.txt Hi

Creating an empty branch

As seen in this git screencast or in the Git Community Book, you can create a branch without a parent or any initial contents:

git symbolic-ref HEAD refs/heads/foo rm .git/index git clean -fdx

To finish it off, add what you were planning on storing there and commit it:

git add myfile git commit -m 'Initial commit'

Excluding files from git archive exports using gitattributes

2010-02-09T16:45:00.000+13:00

git archive provides an easy way of producing a tarball directly from a project's git branch.

For example, this is what we use to build the Mahara tarballs:

git archive --format=tar --prefix=mahara-${VERSION}/ ${RELEASETAG} | bzip2 -9 > ${CURRENTDIR}/mahara-${RELEASE}.tar.bz2

If you do this however, you end up with the entire contents of the git branch, including potentially undesirable files like .gitignore.

There is an easy, though not very well-documented, way of specifying files to exclude from such exports: gitattributes.

This is what the Mahara .gitattributes file looks like:

/test export-ignore .gitattributes export-ignore .gitignore export-ignore

With this file in the root directory of our repository, tarballs we generate using git archive no longer contain the selenium tests or the git config files.

If you start playing with this feature however, make sure you commit the .gitattributes file to your repository before running git archive. Otherwise the settings will not be picked up by git archive.

Ignoring files in git repositories

2009-12-27T21:45:00.001+13:00

According to the man page, there are three ways to exclude files from being tracked by git.

Shared list of files to ignore

The most well-known way of preventing files from being part of a git branch is to add such files in .gitignore. (This is analogous to CVS' .cvsignore files.)

Here's an example:

*.generated.html
/config.php

The above ignore list will prevent automatically generated HTML files from being committed by mistake to the repository. Because this is useful to all developers on the project, .gitignore is a good place for this.

The next line prevents the local configuration file from being tracked by git, something else that all developers will want to have.

One thing to note here is the use of a leading slash character with config.php. This is to specifically match the config file in the same directory as the .gitignore file (in this case, the root directory of the repository) but no other. Without this slash, the following files would also be ignored by git:

/app/config.php
/plugins/address/config.php
/module/config.php

Local list (specific to one project)

For those custom files that you don't want version controlled but that others probably don't have or don't want to automatically ignore, git provides a second facility: .git/info/exclude

It works the same way as .gitignore but be aware that this list is only stored locally and only applies to the repository in which it lives.

(I can't think of a good example for when you'd want to use this one because I don't really use it. Feel free to leave a comment if you do use it though, I'm curious to know what others do with it.)

Local list (common to all projects)

Should you wish to automatically ignore file patterns in all of your projects, you will need to use the third gitignore method: core.excludesfile

Put this line in your ~/.gitconfig:

[core]
excludesfile = /home/username/.gitexcludes

(you need to put the absolute path to your home directory, ~/ will not work here unless you use git 1.6.6 or later)

and then put the patterns to ignore in ~/.gitexcludes. For example, this will ignore the automatic backups made by emacs when you save a file:

*~

This is the ideal place to put anything that is generated by your development tools and that doesn't need to appear in your project repositories.

Reducing website bandwidth usage

2009-10-09T18:30:00.004+13:00

There are lots of excellent tools to help web developers optimise their websites.

Here are two simple things you have no excuse for overlooking on your next project.

HTML, XML, Javascript and CSS files

One of the easiest ways to speed up a website (often to a surprising degree) is to turn on compression of plaintext content through facilities like mod_deflate or the Gzip Module.

Here's the Apache configuration file I normally use:

AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript text/xml application/x-javascript application/javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch "MSIE 6" no-gzip dont-vary
BrowserMatch ^Mozilla/4\.0[678] no-gzip

Images

As far as images go, the following tools will reduce file sizes through lossless compression (i.e. with no visual changes at all):

gifsicle -O2 -b image.gif
jpegoptim -p --strip-all image.jpg
optipng -o7 -q image.png

(An alternative to optipng is pngcrush.)

Note that the --strip-all argument to jpegoptim will remove any EXIF/comments tags that may be present.

3 ways to improve your source control history

2009-07-29T23:52:00.000+12:00

A few weeks ago, Linus Torvalds wrote a blog post talking about how happy he was with the way that the last Linux merge window was going. Especially given how flexible distributed source control systems are and how long it takes before developers understand how to use them properly.

Most Free Software projects collaborate with external contributors through the use of patches and with project members through shared source control systems. Keeping a clean source control history in this case can:

ease maintenance of the codebase (especially when merging between multiple active branches) and
facilitate collaboration with other developers who need to quickly grasp what your changes are all about.

Here are three things I've put in practice in a few recent projects.

Reduce the noise on the main branch

If you use your source control system in a traditional centralised way, make sure that the main branch has as little noise as possible:

use feature branches when developing new functionality (i.e. use one branch per feature)
rebase into a small number of commits instead of merging directly into the main branch (or use something like git cherry-pick -n on your commits)

It will then be much easier to port these changes across to a different version and to revisit the changes several months later.

Make each commit a logical unit

Ideally, the main branch should be fully usable at any point in its history. For example, if a refactoring commit breaks an existing feature, it should also include any necessary fixes. Progress and fixup commits belong to private branches and should be cleaned up before the code is published on a public repository.

A commit shouldn't be too small, but at the same time, it shouldn't be too large. Before pushing a large commit, ask yourself whether or not it could be broken down into smaller or more meaningful units. Introducing a new feature through a number of smaller self-contained units often makes a branch's history much more readable.

Write meaningful commit messages

A commit message will typically be seen along with its patch. Therefore the ideal commit message will include everything which is not immediatly obvious from the code changes:

A short description of the new feature it introduces (or how to reproduce the bug it fixes)
Links to related commits (e.g. "Fix bug introduced in COMMIT_ID")
Bug numbers related to this commit
The main area of the code (or subsystem) that this commit touches (as part of the short description, so that it's easily grepable)

Here's an example of a good commit message from version 2.6.29 of the Linux kernel:

commit 4869fdea77052c480c54b9e66a927ba036eab29e
Author: Mikulas Patocka 
Date:   Mon Jun 22 10:08:02 2009 +0100

dm mpath: validate table argument count

commit 0e0497c0c017664994819f4602dc07fd95896c52 upstream.

The parser reads the argument count as a number but doesn't check that
sufficient arguments are supplied. This command triggers the bug:

dmsetup create mpath --table "0 `blockdev --getsize /dev/mapper/cr0`
 multipath 0 0 2 1 round-robin 1000 0 1 1 /dev/mapper/cr0
 round-robin 0 1 1 /dev/mapper/cr1 1000"
kernel BUG at drivers/md/dm-mpath.c:530!

Good commits and useful source control histories are great ways of documenting your code. If there are other conventions or practices you've found useful in your use of source control systems, please leave a comment!

Writing the perfect patch

2009-06-30T23:00:00.001+12:00

Other people have written and talked (in Lecture 3) about writing the perfect patch for a Free Software project. The goal there is to increase the likelihood that a patch will be accepted by the project developers.

Integrating and testing patches takes time and so reducing that burden is essential when interacting with busy maintainers. Especially if they're volunteers.

Here's what I try to keep in mind when preparing a patch.

Use the right options to `diff`

These two options should always be part of your call to the diff command:

-u: use the most common patch format, unidiff.
-p: include the name of the function that's being changed.

and this one can be useful if the output seems unnecessarily large:

-d: try hard to find a smaller set of changes.

Minimize the number of changes

You need to draw attention to the changes that you're proposing and remove all other potential distractions:

Follow the coding style of the original file. Your changes must fully blend in or they are likely to be rejected.
Do not re-indent existing code. This will make it look like you modified every line.
Pay attention to whitespace changes. In particular: end-of-line characters, trailing spaces and tab-versus-space differences. Use the dos2unix or unix2dos commands if you need to.
Gratuitous refactoring of existing code. Unless the refactoring makes your change smaller or easier to understand, keep it for another patch.

Of course all of the above would be acceptable patches on their own, just not combined with other types of changes.

Only one logical change at a time

Patches often need to be broken up into a series of logical changes to avoid these two extremes:

the gigantic patch which adds a number of features and fixes a couple of bugs but scares everybody
a series of interdependent patches which all relate to the same change and must all be applied together

It's a bit of a balancing act, but a good rule of thumb is:

to have one patch per feature or bug and
to try to find the smallest (yet meaningful) change which can be applied on its own.

It's not just about the patch

Your patch can be really good, but the email (or the bug tracker update) announcing it should also contain:

a good description of the problem it solves and how it solves it
the output of diffstat to give an idea of the size of the change

Troubleshooting Postgres Performance Problems

2009-05-30T17:00:00.004+12:00

There are great resources to help tune PostgresQL for performance, but say you've got a Postgres database which suddenly becomes very slow. How can you figure out what's going on?

General system state

A slow database server will typically be running out of:

cpu,
memory or
disk.

So the first thing you should do is to get an overview of the system state using common UNIX tools such as:

top (press c to see the full command line): look at CPU and memory usage
iostat -x -m 5: look at i/o wait and service time

If you notice that memory is exhausted, you could play with the following postgresql.conf settings:

shared_buffers
work_mem
effective_cache_size

If you see a lot of I/O, then try adjusting these settings:

checkpoint_segments
autovacuum_naptime
wal_buffers

Finding slow queries

On the database side, start by determining whether:

there is one large query holding up all of the resources or
the number of queries is the main problem.

To do this, make sure that the following setting is turned on in your postgresql.conf:

stats_command_string = on

Then fire up psql dbname and look at the currently executing queries:

SELECT usename, current_query, query_start
FROM pg_stat_activity;

(You can also add the procid column to the query if you want to map a query to the process ID you see in top.)

If you can't get anything useful out of pg_stat_activity, you may want to enable slow query logging by adding this to your postgresql.conf:

log_min_duration_statement = 100

(All queries which take more than 100 ms to execute will be printed to the main Postgres log file.)

Examining a particular query

Once you have identified a slow query, you can time it by running this command before executing the query manually:

\timing

To get an idea of where Postgres spends its time when executing that query, look at the query plan:

EXPLAIN your_query;

The numbers you see there are estimates. To run the query and get actual numbers, use this instead:

EXPLAIN ANALYZE your_query;

If you still can't figure out why a certain query takes so long, have a look at the ratio between live and dead rows in the relevant tables:

ANALYZE VERBOSE tablename;

Having too many dead rows is often a sign of insufficient vacuuming. You might want to turn autovacuum on if it isn't already, or to make it a bit more aggressive by tweaking these settings:

autovacuum_vacuum_scale_factor
autovacuum_analyze_scale_factor

Troubleshooting database performance is a bit of a black art, many thanks to Mark Kirkwood for sharing his Postgres wisdom with me.

If there's anything else you have successfully used to find the cause of your performance woes, please feel free to leave a comment.

Handling security bugs in your Free Software project

2009-03-10T21:45:00.001+13:00

If you are managing a Free Software project, you may eventually be confronted with a security vulnerability. Normally bugs in the Open Source world are discussed in a transparent way on public forums. In the case of security bugs however, there are benefits to temporarily withholding these details from the public.

Some people describe this approach as responsible disclosure. It boils down to this:

if the vulnerability is not publicly known, warn the vendors first and give them some time to fix it before making the details public.

If the vulnerability is already public knowledge, then you can focus on fixing it as soon as possible and maybe let reporters know about a better way to report their findings.

Sample Security Policy

Here's the procedure we now follow in the Mahara project for security bugs we found ourselves or which have been privately disclosed to us through security@mahara.org:

Figure out the extent of the problem: which versions of Mahara are affected by this problem?
Fix the problem on all supported branches of the project in a private source control repository.
Share the vulnerability information with vendor-sec and request a CVE identifier.
Prepare release tarballs and packages.
Draft a security advisory for the Security forum.
Wait until the embargo date to push the release out.

Emails to vendor-sec should include:

a description of the problem
any proof of concept code demonstrating the vulnerability
the list of affected versions
any patches you have prepared
a proposed embargo date

Benefits of properly handling security bugs

First and foremost, it protects end-users by giving them a chance to download and install fixed versions of your software before widespread exploitation of the security flaws.

Secondly, it may increase your project credibility. While you are admitting that your software has flaws, you are also demonstrating that your project is committed to dealing with the most serious ones promptly and in a responsible manner.

Finally, sharing security flaws and having your fixes reviewed by a select group of experts may reveal extra vulnerabilities you missed while preparing your patches. This gives you an opportunity to improve your fixes before they are released and to avoid having to issue yet another security advisory a few days later.

Big thanks to Steffen Joeris for his help in shaping the Mahara policy!