Reducing website bandwidth usage

There are lots of excellent tools to help web developers optimise their websites.

Here are two simple things you have no excuse for overlooking on your next project.

HTML, XML, Javascript and CSS files

One of the easiest ways to speed up a website (often to a surprising degree) is to turn on compression of plaintext content through facilities like mod_deflate or the Gzip Module.

Here's the Apache configuration file I normally use:

AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript text/xml application/x-javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch "MSIE 6" no-gzip dont-vary
BrowserMatch ^Mozilla/4\.0[678] no-gzip

Images

As far as images go, the following tools will reduce file sizes through lossless compression (i.e. with no visual changes at all):

• gifsicle -O2 -b image.gif
• jpegoptim -p --strip-all image.jpg
• optipng -o7 -q image.png

(An alternative to optipng is pngcrush.)

Note that the --strip-all argument to jpegoptim will remove any EXIF/comments tags that may be present.

Keeping track of what others are saying about your project

There are a few ways to easily keep track of what others are saying about your Free Software project. In a very popular project, you may already have enough feedback from its busy forum or mailing list, but in new and small projects, you often have to actively seek feedback/comments from end-users.

Here are a few ways to find out what people are saying online about your project.

RSS Feeds

First of all, here are the RSS feeds I subscribe to for one of my projects (safe-rm):

• Technorati (blog posts)
• Twitter tweets (tweets)
• Identica (notices)
• TweetMeme (tweets)
• BackTweets (tweets)
• Freshmeat (comments)
• Ohloh (journal entries)

Google Alerts

I have also signed up for the following Google Alert:

Search terms:	link:safe-rm.org.nz
Type:	Comprehensive
Deliver to:	Email
How often:	as-it-happens

which emails me anytime someone links to my project's homepage in one of the resources indexed by Google.

Backtype Alerts

Finally, these Backtype Alerts notify me anytime safe-rm features in a blog comment:

• safe-rm
• safe-rm.org.nz

Anything else I should subscribe to or follow?

OpenStreetMap/OpenLayers and Privoxy

I have been meaning to switch to OpenStreetMap for a while now, but the fact that it didn't work with Privoxy was holding me back. I don't enjoy surfing the web without protection and Privoxy is one of the most convenient ways of enhancing one's browsing experience on Debian and Ubuntu.

So I decided to spend some time trying to figure out what was wrong. If you want to skip the details and just get the fix, scroll down to the bottom of this post for the solution.

Finding all affected URLs

In order to get a list of all of the URLs that Privoxy filters when one visits the front page of OpenStreetMap, I did the following:

• Added "debug 1" to /etc/privoxy/config


• Restarted Privoxy: sudo /etc/init.d/privoxy restart


• Cleared Firefox cache: Tools | Clear Private Data...


• Loaded www.openstreetmap.org in Firefox
  


• Copied from the Privoxy log (/var/log/privoxy/logfile) all requested URLs (except the image ones)

Comparing the files

Armed with the list of all affected URLs, I downloaded them through Privoxy using wget into a privoxy/ directory:

$ http_proxy=localhost:8118 wget URL

Then I downloaded the same URLs without Privoxy into a noproxy/ directory:

$ http_proxy= wget URL

To find identical files, I ran md5sums in each directory:

md5sum *

After deleting all identical files, I was left with:

index.html
OpenLayers.js?1251388304

Which I diff'ed together:

colordiff -u noproxy/FILE privoxy/FILE

The first file, index.html did not have any relevant changes, but I noticed in the second one that a few of these

this.moveTo(this.position)

were replaced with

''.concat(this.position)

Searching for "concat" in /etc/privoxy/, I found that the "jumping-windows" filter was the culprit. After disabling it, all of the problems went away.

I have filed a bug upstream.

The Solution

Here's what you need to put in your /etc/privoxy/user.action:

{-filter{jumping-windows}}
.openstreetmap.org

Twitter tracking clicks to external sites

Lately, while using the Twitter web interface with Javascript turned on, I've been redirected to a Privoxy error page whenever I click on an external link:

Request for blocked URL

Your request for http://twitter.com/link_click_count?url=http%3A%2F%2Fexample.com&linkType=web&tweetId=1234567890&userId=12345678&authenticity_token=f5920ebf56bfa111c54bb9f86814cafa76de00fa was blocked.

Block reason: Path matches generic block pattern.

Looking a little closer at the blocked URL, I noticed that for every external link you click, Twitter logs the destination URL, the originating tweet and your user ID.

So what can you do to opt out of this tracking?

Disable Javascript

This is by far the easiest solution. You can simply install an extension like Noscript and use Twitter with Javascript turned off. That's what I do most of the time since the website is mostly usable without it.

A very fortunate side-effect of using Twitter without Javascript is that you can avoid a number of the worms which propagate through that service from time to time.

Use Privoxy to disable the link counter

For those times when I have to enable Javascript on the site, I use Privoxy, a great privacy-enhancing web proxy, to alter the Javascript code executed when clicking on external links. This effectively disables the extra tracking and takes me directly to the intended URL.

First of all, make sure you have this line in your /etc/privoxy/config:

filterfile user.filter

then, add this filter to /etc/privoxy/user.filter:

FILTER: twitter-link-count Disable the Twitter link tracker
s@{var A=\$\("\#content a\.tweet-url"\);A\.live\("mousedown",twttr\.countClick\)}@{}@ig

and finally turn this filter on for the affected URLs (/etc/privoxy/user.action):

{ +force-text-mode +filter{twitter-link-count} }
.twimg.com/.*/javascripts/front.js

Switch to Identica

If you want to avoid similar privacy problems in the future, you can also switch to an Open platform which respects your basic freedoms.

The main contestant in this space is Identica. It's a great alternative to Twitter and has a lot of interesting features (like groups and XMPP integration) which are not yet available on Twitter.

The StatusNet developers have kindly released their source code under the AGPL and their goal is to build the most open micro-blogging system out there.

So go over there now and claim your username!

grub on a bootable USB rescue stick

If you've got a cheap USB key lying around which is too small to be useful, follow these instructions to create a rescue "stick" in case your boot loader stops working.

(This is based on these instructions but simplified and updated for grub 2.)

Before you start, make sure you've got both the parted and the grub-pc packages installed.

Make the USB stick bootable

Plug in your USB stick and replace the device name and capacity with the appropriate values. (You can find the total capacity of the stick by typing print inside parted.)

$ sudo parted /dev/sda
(parted) rm 1
(parted) mkpart primary 1MB 4009MB
(parted) mkfs 1 fat32
(parted) toggle 1 boot
(parted) quit

Install grub

Again, substitute sda with the correct device name.

mount /dev/sda /mnt

mkdir -p /mnt/boot/grub
cp /usr/lib/grub/i386-pc/* /mnt/boot/grub/
echo '(hd0) /dev/sda' > /mnt/boot/grub/device.map
sudo grub-install --root-directory=/mnt /dev/sda

umount /mnt

That's it, now all you have to do is reboot with the stick plugged in and test booting your box manually with it.