Debugging OpenWRT routers by shipping logs to a remote syslog server

Trying to debug problems with consumer-grade routers is notoriously difficult due to a lack of decent debugging information. It's quite hard to know what's going on without at least a few good error messages.

Here is how I made my OpenWRT-based Gargoyle router send its log messages to a network server running rsyslog.

Server Configuration

Given that the router (192.168.1.1) will be sending its log messages on UDP port 514, I started by opening that port in my firewall:

iptables -A INPUT -s 192.168.1.1 -p udp --dport 514 -j ACCEPT

Then I enabled the UDP module for rsyslog and redirected messages to a separate log file (so that it doesn't fill up /var/log/syslog) by putting the following (a modified version of these instructions) in /etc/rsyslog.d/10-gargoyle-router.conf:

$ModLoad imudp
$UDPServerRun 514
:fromhost-ip, isequal, "192.168.1.1" /var/log/gargoyle-router.log
& ~

The name of the file is important because this configuration snipet needs to be loaded before the directive which writes to /var/log/syslog for the discard statement (the "& ~" line) to work correctly.

Router Configuration

Finally, I followed the instructions on the Gargoyle wiki to get the router to forward its log messages to my server (192.168.1.2).

After logging into the router via ssh, I ran the following commands:

uci set system.@system[0].log_ip=192.168.1.2
uci set system.@system[0].conloglevel=7
uci commit

before rebooting the router.


Now whenever I have to troubleshoot network problems, I can keep a terminal open on my server and get some visibility on what the router is doing:

tail -f /var/log/gargoyle-router.log

Installing Etherpad on Debian/Ubuntu

Etherpad is an excellent Open Source web application for collaborative text editing. Like Google Docs, it allows you to share documents with others through a secret URL or to set up private documents for which people need a login.

It's a little tricky to install so here's how I did it.

Build a Debian package

Because the official repository is not kept up to date, you must build the package yourself:

1. Grab the master branch from the official git repository:
   

   git clone git://github.com/ether/pad.git etherpad

2. Build the package:
   

   dpkg-buildpackage -us -uc

Now, install some of its dependencies:

apt-get install --no-install-recommends dbconfig-common python-uno mysql-server

before installing the .deb you built:

dpkg -i etherpad_1.1.deb
apt-get install --no-install-recommends -f

Application configuration

You will likely need to change a few minor things in the default configuration at /etc/etherpad/etherpad.local.properties:

useHttpsUrls = true
customBrandingName = ExamplePad
customEmailAddress = etherpad@example.com
topdomains = etherpad.example.com,your.external.ip.address,127.0.0.1,localhost,localhost.localdomain

Nginx configuration

If you use Nginx as your web server of choice, create a vhost file in /etc/nginx/sites-available/etherpad:

server {
  listen 443;
  server_name etherpad.example.com *.etherpad.example.com;
  add_header Strict-Transport-Security max-age=15768000;

  ssl on;
  ssl_certificate  /etc/ssl/certs/etherpad.example.com.crt;
  ssl_certificate_key  /etc/ssl/certs/etherpad.example.com.pem;

  ssl_session_timeout  5m;
  ssl_session_cache shared:SSL:1m;

  ssl_protocols  TLSv1;
  ssl_ciphers  RC4-SHA:HIGH:!kEDH;
  ssl_prefer_server_ciphers   on;

  access_log /var/log/nginx/etherpad.access.log;
  error_log /var/log/nginx/etherpad.error.log;

  location / {
    proxy_pass http://localhost:9000/;
    proxy_set_header Host $host;
  }
}

and then enable it and restart Nginx:

/etc/init.d/nginx restart

Apache configuration

If you prefer to use Apache instead, make sure that the required modules are enabled:

a2enmod proxy
a2enmod proxy_http

and then create a vhost file in /etc/apache2/sites-available/etherpad:

<VirtualHost *:443>
   ServerName etherpad.example.com
   ServerAlias *.etherpad.example.com

   SSLEngine on
   SSLCertificateFile /etc/apache2/ssl/etherpad.example.com.crt
   SSLCertificateKeyFile /etc/apache2/ssl/etherpad.example.com.pem
   SSLCertificateChainFile /etc/apache2/ssl/etherpad.example.com-chain.pem

   SSLProtocol TLSv1
   SSLHonorCipherOrder On
   SSLCipherSuite RC4-SHA:HIGH:!kEDH
   Header add Strict-Transport-Security: "max-age=15768000"

   <Proxy>
       Order deny,allow
       Allow from all
   </Proxy>

   Alias /sitemap.xml /ep/tag/\?format=sitemap
   Alias /static /usr/share/etherpad/etherpad/src/static

   ProxyPreserveHost On
   SetEnv proxy-sendchunked 1
   ProxyRequests Off
   ProxyPass / http://localhost:9000/
   ProxyPassReverse / http://localhost:9000/
</VirtualHost>

before enabling that new vhost and restarting Apache:

a2ensite etherpad
apache2ctl configtest
apache2ctl graceful

DNS setup

The final step is to create these two DNS entries to point to your web server:

• *.etherpad.example.com
• etherpad.example.com

Also, as a precaution against an OpenOffice/LibreOffice-related bug, I suggest that you add the following entry to your web server's /etc/hosts file to avoid flooding your DNS resolver with bogus queries:

127.0.0.1 localhost.(none) localhost.(none).fulldomain.example.com

where fulldomain.example.com is the search base defined in /etc/resolv.conf.

Other useful instructions

Here are the most useful pages I used while setting this up:

• Official installation instructions
• Ubuntu Wiki instructions
• Instructions for Debian Lenny
• Nginx Configuration

Optimising PNG files

I have written about using lossless optimisations techniques to reduce the size of images before, but I recently learned of a few other tools to further reduce the size of PNG images.

Basic optimisation

While you could use Smush.it to manually optimise your images, if you want a single Open Source tool you can use in your scripts, optipng is the most effective one:

optipng -o9 image.png

Removing unnecessary chunks

While not as effective as optipng in its basic optimisation mode, pngcrush can be used remove unnecessary chunks from PNG files:

pngcrush -q -rem gAMA -rem alla -rem text image.png image.crushed.png

Depending on the software used to produce the original PNG file, this can yield significant savings so I usually start with this.

Reducing the colour palette

When optimising images uploaded by users, it's not possible to know whether or not the palette size can be reduced without too much quality degradation. On the other hand, if you are optimising your own images, it might be worth trying this lossy optimisation technique.

For example, this image went from 7.2 kB to 5.2 kB after running it through pngnq:

pngnq -f -n 32 -s 3 image.png

Re-compressing final image

Most PNG writers use zlib to compress the final output but it turns out that there are better algorithms to do this.

Using AdvanceCOMP I was able to bring the same image as above from 5.1kB to 4.6kB:

advpng -z -4 image.png

When the source image is an SVG

Another thing I noticed while optimising PNG files is that rendering a PNG of the right size straight from an SVG file produces a smaller result than exporting a large PNG from that same SVG and then resizing the PNG to smaller sizes.

Here's how you can use Inkscape to generate an 80x80 PNG:

inkscape --without-gui --export-width=80 --export-height=80 --export-png=80.png image.svg

Ideal OpenSSL configuration for Apache and nginx

After recently reading a number of SSL/TLS-related articles, I decided to experiment and look for the ideal OpenSSL configuration for Apache (using mod_ssl since I haven't tried mod_gnutls yet) and nginx.

By "ideal" I mean that this configuration needs to be compatible with most user agents likely to interact with my website as well as being fast and secure.

Here is what I came up with for Apache:

SSLProtocol TLSv1
SSLHonorCipherOrder On
SSLCipherSuite RC4-SHA:HIGH:!kEDH

and for nginx:

ssl_protocols  TLSv1;
ssl_ciphers  RC4-SHA:HIGH:!kEDH;
ssl_prefer_server_ciphers   on;

Cipher and protocol selection

In terms of choosing a cipher to use, this configuration does three things:

• disables all weak ciphers and protocols
• disables very slow ciphers that use ephemeral Diffie-Hellman exchanges
• gives priority to the RC4 cipher to minimize CPU usage and defend against the BEAST attack

Testing tools

The main tool I used while testing various configurations was the SSL labs online tool. The CipherFox extension for Firefox was also quite useful to quickly identify the selected cipher.

Of course, you'll want to make sure that your configuration works in common browsers, but you should also test with tools like wget, curl and httping. Many of the online monitoring services are based on these.

Other considerations

To increase the performance and security of your connections, you should ensure that the following features are enabled:

• SSL session caching with a session store shared between all of your web servers
• HSTS headers to let browsers know that they should always visit your site over HTTPS

Note: If you have different SSL-enabled name-based vhosts on the same IP address (using SNI), make sure that their SSL cipher and protocol settings are identical.

Using BrowserID and Content Security Policy together

While looking into why BrowserID logins on Libravatar didn't work on Firefox, I remembered that I had recently added Content Security Policy headers. Here's what I had to do to make BrowserID work on a CSP-enabled site.

Create a hidden form and a login link

This is what the login button looked like before CSP:

<form id="browserid-form" action="/account/login_browserid" method="post">
<input id="browserid-assertion" type="hidden" name="assertion" value="">
<input style="display: none" type="submit">
</form>

<a href="javascript:try_browserid()">Login with BrowserID</a>

<script type="text/javascript">
function try_browserid() {
navigator.id.getVerifiedEmail(function(assertion) {
    if (assertion) {
        document.getElementById('browserid-assertion').setAttribute('value', assertion);
        document.getElementById('browserid-form').submit();
    }
});
}
</script>

The hidden form is there because the assertion needs to be sent to the application via POST to avoid leaking it out, but otherwise the code is pretty straightforward.

Now of course, with CSP turned ON, there are two problems:

• links cannot use javascript: URIs
• inline Javascript is forbidden

So we can start by converting the login link to:

<a id="browserid-link" href="#">Login with BrowserID</a>

<script src="browserid_stuff.js" type="text/javascript">

then moving the try_browserid() function to a separate file to be served from the same domain and finally hooking it up to the try_browserid() function using Javascript (in that same browserid_stuff.js file):

var link = document.getElementById('browserid-link');
link.onclick = try_browserid;
link.addEventListener('click', try_browserid, false);

Exposing the right X-Content-Security-Policy header

In order to load the Javascript code from browserid.org, we need the following as part of the policy:

script-src https://browserid.org

but that's not enough since the BrowserID login form seems to use some sort of <iframe> trick and so we need to add this extra permission as well:

frame-src https://browserid.org

Here is the final policy I ended up setting (using Apache mod_headers) for the Libravatar login page:

<Location /account/login>
Header set X-Content-Security-Policy: "default-src 'self'; frame-src 'self' https://browserid.org ; script-src 'self' https://browserid.org"
</Location>