Debugging Gearman configuration

Gearman is a queuing system that has been in Debian for a long time and is quite reliable.

I ran into problems however when upgrading a server from Debian squeeze to wheezy however. Here's how I debugged my Gearman setup.

Log verbosity

First of all, I started by increasing the verbosity level of the daemon by adding --verbose=INFO to /etc/default/gearman-job-server (the possible values of the verbose option are in the libgearman documentation) and restarting the daemon:

/etc/init.d/gearman-job-server restart

I opened a second terminal to keep an eye on the logs:

tail -f /var/log/gearman-job-server/gearman.log

Listing available workers

Next, I registered a very simple worker:

gearman -w -f mytest cat

and made sure it was connected properly by telneting into the Gearman process:

telnet localhost 4730

and listing all currently connected workers using the workers command (one of the commands available in the Gearman TEXT protocol).

There should be an entry similar to this one:

30 127.0.0.1 - : mytest

Because there are no exit or quit commands in the TEXT protocol, you need to terminate the telnet connection like this:

  1. Press Return
  2. Press Ctrl + ]
  3. Press Return
  4. Type quit at the telnet prompt and press Return.

Finally, I sent some input to the simple worker I setup earlier:

echo "hi there" | gearman -f mytest

and got my input repeated on the terminal:

hi there

Gearman bug

I traced my problems down to this error message when I sent input to the worker:

gearman: gearman_client_run_tasks : connect_poll(Connection refused)
getsockopt() failed -> libgearman/connection.cc:104

It turns out that it is a known bug that was fixed upstream but still affects Debian wheezy and some versions of Ubuntu. The bug report is pretty unhelpful since the work-around is hidden away in the comments of this "invalid" answer: be explicit about the hostname and port number in both gearman calls.

So I was able to make it work like this:

gearman -w -h 127.0.0.1 -p 4730 -f mytest cat    
echo "hi there" | gearman -h 127.0.0.1 -p 4730 -f mytest

where the hostname matches exactly what's in /etc/default/gearman-job-server.

Server Migration Plan

I recently had to migrate the main Libravatar server to a new virtual machine. In order to minimize risk and downtime, I decided to write a migration plan ahead of time.

I am sharing this plan here in case it gives any ideas to others who have to go through a similar process.

Prepare DNS

  • Change the TTL on the DNS entries for libravatar.org and libravatar.com (i.e. bare A and AAAA records) to 3600 seconds.
  • Remove the mirrors I don't control from the DNS load balancer (cdn and seccdn).
  • Remove the main server from cdn and seccdn in DNS.

Preparing the new server

  • Setup the new server.
  • Copy the database from the old site and restore it.
  • Copy /var/lib/libravatar from the old site.

  • Hack my local /etc/hosts file to point to the new server's IP address:

    xxx.xxx.xxx.xxx www.libravatar.org stats.libravatar.org cdn.libravatar.org

  • Test all functionality on the new site.

Preparing the old server

  • Prepare a static "under migration" Apache config in /etc/apache2/sites-enabled.static/default.conf:

    <VirtualHost *:80>
    RewriteEngine On
    RewriteRule ^ https://www.libravatar.org [redirect=301,last]
</VirtualHost>

<VirtualHost *:443>
    SSLEngine on

    SSLCertificateFile /etc/libravatar/www.crt
    SSLCertificateKeyFile /etc/libravatar/www.pem
    SSLCertificateChainFile /etc/libravatar/www-chain.pem

    RewriteEngine On
    RewriteRule ^ /var/www/html/migration.html [last]

    <Directory /var/www/html>
        Allow from all
        Options -Indexes
    </Directory>
</VirtualHost>

  • Put this static file in /var/www/html/migration.html:

    <html>
<body>
<p>We are migrating to a new server. See you soon!</p>
<p>- <a href="https://identi.ca/libravatar">@libravatar</a></p>
</body>
</html>

  • Enable the rewrite module:

    a2enmod rewrite

  • Prepare an Apache config proxying to the new server in /etc/apache2/sites-enabled.proxy/default.conf:

    <VirtualHost *:80>
    RewriteEngine On
    RewriteRule ^ https://www.libravatar.org [redirect=301,last]
</VirtualHost>

<VirtualHost *:443>
    SSLEngine on

    SSLCertificateFile /etc/libravatar/www.crt
    SSLCertificateKeyFile /etc/libravatar/www.pem
    SSLCertificateChainFile /etc/libravatar/www-chain.pem

    SSLProxyEngine on
    ProxyPass / https://www.libravatar.org/
    ProxyPassReverse / https://www.libravatar.org/
</VirtualHost>

  • Enable the proxy-related modules for Apache:

    a2enmod proxy
a2enmod proxy_connect
a2enmod proxy_http

Migrating servers

  • Tweet and dent about the upcoming migration.

  • Enable the static file config on the old server (disabling the Django app):

    cd /etc/apache2/
mv sites-enabled sites-enabled.django
mv sites-enabled.static sites-enabled
apache2ctl configtest
systemctl restart apache2.service

  • Disable pgbouncer to ensure that Django cannot access postgres anymore:

    systemctl stop pgbouncer.service

  • Copy the database from the old server and restore it on the new server making sure it's in the UTF8 encoding:

    dropdb libravatar
createdb -O djangouser -E utf8 libravatar
pg_restore -d libravatar libravatar20180812.pg

  • Copy /var/lib/libravatar from the old server to the new one.

    • On the new server:

      chmod a+w /var/lib/libravatar/avatar
rm -rf /var/lib/libravatar/avatar/*
chmod a+w /var/lib/libravatar/user
rm -rf /var/lib/libravatar/user/*

    • From laptop:

      rsync -a -H -v old.libravatar.org:/var/lib/libravatar/avatar .
rsync -a -H -v old.libravatar.org:/var/lib/libravatar/user .
rsync -a -H -v avatar/* new.libravatar.org:/var/lib/libravatar/avatar/
rsync -a -H -v user/* new.libravatar.org:/var/lib/libravatar/user/

    • On the new server:

      chmod go-w /var/lib/libravatar/avatar
chmod go-w /var/lib/libravatar/user
chown -R root:root /var/lib/libravatar/avatar/* /var/lib/libravatar/user/*

Disable mirror sync

  • Log into each mirror and comment out the update cron jobs in /etc/cron.d/libravatar-slave.
  • Make sure mirrors are no longer able to connect to the old server by moving /var/lib/libravatar/master/.ssh/authorized_keys to the new server and removing it from the old server.

Testing the main site

  • Hack my local /etc/hosts file to point to the new server's IPv4 address:

    xxx.xxx.xxx.xxx www.libravatar.org stats.libravatar.org cdn.libravatar.org seccdn.libravatar.org

  • Test all functionality on the new site.

  • Do a basic version of the previous test using IPv6.

  • If testing is successful, update DNS A and AAAA records (libravatar.org and libravatar.com) to point to the new server with a short TTL (in case we need to revert).

  • Enable the proxy config on the old server.

    cd /etc/apache2/
mv sites-enabled sites-enabled.static
mv sites-enabled.proxy/ sites-enabled
apache2ctl configtest
systemctl restart apache2.service

  • Hack my local /etc/hosts file to point to the old server's IP address.

  • Test basic functionality going through the proxy.
  • Remove local /etc/hosts hacks.

Re-enable mirror sync

  • Build a new libravatar-slave package with an updated known_hosts file for the new server.
  • Log into each server I control and update that package.

  • Test the connection to the master (hacking /etc/hosts on the mirror if needed):

    sudo -u libravatar-slave ssh libravatar-master@0.cdn.libravatar.org

  • Uncomment the sync cron jobs in /etc/cron.d/libravatar-slave.

  • An hour later, make sure that new images are copied over and that the TLS certs are still working.
  • Remove /etc/hosts hacks from all mirrors.

Post migration steps

  • Tweet and dent about the fact that the migration was successful.

  • Send a test email to the support address included in the tweet/dent.

  • Take a backup of config files and data on the old server in case I forgot to copy something to the new one.

  • Get in touch with mirror owners to tell them to update libravatar-slave package and test ssh configuration.

  • Add third-party controlled mirrors back to the DNS load-balancer once they are up to date.

  • A few days later, change the TTL for the main site back to 43200 seconds.

  • A week later, kill the proxy on the old server by shutting it down.
How many Australasian banks use HSTS?

HTTP Strict Transport Security is a simple mechanism that secure sites can use to protect their users against an sslstrip-style HTTPS-to-HTTP downgrade attack.

Typical attack

The typical HTTPS-to-HTTP downgrade attack looks like this:

  1. victim connects to a compromised wifi access point
  2. victim connects to bank.com using attacker's DNS resolver
  3. attacker directs victim to a local server proxying the bank.com homepage
  4. victim clicks on "online banking" link as usual not noticing that it's an HTTP link instead of the usual HTTPS link
  5. attacker mounts a man-in-the-middle attack over that HTTP online banking login page
  6. victim leaks credentials to attacker

You can watch a short video demo of this attack, but if you don't want to set any of this up on your server, it turns out you can buy a little USB device that does it all for you.

What HSTS does

The fix is simple: let the browser know that it should never connect to the online banking site over plain HTTP. It should automatically upgrade to an encrypted HTTPS connection.

How should a site let the browser know? By including an HTTP header in its responses:

Strict-Transport-Security: max-age=10886400

It works in Chrome, Firefox and Opera. Other browsers don't benefit from this protection, but it also doesn't interfere with anything on those other browsers. So anybody with an HTTPS-only site should make use of this.

How many banks use it?

Given how easy it is to implement (and the fact that it's been in browsers since Chrome 4 and Firefox 4), how many of the Australasian banks actually make use of it? After all, almost all of the documentation explaining the motivation behind HSTS uses online banking as an example.

Here are all of the New Zealand banks I tested:

Bank Online Banking URL Header?
ASB https://fnc.asbbank.co.nz/1/User/LogOn YES!
ANZ https://secure.anz.co.nz/IBCS/pgLogin no
BankDirect https://vault.bankdirect.co.nz/default.asp no
BNZ https://www.bnz.co.nz/ib/app/login no
HSBC https://www.hsbc.co.nz/1/2/HUB_IDV2/IDV_EPP... no
Kiwibank https://www.ib.kiwibank.co.nz/ no
Rabobank https://secure1.rabodirect.co.nz/exp/authenticationDGPEN.jsp no
SBS https://sbsbanking.sbs.net.nz/secure/ no
TSB https://homebank.tsbbank.co.nz/online/ no
Westpac https://sec.westpac.co.nz/IOLB/Login.jsp no

and the Australian banks I looked at:

Bank Online Banking URL Header?
ANZ https://www.anz.com/INETBANK/bankmain.asp no
Bank of China https://ebs.boc.cn/BocnetClient/LoginFrameAbroad.do?_locale=en_US no
Bank of Melbourne https://ibanking.bankofmelbourne.com.au/ibank/loginPage.action no
Bankwest https://ibs.bankwest.com.au/BWLogin/rib.aspx no
Bendigobank https://www.bendigobank.com.au/banking/BBLIBanking/ no
Bank of Queensland https://www.ib.boq.com.au/boqbl no
Citibank https://www.citibank.com.au/AUGCB/JSO/signon/DisplayUsernameSignon.do no
Commonwealth Bank https://www.my.commbank.com.au/netbank/Logon/Logon.aspx no
Heritage Bank https://online.hbs.net.au/hbsv47/ntv471.asp?wci=entry no
HSBC https://www.hsbc.com.au/1/2/HUB_IDV2/IDV_EPP... no
Mebank https://ib.mebank.com.au/ME no
NAB https://ib.nab.com.au/nabib/index.jsp no
Rabobank https://secure.rabodirect.com.au/exp/policyenforcer/pages/loginB2CDGPEN.jsf?login no
St. George https://ibanking.stgeorge.com.au/ibank/loginPage.action no
Suncorp Bank https://internetbanking.suncorpbank.com.au/ no
Westpac https://online.westpac.com.au/esis/Login/SrvPage no

Conclusion

So, well done ASB! Not only do you stand out from your peers, but you also allowed New Zealand to beat Australia in terms of HSTS coverage :)

Here's the script I used to generate these results: https://github.com/fmarier/hsts-check. Feel free to leave a comment or email me if I missed an Australasia-based banking site.