I recently had to migrate the main Libravatar server to a new virtual machine. In order to minimize risk and downtime, I decided to write a migration plan ahead of time.

I am sharing this plan here in case it gives any ideas to others who have to go through a similar process.

Prepare DNS

  • Change the TTL on the DNS entries for libravatar.org and libravatar.com (i.e. bare A and AAAA records) to 3600 seconds.
  • Remove the mirrors I don't control from the DNS load balancer (cdn and seccdn).
  • Remove the main server from cdn and seccdn in DNS.

Preparing the new server

  • Setup the new server.
  • Copy the database from the old site and restore it.
  • Copy /var/lib/libravatar from the old site.

  • Hack my local /etc/hosts file to point to the new server's IP address:

    xxx.xxx.xxx.xxx www.libravatar.org stats.libravatar.org cdn.libravatar.org

  • Test all functionality on the new site.

Preparing the old server

  • Prepare a static "under migration" Apache config in /etc/apache2/sites-enabled.static/default.conf:

    <VirtualHost *:80>
    RewriteEngine On
    RewriteRule ^ https://www.libravatar.org [redirect=301,last]
</VirtualHost>

<VirtualHost *:443>
    SSLEngine on

    SSLCertificateFile /etc/libravatar/www.crt
    SSLCertificateKeyFile /etc/libravatar/www.pem
    SSLCertificateChainFile /etc/libravatar/www-chain.pem

    RewriteEngine On
    RewriteRule ^ /var/www/html/migration.html [last]

    <Directory /var/www/html>
        Allow from all
        Options -Indexes
    </Directory>
</VirtualHost>

  • Put this static file in /var/www/html/migration.html:

    <html>
<body>
<p>We are migrating to a new server. See you soon!</p>
<p>- <a href="https://identi.ca/libravatar">@libravatar</a></p>
</body>
</html>

  • Enable the rewrite module:

    a2enmod rewrite

  • Prepare an Apache config proxying to the new server in /etc/apache2/sites-enabled.proxy/default.conf:

    <VirtualHost *:80>
    RewriteEngine On
    RewriteRule ^ https://www.libravatar.org [redirect=301,last]
</VirtualHost>

<VirtualHost *:443>
    SSLEngine on

    SSLCertificateFile /etc/libravatar/www.crt
    SSLCertificateKeyFile /etc/libravatar/www.pem
    SSLCertificateChainFile /etc/libravatar/www-chain.pem

    SSLProxyEngine on
    ProxyPass / https://www.libravatar.org/
    ProxyPassReverse / https://www.libravatar.org/
</VirtualHost>

  • Enable the proxy-related modules for Apache:

    a2enmod proxy
a2enmod proxy_connect
a2enmod proxy_http

Migrating servers

  • Tweet and dent about the upcoming migration.

  • Enable the static file config on the old server (disabling the Django app):

    cd /etc/apache2/
mv sites-enabled sites-enabled.django
mv sites-enabled.static sites-enabled
apache2ctl configtest
systemctl restart apache2.service

  • Disable pgbouncer to ensure that Django cannot access postgres anymore:

    systemctl stop pgbouncer.service

  • Copy the database from the old server and restore it on the new server making sure it's in the UTF8 encoding:

    dropdb libravatar
createdb -O djangouser -E utf8 libravatar
pg_restore -d libravatar libravatar20180812.pg

  • Copy /var/lib/libravatar from the old server to the new one.

    • On the new server:

      chmod a+w /var/lib/libravatar/avatar
rm -rf /var/lib/libravatar/avatar/*
chmod a+w /var/lib/libravatar/user
rm -rf /var/lib/libravatar/user/*

    • From laptop:

      rsync -a -H -v old.libravatar.org:/var/lib/libravatar/avatar .
rsync -a -H -v old.libravatar.org:/var/lib/libravatar/user .
rsync -a -H -v avatar/* new.libravatar.org:/var/lib/libravatar/avatar/
rsync -a -H -v user/* new.libravatar.org:/var/lib/libravatar/user/

    • On the new server:

      chmod go-w /var/lib/libravatar/avatar
chmod go-w /var/lib/libravatar/user
chown -R root:root /var/lib/libravatar/avatar/* /var/lib/libravatar/user/*

Disable mirror sync

  • Log into each mirror and comment out the update cron jobs in /etc/cron.d/libravatar-slave.
  • Make sure mirrors are no longer able to connect to the old server by moving /var/lib/libravatar/master/.ssh/authorized_keys to the new server and removing it from the old server.

Testing the main site

  • Hack my local /etc/hosts file to point to the new server's IPv4 address:

    xxx.xxx.xxx.xxx www.libravatar.org stats.libravatar.org cdn.libravatar.org seccdn.libravatar.org

  • Test all functionality on the new site.

  • Do a basic version of the previous test using IPv6.

  • If testing is successful, update DNS A and AAAA records (libravatar.org and libravatar.com) to point to the new server with a short TTL (in case we need to revert).

  • Enable the proxy config on the old server.

    cd /etc/apache2/
mv sites-enabled sites-enabled.static
mv sites-enabled.proxy/ sites-enabled
apache2ctl configtest
systemctl restart apache2.service

  • Hack my local /etc/hosts file to point to the old server's IP address.

  • Test basic functionality going through the proxy.
  • Remove local /etc/hosts hacks.

Re-enable mirror sync

  • Build a new libravatar-slave package with an updated known_hosts file for the new server.
  • Log into each server I control and update that package.

  • Test the connection to the master (hacking /etc/hosts on the mirror if needed):

    sudo -u libravatar-slave ssh libravatar-master@0.cdn.libravatar.org

  • Uncomment the sync cron jobs in /etc/cron.d/libravatar-slave.

  • An hour later, make sure that new images are copied over and that the TLS certs are still working.
  • Remove /etc/hosts hacks from all mirrors.

Post migration steps

  • Tweet and dent about the fact that the migration was successful.

  • Send a test email to the support address included in the tweet/dent.

  • Take a backup of config files and data on the old server in case I forgot to copy something to the new one.

  • Get in touch with mirror owners to tell them to update libravatar-slave package and test ssh configuration.

  • Add third-party controlled mirrors back to the DNS load-balancer once they are up to date.

  • A few days later, change the TTL for the main site back to 43200 seconds.

  • A week later, kill the proxy on the old server by shutting it down.