Feeding the Cloud

I used to rely on ifupdown to bring up my iptables firewall automatically using a config like this in /etc/network/interfaces:

allow-hotplug eno1
iface eno1 inet dhcp
    pre-up iptables-restore /etc/network/iptables.up.rules

iface eno1 inet6 dhcp
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules

but I wanted to modernize my network configuration and make use of systemd-networkd after upgrading one of my servers to Debian bookworm.

Since I already wrote an iptables dispatcher script for NetworkManager, I decided to follow the same approach for systemd-networkd.

I started by installing networkd-dispatcher:

apt install networkd-dispatcher

and then adding a script for the routable state in /etc/networkd-dispatcher/routable.d/iptables:

#!/bin/sh

LOGFILE=/var/log/iptables.log

if [ "$IFACE" = lo ]; then
    echo "$0: ignoring $IFACE for \`$STATE'" >> $LOGFILE
    exit 0
fi

case "$STATE" in
    routable)
        echo "$0: restoring iptables rules for $IFACE" >> $LOGFILE
        /sbin/iptables-restore /etc/network/iptables.up.rules >> $LOGFILE 2>&1
        /sbin/ip6tables-restore /etc/network/ip6tables.up.rules >> $LOGFILE 2>&1
        ;;
    *)
        echo "$0: nothing to do with $IFACE for \`$STATE'" >> $LOGFILE
        ;;
esac

before finally making that script executable (otherwise it won't run):

chmod a+x /etc/NetworkManager/dispatcher.d/pre-up.d/iptables

With this in place, I can put my iptables rules in the usual place (/etc/network/iptables.up.rules and /etc/network/ip6tables.up.rules) and use the handy iptables-apply and ip6tables-apply commands to test any changes to my firewall rules.

Looking at /var/log/iptables.log confirms that it is being called correctly for each network interface as they are started.

A few weeks ago, I upgraded a few machines from Ubuntu 20.04 (focal) to 22.04 (jammy). Here are the things that needed fixing after the upgrade.

Network problems

Firstly, I had to fix the resolution of .local domains the same way as I did when I upgraded a different machine from 18.04 (bionic) to 20.04 (focal).

ssh agent problems

Then, I found that ssh-add no longer worked and instead returned this error:

Could not open connection to your authentication agent

While this appears to be a known issue, the work-around suggested in the i3 forum didn't work for me. What did work was the solution described in this blog post:

Add this to my ~/.bash_profile:

eval $(systemctl --user show-environment | grep SSH_AUTH_SOCK)
export SSH_AUTH_SOCK

Add this to my startup script:

/usr/bin/systemctl --user start ssh-agent.service

I'm not sure why ED25519 keys don't work in gnome-keyring since that bug was supposedly fixed a while back, but starting gnome-keyring-ssh.service instead of ssh-agent.service didn't work for me.

Packages

When it comes to specific packages, I removed this obsolete package:

popularity-contest

I also installed these two new packages:

rng-tools5 (a more modern version of rng-tools)
exfatprogs (eliminates exfat-fuse's dependency on fuse)

As always, I put any packages I backport from Debian unstable into my PPA. So far with jammy, I only had to update tiger to silence some bogus warnings.

I ran into what seemed to be a DNS problem when building a Docker container:

Err http://archive.ubuntu.com jammy Release.gpg
  Could not resolve 'archive.ubuntu.com'

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/Release.gpg  Could not resolve 'archive.ubuntu.com'

W: Some index files failed to download. They have been ignored, or old ones used instead.

I found that many solutions talked about setting the default DNS server explicitly in /etc/docker/daemon.json:

{
    "dns": ["1.1.1.1"]
}

but that didn't work for me.

I noticed however that I was having these problems whenever I connected to my VPN. So what did work for me was restarting the docker daemon whenever there is a change in networking (e.g. enabling/disabling VPN) by putting the following in /etc/NetworkManager/dispatcher.d/docker-local:

#!/bin/sh

LOGFILE=/var/log/docker-restarts.log

if [ -z "$1" ]; then
    echo "$0: called with no interface" >> $LOGFILE
    exit 1;
fi

if [ "$1" = lo ]; then
    # Ignoring the loopback interface
    exit 0;
fi

case "$2" in
    up|vpn-up|down|vpn-down)
        echo "$0: restarting docker due to action \`$2' on interface \`$1'" >> $LOGFILE
        /bin/systemctl restart docker.service
        ;;
    *)
        echo "$0: ignoring action \`$2' on interface \`$1'" >> $LOGFILE
        ;;
esac

and then making that new file executable:

chmod +x /etc/NetworkManager/dispatcher.d/docker-local

You can confirm that it's working as intended by watching the logs:

tail -f /var/log/docker-restarts.log

while enabling/disable your VPN or your network connection. If you don't see any output, then something is wrong and the Docker restart isn't happening.

I have a computer that serves as a home server as well as a desktop machine. It has an encrypted home directory to protect user files and, in the default configuration, that unfortunately interferes with unattended reboots since someone needs to be present to enter the encryption password.

Here's how I added a timeout and made /home optional on that machine.

I started by adding a one-minute timeout on the password prompt by adding timeout=60 in my /etc/crypttab:

crypt  UUID=7e12c123-abcd-5555-8c40-900d1f8cc281  none  luks,timeout=60

then I made /home optional by adding nofail to the appropriate mount point in /etc/fstab:

/dev/mapper/crypt  /home  ext4  nodev,noatime,nosuid,nofail  0  2

Before that, the password prompt would timeout but the system would be unable to boot since one of the required partitions had failed to mount.

Now, to ensure that I don't accidentally re-create home directories for users when the system is mounted without a /home, I made the /home directory on the non-encrypted drive read-only:

umount /home
cd /home
chmod a-w .

Finally, with all of this in place, I was now happy to configure the machine to automatically reboot after a kernel panic by putting the following in /etc/sysctl.d/local.conf:

# Automatic reboot 10 seconds after a kernel panic
kernel.panic = 10

since I know that the machine will come back up just fine and that all services will be running. I simply won't be able to log into that machine as any other user than root until I manually unlock and mount /home.

After upgrading to Ubuntu Jammy and Asterisk 18.10, I saw the following messages in my logs:

WARNING[360166]: loader.c:2487 in load_modules: Module 'chan_sip' has been loaded but was deprecated in Asterisk version 17 and will be removed in Asterisk version 21.
WARNING[360174]: chan_sip.c:35468 in deprecation_notice: chan_sip has no official maintainer and is deprecated.  Migration to
WARNING[360174]: chan_sip.c:35469 in deprecation_notice: chan_pjsip is recommended.  See guides at the Asterisk Wiki:
WARNING[360174]: chan_sip.c:35470 in deprecation_notice: https://wiki.asterisk.org/wiki/display/AST/Migrating+from+chan_sip+to+res_pjsip
WARNING[360174]: chan_sip.c:35471 in deprecation_notice: https://wiki.asterisk.org/wiki/display/AST/Configuring+res_pjsip

and so I decided it was time to stop postponing the overdue migration of my working setup from chan_sip to res_pjsip.

It turns out that it was not as painful as I expected, though the conversion script bundled with Asterisk didn't work for me out of the box.

Debugging

Before you start, one very important thing to note is that the SIP debug information you used to see when running this in the asterisk console (asterisk -r):

sip set debug on

now lives behind this command:

pjsip set logger on

SIP phones

The first thing I migrated was the config for my two SIP phones (Snom 300 and Snom D715).

The original config for them in sip.conf was:

[2000]
; Snom 300
type=friend
qualify=yes
secret=password123
encryption=no
context=full
host=dynamic
nat=no
directmedia=no
mailbox=10@internal
vmexten=707
dtmfmode=rfc2833
call-limit=2
disallow=all
allow=g722
allow=ulaw

[2001]
; Snom D715
type=friend
qualify=yes
secret=password456
encryption=no
context=full
host=dynamic
nat=no
directmedia=yes
mailbox=10@internal
vmexten=707
dtmfmode=rfc2833
call-limit=2
disallow=all
allow=g722
allow=ulaw

and that became the following in pjsip.conf:

[transport-udp]
type = transport
protocol = udp
bind = 0.0.0.0
external_media_address = myasterisk.dyn.example.com
external_signaling_address = myasterisk.dyn.example.com
local_net = 192.168.0.0/255.255.0.0

[2000]
type = aor
max_contacts = 1

[2000]
type = auth
username = 2000
password = password123

[2000]
type = endpoint
context = full
dtmf_mode = rfc4733
disallow = all
allow = g722
allow = ulaw
direct_media = no
mailboxes = 10@internal
auth = 2000
outbound_auth = 2000
aors = 2000

[2001]
type = aor
max_contacts = 1

[2001]
type = auth
username = 2001
password = password456

[2001]
type = endpoint
context = full
dtmf_mode = rfc4733
disallow = all
allow = g722
allow = ulaw
direct_media = yes
mailboxes = 10@internal
auth = 2001
outbound_auth = 2001
aors = 2001

The different direct_media line between the two phones has to do with how they each connect to my Asterisk server and whether or not they have access to the Internet.

Internal calls

For some reason, my internal calls (from one SIP phone to the other) didn't work when using "aliases". I fixed it by changing this blurb in extensions.conf from:

[speeddial]
exten => 1000,1,Dial(SIP/2000,20)
exten => 1001,1,Dial(SIP/2001,20)

to:

[speeddial]
exten => 1000,1,Dial(${PJSIP_DIAL_CONTACTS(2000)},20)
exten => 1001,1,Dial(${PJSIP_DIAL_CONTACTS(2001)},20)

I have not yet dug into what this changes or why it's necessary and so feel free to leave a comment if you know more here.

PSTN trunk

Once I had the internal phones working, I moved to making and receiving phone calls over the PSTN, for which I use VoIP.ms with encryption.

I had to change the following in my sip.conf:

[general]
register => tls://555123_myasterisk:password789@vancouver2.voip.ms
externhost=myasterisk.dyn.example.com
localnet=192.168.0.0/255.255.0.0
tcpenable=yes
tlsenable=yes
tlscertfile=/etc/asterisk/asterisk.cert
tlsprivatekey=/etc/asterisk/asterisk.key
tlscapath=/etc/ssl/certs/

[voipms]
type=peer
host=vancouver2.voip.ms
secret=password789
defaultuser=555123_myasterisk
context=from-voipms
disallow=all
allow=ulaw
allow=g729
insecure=port,invite
canreinvite=no
trustrpid=yes
sendrpid=yes
transport=tls
encryption=yes

to the following in pjsip.conf:

[transport-tls]
type = transport
protocol = tls
bind = 0.0.0.0
external_media_address = myasterisk.dyn.example.com
external_signaling_address = myasterisk.dyn.example.com
local_net = 192.168.0.0/255.255.0.0
cert_file = /etc/asterisk/asterisk.cert
priv_key_file = /etc/asterisk/asterisk.key
ca_list_path = /etc/ssl/certs/
method = tlsv1_2

[voipms]
type = registration
transport = transport-tls
outbound_auth = voipms
client_uri = sip:555123_myasterisk@vancouver2.voip.ms
server_uri = sip:vancouver2.voip.ms

[voipms]
type = auth
password = password789
username = 555123_myasterisk

[voipms]
type = aor
contact = sip:555123_myasterisk@vancouver2.voip.ms

[voipms]
type = identify
endpoint = voipms
match = vancouver2.voip.ms

[voipms]
type = endpoint
context = from-voipms
disallow = all
allow = ulaw
allow = g729
from_user = 555123_myasterisk
trust_id_inbound = yes
media_encryption = sdes
auth = voipms
outbound_auth = voipms
aors = voipms
rtp_symmetric = yes
rewrite_contact = yes
send_rpid = yes
timers = no

The TLS method line is needed since the default in Debian OpenSSL is too strict. The timers line is to prevent outbound calls from getting dropped after 15 minutes.

Finally, I changed the Dial() lines in these extensions.conf blurbs from:

[from-voipms]
exten => 5551231000,1,Goto(2000,1)
exten => 2000,1,Dial(SIP/2000&SIP/2001,20)
exten => 2000,n,Goto(in2000-${DIALSTATUS},1)
exten => 2000,n,Hangup
exten => in2000-BUSY,1,VoiceMail(10@internal,su)
exten => in2000-BUSY,n,Hangup
exten => in2000-CONGESTION,1,VoiceMail(10@internal,su)
exten => in2000-CONGESTION,n,Hangup
exten => in2000-CHANUNAVAIL,1,VoiceMail(10@internal,su)
exten => in2000-CHANUNAVAIL,n,Hangup
exten => in2000-NOANSWER,1,VoiceMail(10@internal,su)
exten => in2000-NOANSWER,n,Hangup
exten => _in2000-.,1,Hangup(16)

[pstn-voipms]
exten => _1NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _1NXXNXXXXXX,n,Dial(SIP/voipms/${EXTEN})
exten => _1NXXNXXXXXX,n,Hangup()
exten => _NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _NXXNXXXXXX,n,Dial(SIP/voipms/1${EXTEN})
exten => _NXXNXXXXXX,n,Hangup()
exten => _011X.,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _011X.,n,Authenticate(1234)
exten => _011X.,n,Dial(SIP/voipms/${EXTEN})
exten => _011X.,n,Hangup()
exten => _00X.,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _00X.,n,Authenticate(1234)
exten => _00X.,n,Dial(SIP/voipms/${EXTEN})
exten => _00X.,n,Hangup()

to:

[from-voipms]
exten => 5551231000,1,Goto(2000,1)
exten => 2000,1,Dial(PJSIP/2000&PJSIP/2001,20)
exten => 2000,n,Goto(in2000-${DIALSTATUS},1)
exten => 2000,n,Hangup
exten => in2000-BUSY,1,VoiceMail(10@internal,su)
exten => in2000-BUSY,n,Hangup
exten => in2000-CONGESTION,1,VoiceMail(10@internal,su)
exten => in2000-CONGESTION,n,Hangup
exten => in2000-CHANUNAVAIL,1,VoiceMail(10@internal,su)
exten => in2000-CHANUNAVAIL,n,Hangup
exten => in2000-NOANSWER,1,VoiceMail(10@internal,su)
exten => in2000-NOANSWER,n,Hangup
exten => _in2000-.,1,Hangup(16)

[pstn-voipms]
exten => _1NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _1NXXNXXXXXX,n,Dial(PJSIP/${EXTEN}@voipms)
exten => _1NXXNXXXXXX,n,Hangup()
exten => _NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _NXXNXXXXXX,n,Dial(PJSIP/1${EXTEN}@voipms)
exten => _NXXNXXXXXX,n,Hangup()
exten => _011X.,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _011X.,n,Authenticate(1234)
exten => _011X.,n,Dial(PJSIP/${EXTEN}@voipms)
exten => _011X.,n,Hangup()
exten => _00X.,1,Set(CALLERID(all)=Francois Marier <5551231000>)
exten => _00X.,n,Authenticate(1234)
exten => _00X.,n,Dial(PJSIP/${EXTEN}@voipms)
exten => _00X.,n,Hangup()

Note that it's not just replacing SIP/ with PJSIP/, but it was also necessary to use a format supported by pjsip for the channel since SIP/trunkname/extension isn't supported by pjsip.

First of all, I am starting from a working Streamzap remote in Kodi. If VLC is the first application you are setting up with the Streamzap remote then you will probably need to read the above blog post first.

Once you know you have a working remote, put the following lircrc config into /home/pi/.lircrc:

begin
  prog = vlc
  button = KEY_PLAY
  config = key-play
end

begin
  prog = vlc
  button = KEY_PAUSE
  config = key-pause
end

begin
  prog = vlc
  button = KEY_STOP
  config = key-stop
end

begin
  prog = vlc
  button = KEY_POWER
  config = key-quit
end

begin
  prog = vlc
  button = KEY_NEXT
  config = key-next
end

begin
  prog = vlc
  button = KEY_PREVIOUS
  config = key-prev
end

begin
  prog = vlc
  button = KEY_RED
  config = key-toggle-fullscreen
end

begin
  prog = vlc
  button = KEY_REWIND
  config = key-slower
end

begin
  prog = vlc
  button = KEY_FORWARD
  config = key-faster
end

begin
  prog = vlc
  button = KEY_VOLUMEDOWN
  config = key-vol-down
end

begin
  prog = vlc
  button = KEY_VOLUMEUP
  config = key-vol-up
end

begin
  prog = vlc
  button = KEY_BLUE
  config = key-audio-track
end

begin
  prog = vlc
  button = KEY_MUTE
  config = key-vol-mute
end

begin
  prog = vlc
  button = KEY_LEFT
  config = key-nav-left 
end

begin
  prog = vlc
  button = KEY_DOWN
  config = key-nav-down
end

begin
  prog = vlc
  button = KEY_UP
  config = key-nav-up
end

begin
  prog = vlc
  button = KEY_RIGHT
  config = key-nav-right
end

begin
  prog = vlc
  button = KEY_MENU
  config = key-nav-activate
end

begin
  prog = vlc
  button = KEY_GREEN
  config = key-subtitle-track
end

and then after starting VLC:

Open Tools | Preferences.
Select All under Show Settings in the bottom left corner.
Open Interface | Control Interfaces in the left side-bar.
Enable Infrared remote control interface.

Now you should see lirc in the text box at the bottom of Control Interfaces and the following in your ~/.config/vlc/vlcrc:

[core]
control=lirc

If you're looking to customize the above key mapping, you can find the VLC key codes in the output of vlc -H --extended | grep -- --key-.

As part of debugging an upstream connection problem I've been seeing recently, I wanted to be able to monitor the logs from my Turris Omnia router. Here's how I configured it to send its logs to a server I already had on the local network.

Server setup

The first thing I did was to open up my server's rsyslog (Debian's default syslog server) to remote connections since it's going to be the destination host for the router's log messages.

I added the following to /etc/rsyslog.d/router.conf:

module(load="imtcp")
input(type="imtcp" port="514")

if $fromhost-ip == '192.168.1.1' then {
    if $syslogseverity <= 5 then {
        action(type="omfile" file="/var/log/router.log")
    }
    stop
}

This is using the latest rsyslog configuration method: a handy scripting language called RainerScript. Severity level 5 maps to "notice" which consists of unusual non-error conditions, and 192.168.1.1 is of course the IP address of the router on the LAN side. With this, I'm directing all router log messages to a separate file, filtering out anything less important than severity 5.

In order for rsyslog to pick up this new configuration file, I restarted it:

systemctl restart rsyslog.service

and checked that it was running correctly (e.g. no syntax errors in the new config file) using:

systemctl status rsyslog.service

Since I added a new log file, I also setup log rotation for it by putting the following in /etc/logrotate.d/router:

/var/log/router.log
{
    rotate 4
    weekly
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

In addition, since I use logcheck to monitor my server logs and email me errors, I had to add /var/log/router.log to /etc/logcheck/logcheck.logfiles.

Finally I opened the rsyslog port to the router in my server's firewall by adding the following to /etc/network/iptables.up.rules:

# Allow logs from the router
-A INPUT -s 192.168.1.1 -p tcp --dport 514 -j ACCEPT

and ran iptables-apply.

With all of this in place, it was time to get the router to send messages.

Router setup

As suggested on the Turris forum, I ssh'ed into my router and added this in /etc/syslog-ng.d/remote.conf:

destination d_loghost {
        network("192.168.1.200" time-zone("America/Vancouver"));
};

source dns {
        file("/var/log/resolver");
};

log {
        source(src);
        source(net);
        source(kernel);
        source(dns);
        destination(d_loghost);
};

Setting the timezone to the same as my server was needed because the router messages were otherwise sent with UTC timestamps.

To ensure that the destination host always gets the same IP address (192.168.1.200), I went to the advanced DHCP configuration page and added a static lease for the server's MAC address so that it always gets assigned 192.168.1.200. If that wasn't already the server's IP address, you'll have to restart it for this to take effect.

Finally, I restarted the syslog-ng daemon on the router to pick up the new config file:

/etc/init.d/syslog-ng restart

Testing

In order to test this configuration, I opened three terminal windows:

tail -f /var/log/syslog on the server
tail -f /var/log/router.log on the server
tail -f /var/log/messages on the router

I immediately started to see messages from the router in the third window and some of these, not all because of my severity-5 filter, were flowing to the second window as well. Also important is that none of the messages make it to the first window, otherwise log messages from the router would be mixed in with the server's own logs. That's the purpose of the stop command in /etc/rsyslog.d/router.conf.

To force a log messages to be emitted by the router, simply ssh into it and issue the following command:

logger Test

It should show up in the second and third windows immediately if you've got everything setup correctly

Timezone problems

If I do the following on my router:

/etc/init.d/syslog-ng restart logger TestA

I see the following in /var/log/messages:

Aug 14 20:39:35 hostname syslog-ng[9860]: syslog-ng shutting down; version='3.37.1' Aug 14 20:39:36 hostname syslog-ng[10024]: syslog-ng starting up; version='3.37.1' Aug 15 03:39:49 hostname root: TestA

The correct timezone is the one in the first two lines. Other daemon messages are displayed using an incorrect timezone like logger.

Thanks to a very helpful syslog-ng mailing list thread, I found that this is actually an upstream OpenWRT bug.

My favourite work-around is to tell syslog-ng to simply ignore the timestamp provided by the application and to use the time of reception (of the log message) instead. To do this, simply change the following in /etc/syslog-ng.conf:

source src {
    internal();
    unix-dgram("/dev/log");
};

to:

source src {
    internal();
    unix-dgram("/dev/log", keep-timestamp(no));
};

Unfortunately, I wasn't able to fix it in a way that would survive a syslog-ng package update, but since this is supposedly fixed in Turris 6.0, it shouldn't be a problem for much longer.

I had some problems getting the Gandi certbot plugin to work in Debian bullseye since the documentation appears to be outdated.

When running certbot renew --dry-run, I saw the following error message:

Plugin legacy name certbot-plugin-gandi:dns may be removed in a future version. Please use dns instead.

Thanks to an issue in another DNS plugin, I was able to easily update my configuration to the new naming convention.

Setup

The plugin we use here relies on Gandi's LiveDNS API and so you'll have to first migrate your domain to LiveDNS if you aren't already using it for your domain.

Start by getting a Developer Access API key from Gandi and then put it in /etc/letsencrypt/gandi.ini:

# live dns v5 api key
dns_gandi_api_key=ABCDEF

before make it only readable by root:

chown root:root /etc/letsencrypt/gandi.ini
chmod 600 /etc/letsencrypt/gandi.ini

Then install the required package:

apt install python3-certbot-dns-gandi

Getting an initial certificate

To get an initial certificate using the Gandi plugin, simply use the following command:

certbot certonly --authenticator dns-gandi --dns-gandi-credentials /etc/letsencrypt/gandi.ini -d example.fmarier.org

Setting up automatic renewal

If you have automatic renewals enabled, you'll want to ensure your /etc/letsencrypt/renewal/example.fmarier.org.conf file looks like this:

# renew_before_expiry = 30 days
version = 1.21.0
archive_dir = /etc/letsencrypt/archive/example.fmarier.org
cert = /etc/letsencrypt/live/example.fmarier.org/cert.pem
privkey = /etc/letsencrypt/live/example.fmarier.org/privkey.pem
chain = /etc/letsencrypt/live/example.fmarier.org/chain.pem
fullchain = /etc/letsencrypt/live/example.fmarier.org/fullchain.pem

[renewalparams]
account = abcdef
authenticator = dns-gandi
server = https://acme-v02.api.letsencrypt.org/directory
dns_gandi_credentials = /etc/letsencrypt/gandi.ini

CrashPlan recently updated itself to version 10 on my Pop!_OS laptop and stopped backing anything up.

When trying to start the client, I got faced with this error message:

Code42 cannot connect to its backend service.

Digging through log files

In /usr/local/crashplan/log/service.log.0, I found the reason why the service didn't start:

[05.18.22 07:40:05.756 ERROR main           com.backup42.service.CPService] Error starting up, java.lang.IllegalStateException: Failed to start authorized services.
STACKTRACE:: java.lang.IllegalStateException: Failed to start authorized services.
        at com.backup42.service.ClientServiceManager.authorize(ClientServiceManager.java:552)
        at com.backup42.service.CPService.startServices(CPService.java:2467)
        at com.backup42.service.CPService.start(CPService.java:562)
        at com.backup42.service.CPService.main(CPService.java:1574)
Caused by: com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.lang.UnsatisfiedLinkError: Unable to load library 'uaw':
libuaw.so: cannot open shared object file: No such file or directory
libuaw.so: cannot open shared object file: No such file or directory
Native library (linux-x86-64/libuaw.so) not found in resource path (lib/com.backup42.desktop.jar:lang)
  at com.code42.service.useractivity.UserActivityWatcherServiceImpl.<init>(UserActivityWatcherServiceImpl.java:67)
  at com.code42.service.useractivity.UserActivityWatcherServiceImpl.class(UserActivityWatcherServiceImpl.java:23)
  while locating com.code42.service.useractivity.UserActivityWatcherServiceImpl
  at com.code42.service.AbstractAuthorizedModule.addServiceWithoutBinding(AbstractAuthorizedModule.java:77)
  while locating com.code42.service.IAuthorizedService annotated with @com.google.inject.internal.Element(setName=,uniqueId=34, type=MULTIBINDER, keyType=)
  while locating java.util.Set<com.code42.service.IAuthorizedService>

1 error
        at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:226)
        at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1097)
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1126)
        at com.backup42.service.ClientServiceManager.getServices(ClientServiceManager.java:679)
        at com.backup42.service.ClientServiceManager.authorize(ClientServiceManager.java:513)
        ... 3 more
Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'uaw':
libuaw.so: cannot open shared object file: No such file or directory
libuaw.so: cannot open shared object file: No such file or directory
Native library (linux-x86-64/libuaw.so) not found in resource path (lib/com.backup42.desktop.jar:lang)
        at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:301)
        at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:461)
        at com.sun.jna.Library$Handler.<init>(Library.java:192)
        at com.sun.jna.Native.load(Native.java:596)
        at com.sun.jna.Native.load(Native.java:570)
        at com.code42.service.useractivity.UserActivityWatcherServiceImpl.<init>(UserActivityWatcherServiceImpl.java:72)
        at com.code42.service.useractivity.UserActivityWatcherServiceImpl$$FastClassByGuice$$4bcc96f8.newInstance(<generated>)
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.code42.service.AuthorizedScope$1.get(AuthorizedScope.java:38)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.code42.service.AuthorizedScope$1.get(AuthorizedScope.java:38)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:198)
        at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:151)
        at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory.get(InternalProviderInstanceBindingImpl.java:113)
        at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1094)
        ... 6 more
        Suppressed: java.lang.UnsatisfiedLinkError: libuaw.so: cannot open shared object file: No such file or directory
                at com.sun.jna.Native.open(Native Method)
                at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
                ... 28 more
        Suppressed: java.lang.UnsatisfiedLinkError: libuaw.so: cannot open shared object file: No such file or directory
                at com.sun.jna.Native.open(Native Method)
                at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:204)
                ... 28 more
        Suppressed: java.io.IOException: Native library (linux-x86-64/libuaw.so) not found in resource path (lib/com.backup42.desktop.jar:lang)
                at com.sun.jna.Native.extractFromResourcePath(Native.java:1119)
                at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:275)
                ... 28 more

[05.18.22 07:40:05.756 INFO  main         42.service.history.HistoryLogger] HISTORY:: Code42 stopped, version 10.0.0
[05.18.22 07:40:05.756 INFO  main           com.backup42.service.CPService] *****  STOPPING  *****
[05.18.22 07:40:05.757 INFO  Thread-0       com.backup42.service.CPService] ShutdownHook...calling cleanup
[05.18.22 07:40:05.759 INFO  STOPPING       com.backup42.service.CPService] SHUTDOWN:: Stopping service...

This suggests that a new library dependency (uaw) didn't get installed during the last upgrade.

Looking at the upgrade log (/usr/local/crashplan/log/upgrade..log), I found that it detected my operating system as "pop 20":

Fri May 13 07:39:51 PDT 2022: Info : Resolve Native Libraries for pop 20...
Fri May 13 07:39:51 PDT 2022: Info :   Keep common libs
Fri May 13 07:39:51 PDT 2022: Info :   Keep pop 20 libs

I unpacked the official installer (login required):

$ tar zxf CrashPlanSmb_10.0.0_15252000061000_303_Linux.tgz 
$ cd code42-install
$ gzip -dc CrashPlanSmb_10.0.0.cpi | cpio -i

and found that libuaw.so is only shipped for 4 supported platforms (rhel7, rhel8, ubuntu18 and ubuntu20):

$ find nlib/
nlib/
nlib/common
nlib/common/libfreeblpriv3.chk
nlib/common/libsoftokn3.chk
nlib/common/libsmime3.so
nlib/common/libnss3.so
nlib/common/libplc4.so
nlib/common/libssl3.so
nlib/common/libsoftokn3.so
nlib/common/libnssdbm3.so
nlib/common/libjss4.so
nlib/common/libleveldb.so
nlib/common/libfreeblpriv3.so
nlib/common/libfreebl3.chk
nlib/common/libplds4.so
nlib/common/libnssutil3.so
nlib/common/libnspr4.so
nlib/common/libfreebl3.so
nlib/common/libc42core.so
nlib/common/libc42archive64.so
nlib/common/libnssdbm3.chk
nlib/rhel7
nlib/rhel7/libuaw.so
nlib/rhel8
nlib/rhel8/libuaw.so
nlib/ubuntu18
nlib/ubuntu18/libuaw.so
nlib/ubuntu20
nlib/ubuntu20/libuaw.so

Fixing the installation script

Others have fixed this problem by copying the files manually but since Pop!_OS is based on Ubuntu, I decided to fix this by forcing the OS to be detected as "ubuntu" in the installer.

I simply edited install.sh like this:

--- install.sh.orig 2022-05-18 16:47:52.176199965 -0700
+++ install.sh  2022-05-18 16:57:26.231723044 -0700
@@ -15,7 +15,7 @@
 readonly IS_ROOT=$([[ $(id -u) -eq 0 ]] && echo true || echo false)
 readonly REQ_CMDS="chmod chown cp cpio cut grep gzip hash id ls mkdir mv sed"
 readonly APP_VERSION_FILE="c42.version.properties"
-readonly OS_NAME=$(grep "^ID=" /etc/os-release | cut -d = -f 2 | tr -d \" | tr '[:upper:]' '[:lower:]')
+readonly OS_NAME=ubuntu
 readonly OS_VERSION=$(grep "^VERSION_ID=" /etc/os-release | cut -d = -f 2 | tr -d \" | cut -d . -f1)

 SCRIPT_DIR="${0:0:${#0} - ${#SCRIPT_NAME}}"

and then ran that install script as root again to upgrade my existing installation.

I've been very happy with my Turris Omnia router and decided recently to take advantage of the fact that is is easily upgradable to replace the original radios for Wave 2 models.

I didn't go for a Wi-Fi 6-capable card because I don't have any devices that support it at this point. There is also an official WiFi 6 upgrade kit in the works and so I might just go with that later.

Wi-Fi card selection

After seeing a report that someone was already using these cards on the Omnia, I decided to look for the following:

Compex themselves don't appear to sell to consumers, but I found an American store that would sell them to me and ship to Canada:

Each card uses 4 antennas, which means that I would need an additional diplexer, an extra pigtail to SMA-RP connector, and two more antennas to wire everything up. Thankfully, the Omnia already comes with two extra holes drilled into the back of the router (covered by plastic caps) and so there is no need for drilling the case.

I put the two cards in the middle and right-most slots (they don't seem to go in the left-most slot because of the SIM card holder being in the way) without worrying about antennas just yet.

Driver installation

I made sure that the chipsets were supported in OpenWRT 19.07 (LTS 4.14 kernel) and found that support for the Qualcomm QCA9984 chipset was added in the ath10k driver as of kernel 4.8 but only for two cards apparently.

I installed the following proprietary firmware package via the advanced configuration interface:

ath10k-firmware-qca9984

and that automatically pulled in the free ath10k driver. After rebooting, I was able to see one of the two cards in the ReForis admin page and configure it.

Note that there is an alternative firmware available in OpenWRT as well (look for packages ending in -ct), but since both firmware/driver combinations gave me the same initial results, I decided to go with the defaults.

Problems

The first problem I ran into is that I could only see one of the two cards in the output of lspci (after sshing into the router). Looking for ath or wlan in the dmesg output, it doesn't look like the second card is being recognized at all.

Neither the 2.4 GHz or 5 GHz Wave 2 card worked in the right-most slot, but either of them works fine when moved to the middle slot. The stock cards work just fine in the right-most slot. I have no explanation for this.

The second problem was that I realized that the antenna holes are not all the same. The two on each end are fully round and can accommodate the diplexers which come with a round SMA-RP connector.

On the other hand, the three middle ones have a notch at the top which can only accommodate the single antenna connectors which have a flat bit on one side. I would have to file one of holes in order to add a third diplexer to my setup.

Final working configuration

Since I didn't see a way to use both new cards at once, I ended up on a different configuration that would nevertheless still upgrade both my 2.4 GHz and 5 GHz Wi-Fi.

I moved the original dual-band card to the right-most slot and switched it to the 2.4 GHz band since it's more powerful (both in dB and in throughput) than the original half-length card.

Then I put the WLE1216V5-20 into the middle slot.

The only extra thing I had to buy were two extra pigtails to SMA-RP connectors and antennas.

Here's what the final product looks like: