I manage a few servers for myself, friends and family and used to maintain some for the Libravatar project. Here is how I customize recent releases of Debian on those servers.

Hardware tests

apt install memtest86+ smartmontools e2fsprogs rasdaemon

Prior to spending any time configuring a new physical server, I like to ensure that the hardware is fine.

To check memory, I boot into memtest86+ from the grub menu and let it run overnight. This currently requires that Secure Boot be disabled (not enforced) in the BIOS.

Then I check the hard drives using:

smartctl -t long /dev/sdX
badblocks -swo badblocks.out /dev/sdX

Configuration

apt install etckeeper git sudo vim dialog tig

Since I use vim for all of my configuration file editing, I make it the default editor:

update-alternatives --config editor

and I turn on syntax highlighting and visual beeping globally by adding the following to /etc/vim/vimrc.local:

syntax on
set background=dark
set visualbell
set nomodeline

To keep track of the configuration changes I make in /etc/, I use etckeeper to keep that directory in a git repository and make the following changes to the default /etc/etckeeper/etckeeper.conf:

• turn off daily auto-commits
• turn off auto-commits before package installs

and then put these config files in /etc/.gitignore:

/aliases.db
/group-
/gshadow-
/passwd-
/shadow-
/subgid-
/subuid-

and this in /etc/.git/config:

[commit]
    gpgsign = false

Note that in order to fully turn off auto-commits, it's also necessary to run the following:

systemctl stop etckeeper.timer
systemctl disable etckeeper.timer

To get more control over the various packages I install, I change the default debconf level to medium:

dpkg-reconfigure debconf

ssh

apt install openssh-server mosh fail2ban

Since most of my servers are set to UTC time, I like to use my local timezone when sshing into them. Looking at file timestamps is much less confusing that way.

I also ensure that the locale I use is available on the server by adding it the list of generated locales:

dpkg-reconfigure locales

Make sure the default locale is using the UTF-8 encoding since that will ensure that things like Postgres default to the One True Encoding when you install/bootstrap them.

Other than that, I harden the ssh configuration, both server and client since my servers sometimes use ssh to back themselves up.

Since fail2ban is used to rate-limit attempts to brute-force ssh connections, you may want to whitelist your own IP addresses by adding them to /etc/fail2ban/jail.d/local.conf:

[DEFAULT]
ignoreip = 127.0.0.1/8 1.2.3.4

Then I remove the "Accepted" filter in /etc/logcheck/ignore.d.server/ssh (first line) to get a notification whenever anybody successfully logs into my server.

I also create a new group and add the users that need ssh access to it:

addgroup sshuser
adduser francois sshuser

and add a timeout for root sessions by putting this in /root/.bash_profile:

TMOUT=600

Finally, I also remove the default license notice at login:

rm /etc/motd

Security checks

apt install logcheck logcheck-database fcheck tiger debsums systemd-coredump rkhunter
apt purge john john-data rpcbind tripwire unhide unhide.rb

Logcheck is the main tool I use to keep an eye on log files, which is why I add a few additional log files to the default list by putting the following in /etc/logcheck/logcheck.logfiles.d/local.logfiles:

/var/log/apache2/error.log
/var/log/fail2ban.log

while ensuring that the apache logfiles are readable by logcheck:

chmod a+rx /var/log/apache2
chmod a+r /var/log/apache2/*

and fixing the log rotation configuration by adding the following to /etc/logrotate.d/apache2:

create 644 root adm

I also modify the main logcheck configuration file (/etc/logcheck/logcheck.conf):

INTRO=0
FQDN=0

Other than that, I enable daily checks in /etc/default/debsums and customize a few tiger settings in /etc/tiger/tigerrc:

Tiger_Check_RUNPROC=Y
Tiger_Check_DELETED=Y
Tiger_Check_APACHE=Y
Tiger_FSScan_WDIR=Y
Tiger_SSH_Protocol='2'
Tiger_Passwd_Hashes='sha512'
Tiger_Running_Procs='rsyslogd cron /usr/sbin/apache2 postgres'
Tiger_Listening_ValidProcs='sshd|mosh-server|ntpd'

I turned off the duplicate chkrootkit cronjob (it gets run automatically by tiger) using:

systemctl disable chkrootkit.timer
systemctl stop chkrootkit.timer

I also add these to /etc/fcheck/fcheck.local.cfg:

Directory      = /var/www

Exclusion      = /etc/.git/
Exclusion      = /etc/.etckeeper
Exclusion      = /etc/.gitignore
Exclusion      = /etc/mtab

and these to /etc/rkhunter.conf.local:

ALLOWHIDDENDIR=/etc/.git
ALLOWHIDDENFILE=/etc/.etckeeper
ALLOWHIDDENFILE=/etc/.gitignore

SCRIPTWHITELIST=/usr/bin/egrep
SCRIPTWHITELIST=/usr/bin/fgrep
SCRIPTWHITELIST=/usr/bin/which

ALLOWDEVFILE=/dev/shm/logcheck.*
ALLOWDEVFILE=/dev/shm/logcheck.*/ignore/*
ALLOWDEVFILE=/dev/shm/logcheck.*/violations/*
ALLOWDEVFILE=/dev/shm/logcheck.*/violations-ignore/*
ALLOWDEVFILE=/dev/shm/logcheck.*/cracking/*
ALLOWDEVFILE=/dev/shm/logcheck.*/logoutput/*
ALLOWDEVFILE=/dev/shm/logcheck.*/logoutput-sorted
ALLOWDEVFILE=/dev/shm/logcheck.*/checked
ALLOWDEVFILE=/dev/shm/PostgreSQL.*

General hardening

apt install apparmor apparmor-profiles apparmor-profiles-extra libpam-tmpdir

Certain kernel security features can be enabled by putting the following in /etc/sysctl.d/local.conf to hide kernel pointers and messages from unprivileged processes:

kernel.dmesg_restrict = 1
kernel.kptr_restrict = 1

and the following to harden the TCP stack:

net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.send_redirects = 0
net.ipv4.tcp_syncookies=1
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_source_route = 0

before reloading these settings using sysctl -p /etc/sysctl.d/local.conf.

Sandboxing in apt can be enabled by putting the following in /etc/apt/apt.conf.d/30seccomp:

APT::Sandbox::Seccomp "1";

I also restrict the use of cron to the root user by putting the following in /etc/cron.allow:

root

Instead of using the default value for UMASK, I explicitly set it to 027 in /etc/login.defs.

Timekeeping

apt install systemd-timesyncd

To keep the system clock accurate, I install the above package and then put the following in /etc/systemd/timesyncd.conf.d/local.conf:

[Time]
NTP=time.cloudflare.com

I also set the server timezone to UTC using dpkg-reconfigure tzdata.

Preventing mistakes

apt install molly-guard safe-rm sl

The above packages are all about catching mistakes (such as accidental deletions).

Package updates

apt install apticron unattended-upgrades deborphan debfoster apt-listchanges reboot-notifier popularity-contest needrestart debian-security-support distro-info

These tools help me keep packages up to date and remove unnecessary or obsolete packages from servers. On Rackspace servers, a small configuration change is needed to automatically update the monitoring tools.

On jessie or later, I install reboot-notifier to send me a notification whenever a kernel update requires a reboot to take effect.

In addition to knowing when you need to reboot your machine, the needrestart package will let you know (and offer to do it for you) when you need to restart a daemon using an obsolete library.

Finally, to check that daemon restarting is taking place, install the checkrestart tool:

apt install --no-install-recommends debian-goodies

Handy utilities

apt install renameutils atool iotop sysstat lsof mtr-tiny mc netcat-openbsd command-not-found apt-file man-db ripgrep zstd

Most of these tools are configuration-free, except for sysstat, which requires enabling data collection in /etc/default/sysstat to be useful.

Apache configuration

apt install apache2
a2dismod status

While configuring apache is often specific to each server and the services that will be running on it, there are a few common changes I make.

I enable these in /etc/apache2/conf-enabled/security.conf:

<Directory />
    AllowOverride None
    Require all denied
</Directory>
ServerTokens Prod
ServerSignature Off

I also create a new /etc/apache2/conf-available/servername.conf which contains:

ServerName machine_hostname.example.com

and then run:

a2enconf servername

Mail

apt install postfix libsasl2-modules
apt purge exim4-base exim4-daemon-light exim4-config sendmail sendmail-bin sendmail-base sendmail-cf

Configuring mail properly is tricky but the following has worked for me.

In /etc/hostname, put the bare hostname (no domain), but in /etc/mailname put the fully qualified hostname.

In /etc/hosts, make sure that the fully qualified hostname is the first alias for 127.0.0.1, followed by the bare hostname and then anything else. For example:

127.0.0.1 hostname.example.com hostname localhost

Change the following in /etc/postfix/main.cf:

inet_interfaces = loopback-only
myhostname = (fully qualified hostname)
smtpd_banner = $myhostname ESMTP
smtp_tls_security_level = may
smtp_tls_protocols = !SSLv2, !SSLv3
initial_destination_concurrency = 2

Set the following aliases in /etc/aliases:

• set francois as the destination of root emails
• set an external email address for francois
• set root as the destination for mon and www-data emails

before running newaliases to update the aliases database.

Create a new cronjob (/etc/cron.hourly/checkmail):

#!/bin/sh
ls /var/mail

to ensure that email doesn't accumulate unmonitored on this box. Don't forget to make the script executable:

chmod +x /etc/cron.hourly/checkmail

Finally, set reverse DNS for the server's IPv4 and IPv6 addresses and then test the whole setup using mail root. You should also use this online tool to make sure everything looks good.

To monitor that mail never stops flowing, add this machine to a free healthchecks.io account and create a /etc/cron.d/healthchecks-io cronjob:

0 1 * * * root echo "ping" | mail xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@hc-ping.com

Monitoring

apt install --no-install-recommends mon libfilesys-diskspace-perl libfilesys-df-perl libsys-filesystem-perl

In order to ensure that the root partition never has less than 1G of free space, I put the following in /etc/mon/mon.cf:

serverbind = 127.0.0.1
trapbind = 127.0.0.1
clientallow = 127.0.0.1

watch localhost
    service freespace
        interval 10m
        monitor freespace.monitor /:1048576 ;;
        period
            numalerts 10
            alert mail.alert root
            upalert mail.alert root
            alertevery 60m

and then systemctl restart mon.service.

Network tuning

To reduce the server's contribution to bufferbloat I change the default kernel queueing discipline (jessie or later) by putting the following in /etc/sysctl.d/local.conf:

net.core.default_qdisc=fq

and the following to improve congestion control and HTTP/2 prioritization:

net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_notsent_lowat = 16384

Power management

In order to ensure that systemd will not power down my servers to save power, I put the following in /etc/systemd/sleep.conf.d/nosuspend.conf:

[Sleep]
AllowSuspend=no
AllowHibernation=no
AllowSuspendThenHibernate=no
AllowHybridSleep=no

Fun

Finally, I also like to install these packages which generate fun stats about each server:

apt install installation-birthday uptimed