Backups with restic to another location

The daily WebOké backup (see Restoring backups from WebOké) always restores your entire VPS and is stored with the same provider. With restic you also make your own backup: encrypted, in another location, and you can restore individual files from it.

What do you need?

• A location outside your VPS that you can reach via SFTP, for example a storage service or a second server. In the examples this is backup.example.net with the user backup.
• A strong password for the backup. If you lose it, your backup is unusable. Also keep it outside the server, in your password manager.

Installing and setting up

The backup runs as root, so it can read everything.

anna@web1:~$ sudo apt install restic
anna@web1:~$ sudo -i
root@web1:~# ssh-keygen -t ed25519 -C "restic web1" -N "" -f /root/.ssh/id_backup
root@web1:~# ssh-copy-id -i /root/.ssh/id_backup.pub backup@backup.example.net
root@web1:~# install -m 600 /dev/null /root/.restic-wachtwoord
root@web1:~# nano /root/.restic-wachtwoord

In /root/.ssh/config, specify that root uses this key for the backup server:

Host backup.example.net
    User backup
    IdentityFile /root/.ssh/id_backup

Then create the repository (the place where restic stores everything):

root@web1:~# export RESTIC_REPOSITORY=sftp:backup@backup.example.net:/restic/web1
root@web1:~# export RESTIC_PASSWORD_FILE=/root/.restic-wachtwoord
root@web1:~# restic init
created restic repository 3c9a1e5b2d at sftp:backup@backup.example.net:/restic/web1

The backup script

Put this in /usr/local/sbin/backup.sh and make it executable with chmod 700 /usr/local/sbin/backup.sh:

#!/bin/sh
export RESTIC_REPOSITORY=sftp:backup@backup.example.net:/restic/web1
export RESTIC_PASSWORD_FILE=/root/.restic-wachtwoord

# 1. Databases dumpen (zie 8.2)
/usr/local/sbin/voor-backup.sh || exit 1

# 2. Apps met SQLite even stilzetten, back-up maken, weer starten
cd /srv/diensten
docker compose stop vaultwarden uptime-kuma n8n
restic backup /etc /home /root /srv /var/backups/db /var/lib/docker/volumes   --exclude /var/lib/docker/volumes/diensten_wp-db
RESULTAAT=$?
docker compose start vaultwarden uptime-kuma n8n
[ "$RESULTAAT" -eq 0 ] || exit 1

# 3. Oude back-ups opruimen
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune || exit 1

At stop and start, only list the apps you actually have; Backing up databases and Docker volumes properly explains why they need to stop briefly. You can also read there what voor-backup.sh contains. The apps always start again, even if the backup fails. With forget and --prune you keep 7 daily, 4 weekly and 6 monthly backups; the rest is cleaned up.

Automatically every night

Create /etc/systemd/system/backup.service:

[Unit]
Description=restic back-up

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup.sh

and /etc/systemd/system/backup.timer:

[Unit]
Description=restic back-up elke nacht

[Timer]
OnCalendar=*-*-* 03:15
Persistent=true

[Install]
WantedBy=timers.target

root@web1:~# systemctl daemon-reload
root@web1:~# systemctl enable --now backup.timer
root@web1:~# systemctl start backup.service
root@web1:~# journalctl -u backup.service -e

Checking and restoring

root@web1:~# restic snapshots
root@web1:~# restic check
root@web1:~# restic restore latest --target /tmp/terug --include /srv/diensten/vw-data

Restore to a separate folder such as /tmp/terug and then copy what you need. Practise this once a quarter: a backup you have never restored is a hope, not a backup.

Tip: Get notified if the backup does not run. Create a Push monitor in Uptime Kuma (see Uptime Kuma: monitor your sites and server) and add these two lines at the bottom of backup.sh. If the signal does not arrive, you get an alert.

sleep 30
curl -fsS -m 10 "https://status.example.com/api/push/JOUWCODE?status=up&msg=OK" >/dev/null