Module 8 · 8.1

English

Nederlands English Deutsch Français

A backup plan for your server

Module 8: Backups and monitoring · Reading time: 11 minutes

Learning objective: After this lesson you can make a backup plan for your own server: what you keep, where, how often and for how long, and how you know for sure that restoring works.

Key concepts: 3-2-1 rule, snapshot, provider backup, your own backup outside the provider, RPO, RTO, database dump (mariadb-dump, pg_dump), configuration in /etc, Docker volumes, encryption, ransomware, append-only, restore test

Why a plan, and not just "there's a backup anyway"?

Almost everyone who has lost a server thought there was a backup. Then it turns out it only contains the operating system and not the database, or it sits on the same server that is now gone, or it was encrypted by the same ransomware, or nobody knows the password. A backup is only worth something if you can restore it. That's why you start with a plan and finish with a test.

A plan answers five questions:

What needs to be restorable?
How much data can you afford to lose? (RPO)
How quickly does it need to work again? (RTO)
Where are the copies, and how many are there?
How do you know it works?

What needs to be restorable?

A server consists of layers. Not every layer needs to be in the backup:

Layer	Example	Backup?
Operating system and packages	Debian, nginx, PHP	Not necessarily: you can reinstall them. But do keep a list: `dpkg --get-selections`
Configuration	`/etc` (nginx, SSH, firewall, systemd units), your compose files	Yes, small and essential
Applications and websites	`/var/www`, `/opt/<app>`, `/srv`	Yes, unless they come from Git
Data	databases, uploads, Docker volumes (`/var/lib/docker/volumes`)	Yes, the most important part
Secrets	passwords, API keys, the key of your backup itself	Yes, but separately and encrypted (see below)
Logs, caches, temporary files	`/var/log`, `/var/cache`, `/tmp`	Usually not

Pay extra attention to databases. A running database writes continuously. If you copy the files of a running MariaDB or PostgreSQL, you may end up with a corrupted copy. So first make a dump: a consistent point-in-time copy as a file, which you then include in the backup.

root@web1:~# mariadb-dump --single-transaction --all-databases | gzip > /var/backups/db/alles-$(date +%F).sql.gz
root@web1:~# sudo -u postgres pg_dumpall | gzip > /var/backups/db/pg-$(date +%F).sql.gz

(mariadb-dump used to be called mysqldump; that command usually still works too.) If the database runs in a container, you do the same with docker compose exec (lesson 9.4).

RPO and RTO: how much can you afford to lose?

Two concepts from professional practice help you make choices:

RPO (recovery point objective): how much time's worth of data can you lose at most? A daily backup at 03:00 means: if the server crashes at 22:00, you've lost almost a day. For an online shop with orders that's too much; for a blog that changes once a week it's fine.
RTO (recovery time objective): how long may it take before everything works again? Restoring a full provider backup can take minutes; building a new server and restoring everything by hand can take hours.

Write them down for your own server. They determine how often you make a backup and how you store it.

Snapshot, provider backup, your own backup

On a VPS you usually have three kinds of safety net. They look alike, but they protect against different things.

1. The snapshot. A point-in-time copy of the entire virtual disk, made in your provider's panel. Quick to make, quick to restore. Ideal just before an upgrade (lesson 4.3). But: with most providers a snapshot sits on the same storage as your server. If that storage fails, or you lose your account, the snapshot is gone too. And you don't keep a snapshot for long: it grows with every change made after it. A snapshot is not a backup.

2. The provider backup. Many providers automatically make a daily copy of your whole server (sometimes as an option), on different storage. At WebOké, for example, this is a daily backup of the entire VPS that is kept for seven days; other providers work in a similar way, with their own retention periods. That's a good second layer against broken hardware and against your own mistakes from yesterday. But it's still one party: your account, your payment, their outage. And restoring usually happens for the whole server at once.

3. Your own backup, outside the provider. A copy of your data with another party or in another place, which you make and manage yourself (lesson 8.2). This is the layer that remains when everything else goes wrong.

The 3-2-1 rule

A common rule of thumb:

3 copies of your data (the original plus two backups);
on 2 different types of storage or systems;
of which 1 is in a different location.

For a VPS that could be: the data on the server, the provider's daily backup, and an encrypted backup of your own made with restic to storage at another party. A modern addition is that at least one copy cannot be deleted by whoever takes over the server (offline, or append-only, lesson 8.2). Ransomware on a server actively searches for backups so it can encrypt or delete those too. A backup that the server itself is allowed to delete offers no protection against that.

Encryption and your backup password

A backup stored with another party must be encrypted: it contains customer data, passwords and keys. restic (lesson 8.2) encrypts by default. But that creates a new risk: your backup password. If you lose it, the backup is useless. And if it's only stored on the server you've lost, the same applies.

So keep the password outside the server as well: in your password manager, and possibly on paper in a safe. The same goes for the login details of the storage where your backup lives.

Testing restores

The most important part of the plan. A restore test means: you actually restore a backup, somewhere it can't break anything, and you check whether the result works.

Small and often: restore one file or one folder to /tmp/terug and compare (diff). For example, monthly.
Big and occasionally: build a complete new test server from the backup, import the database, start the website. For example, once a year, or with every major change. This also doubles as practice for when things really go wrong (lesson 10.3).

Write down how long it took: that's your real RTO.

Retention periods

Don't keep only the latest backup. If you only discover after two weeks that a file was corrupted or deleted, you need a backup from three weeks ago. A common schedule: seven daily, four weekly and six monthly versions. restic handles this with forget (lesson 8.2). Also keep the GDPR in mind: you don't keep personal data longer than necessary, not even in backups. Record your retention periods.

In practice

Use your provider's snapshot just before risky changes and delete it afterwards; leave the provider's daily backup switched on if there is one; and on top of that, make your own encrypted backup of data and configuration to another party. At WebOké this is called Snapshots and Backups in the VPS screen; at other providers the names and retention periods differ, but the principle is the same. A hosting package or managed service often has different arrangements: check what's in place in your case.

Summary

A backup is only worth something if restoring works; start with a plan and finish with a test.
Back up mainly data and configuration: databases as a dump, /etc, websites, Docker volumes, compose files.
Decide on your RPO (how much can you lose) and RTO (how quickly must it work).
Snapshot (same storage, short-term), provider backup (different storage, same party) and your own backup outside the provider are three different layers; follow the 3-2-1 rule with at least one copy that the server itself cannot delete.
Encrypt, keep the backup password outside the server, and test restores small and often, big occasionally.

Quiz

Exercise

Exercise O8.1: Your backup plan. Make a one-page backup plan for your own server: what needs to be restorable (per layer), your RPO and RTO, which copies you have (snapshot, provider, your own) and where, your retention periods, where the backup password is kept, and when you do a restore test. Fill in the form; after submitting it you'll see an example plan to compare with your own.

Exercise O8.1: Your backup plan

You do this exercise yourself; no one checks it. Tick off what you've done and mark the exercise as done.

Decided per layer what I keep: configuration, sites, databases, volumes, secrets
RPO and RTO decided
A copy with a different provider from the server
Retention periods chosen
I know where the backup password is stored
There is a copy the server itself cannot delete

How do you back up your databases?

Where are your copies, and with which provider?

When and how do you test restoring?

Your ticks and notes only stay in this browser; we don't send them anywhere. If you're signed in, we only save in My courses that you've done the exercise.

Cookies at WebOké

Cookie settings

Necessary

Marketing