When things go wrong

Module 10: Staying secure · Reading time: 11 minutes

Learning objective: After this lesson you can recognise the signs of a hacked server, take the first steps without destroying evidence or backups, choose between cleaning up and reinstalling, and you know when you have to report a data breach.

Key concepts: incident, signs, isolating, snapshot as evidence, incident log, reinstalling, replacing keys and passwords, clean backup, data breach notification (72 hours), Dutch Data Protection Authority (Autoriteit Persoonsgegevens, AP), emergency plan

Recognising an incident

A break-in rarely announces itself. Usually you notice something that seems "odd". Signs to take seriously:

• High load for no reason: the processor is constantly at 100%, for example because of a cryptominer. Use top or htop to see which process it is, and which user runs it.
• Unknown processes or services: a program with a strange name, often in /tmp, /dev/shm or /var/tmp, or a systemd service or timer you did not create yourself.
• Outgoing traffic that does not add up: your provider reports that your server is sending spam or attacking other servers, or the network graphs in the panel show spikes.
• New users or keys: a line in ~/.ssh/authorized_keys you do not recognise, a user in /etc/passwd with UID 0, a new line in /etc/sudoers.d/.
• Changed files: unknown PHP files in your website, a modified crontab.
• Logs that have disappeared or are empty: attackers like to cover their tracks.

A few commands for a quick look around:

anna@web1:~$ top -o %CPU
anna@web1:~$ sudo ss -tunap | grep ESTAB
anna@web1:~$ sudo last -n 20
anna@web1:~$ awk -F: '$3 == 0 {print $1}' /etc/passwd
anna@web1:~$ sudo find /etc/systemd/system /etc/cron* -newer /etc/hostname -type f
anna@web1:~$ for u in /root /home/*; do sudo cat $u/.ssh/authorized_keys 2>/dev/null; done

last shows who logged in and when, the awk line lists all accounts with UID 0 (that should only be root), and the find line looks for units and cron files that are newer than a file created during installation.

Be aware: on a server that really has been hacked, these commands can be tampered with too. They help you suspect something, not prove that everything is clean.

The first steps: stay calm, isolate, record

The most important thing in the first few minutes is not to start deleting things in a panic. Work in this order.

1. Start a log. Write down on paper or on your laptop (not on the server) what you see, when, and what you do. You will need this later to find the cause, and for a breach notification, a police report or your insurance. The Dutch NCSC recommends this as well.

2. Isolate the server. Stop it from doing further damage (spam, attacks, sending data out) without switching it off:

• in your provider's firewall (if there is one), block everything except SSH from your own address; or
• do the same on the server itself, via the console;
• if there is no other way: switch the server off via the panel. You will lose the contents of the memory, though.

3. Take a snapshot. A snapshot in the panel records the state of the server as it is now. That is your evidence and your source for working out later how the attacker got in, even if you reinstall the server afterwards.

4. Secure your backups. Check that your backups outside the server are still intact, and make sure the hacked server can no longer reach them (for example by revoking the server's access key to the backup storage). Attackers also try to delete or encrypt backups.

Clean up or reinstall?

The tempting answer is: remove the suspicious file, stop the cryptominer, done. But you do not know what else an attacker has done: a second backdoor, a modified program, an extra key, a timer that puts it all back. A server where someone has had root can no longer be trusted.

That is why the advice from administrators, and from the Dutch NCSC too, is: reinstall affected systems and restore your data from a backup that is not itself infected. In practice:

1. A fresh installation (reinstall in the panel, or a new server).
2. Straight away, the basics from this course: updates, an ordinary user with sudo, SSH with new keys, a firewall (modules 2 to 5).
3. Reinstall your applications from official sources, ideally with your compose files or notes (lesson 10.4).
4. Restore data (database, uploads) from a backup made before the incident. Do not blindly restore code or configuration: it may contain the backdoor. Look critically at restored data too; an attacker may have been inside for longer than you think.
5. Close the hole the attacker used to get in. Otherwise it will happen again. The snapshot from step 3 helps you investigate.

Replace everything that may have leaked

Assume that everything that was on the server is in the attacker's hands:

• SSH keys: remove the public keys that were on the server from everywhere, and create new key pairs if private keys were stored on the server.
• Passwords for databases, admin panels and users of your application.
• API keys and tokens (mail, payments, cloud storage, backup storage) from .env files and configuration: revoke them with the service and replace them.
• Certificates: the private key of your certificate was on the server. Have the certificate revoked and request a new one (with Let's Encrypt that is quick, lesson 7.3).

Reporting: the GDPR and others

If there was personal data on the server (customer details, accounts, email addresses), you may have a data breach. The GDPR (Article 33) says that the controller must report a data breach to the supervisory authority without undue delay and, where possible, within 72 hours of becoming aware of it. In the Netherlands that is the Dutch Data Protection Authority (Autoriteit Persoonsgegevens, AP). A few points from the AP's guidance:

• you do not have to report if the breach is unlikely to pose a risk to the people involved; if in doubt, report it;
• if you do not have all the information yet, report within 72 hours anyway and add to it later;
• inform the people whose data has leaked if there is a high risk to them;
• record every data breach in your own register, even if you do not have to report it.

If you manage the server for a client, you are usually a processor: you must then inform the client (the controller) without undue delay so that they can report it. This lesson is not legal advice; if in doubt, check the AP's website.

Also: report abuse carried out from your server to your provider (they have often already noticed), and in the case of a serious attack, report the crime to the Dutch police (politie). If your organisation falls under the Cyberbeveiligingswet (Cbw, the Dutch Cybersecurity Act implementing NIS2), separate, shorter reporting deadlines apply.

Preparing: the emergency plan

An incident always happens at a bad moment. What you have arranged in advance saves hours:

• Know how to reach your provider's console and snapshot button, and keep your panel login details (with two-factor authentication) somewhere outside the server.
• Know where your backups are, how to restore them and that they work (lesson 8.2).
• Keep a list of all the secrets stored on the server, so you know what needs replacing.
• Keep a call list: your provider, who else needs to know about the server (client, colleague), and the contact details for a data breach notification.

In practice

In the WebOké VPS screen you will find the console, snapshots, backups and the reinstall button; other providers use roughly the same names (console or VNC, snapshot, rescue, reinstall). This course is self-help: nobody is watching your server and nobody will clean up an incident for you. That is exactly why an emergency plan you can carry out yourself is so important.

Summary

• Signs: unexplained load, unknown processes, services or keys, strange outgoing traffic, changed or missing files and logs.
• First steps: keep a log, isolate the server, take a snapshot as evidence, secure your backups.
• A server where someone had root is not cleaned up but reinstalled; restore data from a clean backup, and close the hole.
• Replace all keys, passwords, tokens and certificates that were on the server.
• Personal data affected? Report a data breach to the AP within 72 hours where possible, and record every breach.

Quiz

Exercise

Exercise O10.3: Your emergency plan on one page. Write an emergency plan for your own server of no more than one A4 page: how do you get into the panel, the console and the snapshots; where are your backups and how do you restore them; which secrets are stored on the server; who do you need to inform (and do you have to report a data breach); and which first five steps do you take if you suspect a break-in. Keep the plan outside the server.

Further reading

• No access to your VPS anymore: what now?
• Snapshots and backups of your VPS
• Attaching your own ISO or reinstalling your VPS
• Dutch NCSC, Gehackt, wat nu? (Hacked, what now?, Dutch source, in Dutch): https://www.ncsc.nl/incidenten-en-herstellen/gehackt-wat-nu
• Dutch Data Protection Authority (Autoriteit Persoonsgegevens), Meldplicht datalekken (data breach notification, Dutch source, in Dutch): https://www.autoriteitpersoonsgegevens.nl/nl/onderwerpen/beveiliging/meldplicht-datalekken