Reading logs with journalctl

Module 6: Services and logs · Reading time: 11 minutes

Learning objective: After this lesson you can use journalctl to search your server's logs in a targeted way (by service, priority and time), you know where other log files live, and you can stop logs from filling up your disk.

Key concepts: journald, journal, journalctl -u, -p, --since/--until, -f, -b, -e, -n, priority (err, warning, info), persistent storage, /var/log/journal, --disk-usage, --vacuum-size, /var/log, logrotate, rsyslog

The log as your first witness

When something on your server isn't working, the reason is almost always in a log. A website showing an error page, a service that won't start, a backup that didn't run last night: the server wrote it down. A good administrator looks at the log first, and only then starts changing things.

On modern Linux servers, journald (part of systemd) collects the messages from the kernel, from systemd itself and from all services in one log: the journal. You read it with journalctl.

To read everything, you need to be in the adm or systemd-journal group, or use sudo. Without those permissions you only see your own messages.

Where is it stored? Persistent or volatile

journald keeps the journal in one of two places:

• in /var/log/journal/: persistent, the logs survive a reboot;
• in /run/log/journal/: volatile, in memory, gone after a reboot.

The default setting (Storage=auto in /etc/systemd/journald.conf) chooses persistent if the directory /var/log/journal/ exists. On Debian the systemd package has created that directory since Debian 11, and it exists on Ubuntu too. Check it:

anna@web1:~$ ls -d /var/log/journal
/var/log/journal
anna@web1:~$ journalctl --list-boots | tail -n 3
 -2 3c1f... Sun 2026-11-01 04:30:12 CET Sun 2026-11-08 04:30:01 CET
 -1 9ab7... Sun 2026-11-08 04:30:40 CET Mon 2026-11-09 08:12:05 CET
  0 e41d... Mon 2026-11-09 08:12:30 CET Mon 2026-11-09 11:40:17 CET

If you only see one boot even though the server has been rebooted before, your logs are volatile. Create the directory (sudo mkdir -p /var/log/journal and sudo systemd-tmpfiles --create --prefix /var/log/journal) and restart journald. This matters: after a crash, what you want to know is exactly what happened before the reboot.

Since Debian 12, Debian no longer installs rsyslog by default. That program used to write text files such as /var/log/syslog and /var/log/auth.log. So on a new Debian server those files don't exist; everything is in the journal. Ubuntu still has rsyslog. Guides that say "look in /var/log/auth.log" translate on Debian to journalctl -u ssh.

Searching: the filters you'll actually use

Without a filter, journalctl shows everything, starting with the oldest. That's rarely what you want. The most important filters, which you can combine:

The priorities run from 0 (emerg) to 7 (debug): emerg, alert, crit, err, warning, notice, info, debug. With -p err you get err and everything more serious. -p warning also shows warnings.

A typical search. Since this morning your website has been returning a 502 error:

anna@web1:~$ systemctl status php8.4-fpm
× php8.4-fpm.service - The PHP 8.4 FastCGI Process Manager
     Active: failed (Result: exit-code) since Mon 2026-11-09 07:02:11 CET; 4h ago
...
anna@web1:~$ sudo journalctl -u php8.4-fpm --since today -p warning --no-pager
nov 09 07:02:10 web1 php-fpm8.4[1830]: [09-Nov-2026 07:02:10] ERROR: [/etc/php/8.4/fpm/pool.d/www.conf:42] unknown entry 'pm.max_children '
nov 09 07:02:10 web1 php-fpm8.4[1830]: [09-Nov-2026 07:02:10] ERROR: FPM initialization failed
nov 09 07:02:11 web1 systemd[1]: php8.4-fpm.service: Main process exited, code=exited, status=78/CONFIG
nov 09 07:02:11 web1 systemd[1]: php8.4-fpm.service: Failed with result 'exit-code'.

The cause is spelled out: a typo on line 42 of a configuration file. Status 78 (CONFIG) confirms that it's a configuration error. No guesswork needed.

Two tips for reading logs:

• Read from top to bottom, and look for the first error. One error often triggers a whole series of follow-up messages. The first one is usually the cause.
• Pay attention to the time. What changed around that moment? An automatic update (lesson 4.2: /var/log/unattended-upgrades/), a change you made, a reboot?

Other log files

Not everything is in the journal. Many programs also (or only) write their own files in /var/log/:

You read large text files with less (search with /), the end with tail -n 50, live with tail -f, and filter with grep:

anna@web1:~$ sudo grep " 502 " /var/log/nginx/access.log | tail -n 5

The logs of Docker containers aren't in /var/log but with Docker itself: docker compose logs (lesson 9.3).

Don't let logs fill your disk

A full disk is one of the most common reasons a server "suddenly" stops working: databases stop, updates fail, sometimes you can't even log in. Logs are often the culprit.

The journal limits itself. By default it uses at most 10% of the file system, with a maximum of 4 GB (SystemMaxUse=). How much it uses now and how to clean up:

anna@web1:~$ journalctl --disk-usage
Archived and active journals take up 512.0M in the file system.
anna@web1:~$ sudo journalctl --vacuum-time=30d     # remove older than 30 days
anna@web1:~$ sudo journalctl --vacuum-size=500M    # up to 500 MB

If you want a fixed limit, put this in a drop-in /etc/systemd/journald.conf.d/grootte.conf:

[Journal]
SystemMaxUse=1G

and restart journald with sudo systemctl restart systemd-journald.

Text files in /var/log are maintained by logrotate: every day or week a file is renamed (error.log.1), older versions are compressed (.gz) and the oldest are deleted. The rules per program are in /etc/logrotate.d/. Packages from the distribution bring their own rule. If a program or script of your own writes to its own log file, create a rule yourself, otherwise the file will keep growing forever.

Finding out what takes up space:

anna@web1:~$ df -h /
anna@web1:~$ sudo du -xh /var/log | sort -h | tail -n 5

Logs and privacy

Logs contain IP addresses, usernames and sometimes email addresses: that's personal data. Don't keep them longer than necessary (a few weeks to a few months is common for a small server), don't leave them publicly accessible (a /logs directory in your webroot is a well-known mistake) and don't just copy them to other people's services.

On AlmaLinux and Rocky Linux

journalctl works the same way. There, rsyslog is still enabled by default, with files such as /var/log/messages and /var/log/secure (the counterpart of auth.log). SELinux messages are in /var/log/audit/audit.log; ausearch -m avc -ts recent shows recent denials.

Summary

• journald collects all messages in the journal; read it with journalctl, filtered by service (-u), priority (-p), time (--since) and boot (-b).
• Check that the journal is persistent (/var/log/journal exists), otherwise you lose your logs after a reboot.
• Debian 12 and later have no rsyslog by default, so no /var/log/auth.log and /var/log/syslog; everything is in the journal.
• Look for the first error and check what changed around that moment.
• Keep an eye on size: the journal limits itself (--disk-usage, --vacuum-*), logrotate handles text files.

Quiz

Exercise

Exercise Z6.2: Dissecting a crashing service. The sample file journal-crash-1.txt contains the output of systemctl status and journalctl for a service that keeps restarting and then gives up. Answer: what is the service called, how many times did systemd try to start it, with which exit code did it stop, what is the cause according to the first error message, and at what time did systemd give up? The program compares your answers with the fixed key. Then check on your own server whether journalctl -p err -b shows errors, and whether your journal is persistent.

Further reading

• Manual journalctl(1): https://manpages.debian.org/trixie/systemd/journalctl.1.en.html
• Manual journald.conf(5) (Storage=, SystemMaxUse=): https://manpages.debian.org/trixie/systemd/journald.conf.5.en.html
• Debian 12, release notes 5.1.7 (rsyslog no longer installed by default): https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html