Jerry Labs

Linux logging

journald Retention Needs a Disk, a Limit, and a Proof

A retention drop-in is incomplete until a new record can be read back from the persistent journal on disk.

Reviewed 30 July 20265 min readLinux logging

Choose storage explicitly

Storage=volatile writes under /run and loses logs at reboot. Storage=persistent writes under /var/log/journal. The default, auto, behaves like persistent only when that directory exists.

A leftover directory is not proof that the current daemon writes there. If persistent logs are an operational requirement, state it in configuration instead of depending on directory history.

# /etc/systemd/journald.conf.d/10-retention.conf
[Journal]
Storage=persistent
SystemMaxUse=512M
SystemKeepFree=2G
MaxRetentionSec=14day
Compress=yes

SystemMaxUse caps journal use approximately. SystemKeepFree protects free space and may impose the tighter limit. MaxRetentionSec removes old archived files; it does not promise every record survives the full interval when space limits are reached first.

Create the disk journal correctly

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald.service
sudo journalctl --flush

systemd-tmpfiles applies the distribution's expected ownership and mode. Restarting journald loads the settings; --flush moves eligible runtime records into persistent storage.

After the restart, check the unit journal for parsing warnings. systemd-analyze cat-config systemd/journald.conf shows precedence, but it prints configuration text rather than proving journald accepted every value.

Write a marker and read it from disk

The shortest useful test creates a unique record, synchronizes the journal, and queries the persistent directory directly.

marker="journal-check-$(date +%s)-$$"
printf '%s\n' "$marker" | systemd-cat -t journal-persistence-check
sudo journalctl --sync

journal_dir="/var/log/journal/$(cat /etc/machine-id)"
sudo journalctl --directory="$journal_dir" \
  -t journal-persistence-check --no-pager -o cat \
  | grep -Fx "$marker"
sudo journalctl --directory="$journal_dir" --verify

Checking only test -d /var/log/journal can pass against stale files. Reading the new marker proves that the current service wrote to persistent storage.

Watch growth and rotation

journalctl --disk-usage
journalctl --list-boots
systemctl status systemd-journald.service

Journal limits are enforced during rotation and cleanup, so disk use may not land on an exact byte value immediately. Manual --vacuum-size or --vacuum-time operations affect archived files and should be recorded as maintenance actions.

For a real reboot test, write a marker before restart and query it from the previous boot afterward.

Retention checklist

  • Set Storage=persistent explicitly.
  • Create the journal directory through systemd-tmpfiles.
  • Define a size limit and a free-space floor.
  • Restart journald and check for parse warnings.
  • Write, sync, and read back a unique disk marker.
  • Run journal verification.
  • Check previous-boot visibility after a controlled reboot.