A Full Disk Fails Sideways: Bytes, Inodes, and Read-Only Remounts

Free bytes, usable bytes

df counts reserved blocks as used, and on ext4 they are reserved: root can allocate them, ordinary services cannot. A filesystem that reports a few percent free can still refuse writes from the application while root-only tools keep working, which makes the failure look intermittent.

df -h /var
df --output=source,itotal,iused,ipcent,avail /var
sudo tune2fs -l /dev/sda1 | grep -i 'reserved block'

Check the reserved percentage once, decide whether the default five percent fits the filesystem size, and alert well before zero — a disk that hit 100% yesterday already caused database errors, failed certificate writes, and dropped log records.

Inodes run out first sometimes

A filesystem can have free bytes and no free inodes. The signature is No space left on device from a write of a small file, while df -h shows plenty of room. Session files, maildir queues, and cache directories with millions of tiny files are the usual owners.

df -i /var
sudo du -x --inodes --max-depth=2 /var | sort -n | tail

Use -x to stay on one filesystem; without it, a large neighbor mount makes the numbers meaningless. Deleting is not the first move: identify the producer, because a cache that regenerates a million files will simply refill the space.

Some failures remount read-only

On ext4 with the default error behavior, enough I/O errors flip the filesystem to read-only. Every subsequent write fails, but reads and dashboards stay healthy, so the host looks fine while nothing on it can record state. This is a disk or controller fault presenting as an application outage.

findmnt -no OPTIONS /
sudo dmesg --level=err,warn | grep -iE 'read-only|i/o error|remount'
journalctl -k -b --no-pager | grep -i 'read-only'

Confirm the mount options before trusting a "permissions problem" diagnosis. A read-only remount is repaired by fixing the underlying fault and remounting during a controlled window — not by re-deploying the application.

Find growth before it finds you

Most small-host disk incidents are one of three growers: container logs, the systemd journal, or an application directory with no rotation. Walk the tree once and measure them directly.

sudo du -xh --max-depth=1 /var | sort -h | tail
sudo du -sh /var/lib/docker/containers
sudo du -sh /var/log/journal
sudo lsof +L1

Container runtime JSON logs grow until the daemon's own rotation settings say otherwise; the journal has its own size caps; lsof +L1 lists deleted files still held open, whose space returns only when the holding process restarts. Set a usage alert around eighty percent, and treat any directory that crosses it as a question to answer that week, not a problem to defer.

Headroom checklist

• Check bytes and inodes together; they fail independently.
• Know the reserved-block percentage on each filesystem.
• Alert around eighty percent, not at ninety-nine.
• Recognize ENOSPC with free bytes as inode exhaustion.
• Verify mount options when writes fail but reads succeed.
• Measure container logs, journal, and app growth quarterly.
• Check lsof +L1 before assuming deletion freed space.