Storage · 3 steps

No space left on device

Find the space, including the two classic cases where it is invisible: deleted files still held open, and exhausted inodes.

Every step

Find which filesystem is full, and whether it is space or inodes

ENOSPC has two causes. A filesystem with free bytes can still refuse writes because it has run out of inodes, and du will never show you that.

$ df -h

$ df -i

Look for: Compare the two outputs. Use% at 100% in `df -h` is space; Use% at 100% in `df -i` is inodes.

Walk the tree for the space

du measures files that exist in the directory tree. Doing it per filesystem avoids counting other mounts and wasting minutes on network shares.

$ du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -15

$ du -xh --max-depth=1 /var 2>/dev/null | sort -rh | head

$ find / -xdev -type f -size +1G -exec ls -lh {} + 2>/dev/null | head

Look for: Whether du’s total roughly matches what df reports as used. A large gap is the tell for the next step.

Look for deleted files still held open

Unlinking a file removes the directory entry, but the blocks are only freed when the last file descriptor closes. A process writing to a log someone already deleted holds the space indefinitely, and du cannot see it because the file has no name.

$ lsof +L1 2>/dev/null | head -20

$ lsof -nP 2>/dev/null | grep -i deleted | head

$ ls -l /proc/<PID>/fd | grep deleted

Look for: The `(deleted)` marker and the SIZE column. The NLINK column of 0 in `lsof +L1` is the same thing said precisely.