Linux · 4 steps

Memory is disappearing

Tell cache from genuine usage, find what is growing, and read the OOM killer’s own account of what it did.

Every step

Work out whether memory is actually low

Linux uses free memory for page cache on purpose, so "free is nearly zero" is normal and healthy. The number that matters is `available`, which accounts for cache the kernel can reclaim on demand.

$ free -h

Look for: `available` against `total`. Cache being large is fine. Swap being used at all on a latency-sensitive box is a warning.

Find what is holding the memory

RSS on its own double-counts shared pages across processes. PSS splits shared memory fairly between them, which is the number to argue with.

$ ps -eo pid,comm,rss,vsz --sort=-rss | head -15

$ smem -tk -s pss 2>/dev/null | tail -15 # PSS, if smem is installed

$ cat /proc/<PID>/status | grep -E "VmRSS|VmSwap"

Look for: Whether one process holds most of it, and whether its RSS grows steadily when you repeat the command a minute later.

Find out where inside the process it is going

A growing RSS could be the heap, a memory-mapped file, or thread stacks. The map tells you which, and that decides who owns the bug.

$ pmap -x <PID> | sort -k3 -n | tail -20

$ cat /proc/<PID>/smaps_rollup

$ ls /proc/<PID>/fd | wc -l # file descriptors leak alongside memory

Look for: Whether the growth is in `[heap]`, in an anonymous mapping, or in a mapped file. A climbing fd count alongside it is a strong signal.

Read the OOM killer’s report

The kernel logs exactly which process it chose, why, and what memory looked like at that moment. It is the most direct evidence available and people routinely ignore it.

$ dmesg -T | grep -i -A 20 "out of memory"

$ journalctl -k -S -1h | grep -i -E "oom|killed process"

$ grep -i oom /var/log/syslog | tail

Look for: The `Killed process <pid> (<name>) total-vm:… anon-rss:…` line, and the score table above it showing every candidate.