Linux · 3 steps

CPU is pinned

Separate real CPU work from processes stuck waiting on disk, then find the thread actually burning the cycles.

Every step

Find out what the load is made of

Load average counts processes waiting for disk as well as processes on CPU. A load of 40 with idle CPUs is a storage problem wearing a CPU costume, and chasing it with profilers wastes the outage.

$ uptime

$ vmstat 1 5

$ top -bn1 | head -20

Look for: In vmstat, the `r` column is processes runnable on CPU and `b` is processes blocked on I/O. Then read the CPU line: `us` user, `sy` system, `wa` iowait, `st` steal.

Attribute the CPU to a process, then a thread

A process figure is an aggregate. One runaway thread in a fifty-thread service looks like moderate load until you look per thread, and the fix is completely different.

$ ps -eo pid,ppid,comm,pcpu,pmem,etime --sort=-pcpu | head -15

$ top -H -p <PID>

$ ps -L -o pid,tid,pcpu,comm -p <PID> --sort=-pcpu | head

Look for: Whether one thread dominates or the load is spread evenly across all of them. Note the TID of the worst offender.

Ask the thread what it is doing

You now know where the cycles go but not why. A stack sample separates a genuine hot loop from a syscall storm, and those have opposite fixes.

$ cat /proc/<PID>/status | grep -E "State|Threads|voluntary"

$ strace -c -f -p <PID> # ctrl-c after ~10s for a syscall histogram

$ perf top -p <PID>

Look for: In the strace summary, whether one syscall dominates the count. In perf, whether one symbol dominates the samples.