Kubernetes · 2 steps

OOMKilled

Find out whether the limit was too low or the application is leaking, and why a JVM or Node process ignores the limit you set.

Every step

Confirm the kill and read the limit

Exit code 137 is 128 plus signal 9. It means SIGKILL, which is usually the cgroup OOM killer but not always — a node under pressure evicts differently, and the distinction changes the fix.

$ kubectl describe pod <pod> | grep -A5 "Last State"

$ kubectl get pod <pod> -o jsonpath="{.spec.containers[*].resources}" | jq

$ kubectl top pod <pod> --containers

Look for: `Reason: OOMKilled` against a plain SIGKILL, and the memory limit against what the container was actually using.

Decide whether it is a leak or a bad limit

A limit that is simply too low and a genuine leak both end in OOMKilled. The shape of memory over time tells them apart, and only one of them is fixed by raising the number.

$ kubectl get pod <pod> -o jsonpath="{.status.containerStatuses[*].restartCount}"

$ kubectl exec <pod> -- cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current 2>/dev/null

$ kubectl exec <pod> -- cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null # cgroup v1

Look for: Whether restarts happen at a steady interval — a sawtooth is a leak. Random intervals under load point at a limit that is too tight.