Incident Room

Something broke. Now investigate.

Guided troubleshooting workflows for Linux, networking, TLS, Kubernetes, processes, files, and the systems behind them. Start with the symptom rather than the command — every step tells you why you are running it, what to look for, and where that answer sends you next.

/

Linux 4

Load average is climbing and everything feels slow 3 steps · 6 min Separate real CPU work from processes stuck waiting on disk, then find the thread actually burning the cycles. Free memory is falling, or something just got killed 4 steps · 7 min Tell cache from genuine usage, find what is growing, and read the OOM killer’s own account of what it did. Access refused despite the permissions looking correct 2 steps · 6 min Walk the whole path, then the layers most people forget: ACLs, SELinux, capabilities, mount options and namespaces. systemctl says failed, or the service keeps restarting 3 steps · 6 min Read the Result and the exit status, which name the cause precisely — systemd’s own codes distinguish a missing binary from a bad user from a readiness timeout.

Networking 3

Connection refused, timed out, or hanging 4 steps · 7 min Work outwards from the process to the socket to the firewall. The distinction between refused and timed out narrows it enormously. Name or service not known, or the wrong address comes back 2 steps · 6 min Separate the resolver from the record, and find out which of the several DNS paths on a modern machine is actually being used. The load balancer returns 502 Bad Gateway or 503 Service Unavailable 2 steps · 6 min The two codes mean different things: 502 is a broken conversation with a backend, 503 is having no backend to talk to.

TLS 2

Certificate errors, handshake failures, or a refused client certificate 4 steps · 8 min Get the server to tell you what it is presenting, then work through name, chain, expiry, protocol and client certificates in the order they fail. Clients started rejecting the certificate, or it expires shortly 3 steps · 7 min Find which certificate in the chain actually expired, why a renewal did not take effect, and every other place the same certificate is deployed.

Kubernetes 3

A pod starts, dies, and restarts forever 2 steps · 7 min Read the previous container’s logs and its exit code — those two facts resolve most crash loops before you touch anything else. The pod will not start because the image will not pull 1 steps · 4 min The event message names the cause almost every time — it is a question of knowing which of four failures it describes. A container is killed with exit code 137 2 steps · 6 min Find out whether the limit was too low or the application is leaking, and why a JVM or Node process ignores the limit you set.

Storage 1

Writes are failing, or the disk is full and du disagrees with df 3 steps · 5 min Find the space, including the two classic cases where it is invisible: deleted files still held open, and exhausted inodes.

Process 1

It is running but doing nothing, and will not stop 2 steps · 6 min Read the process state, find what it is blocked on, and know which states cannot be killed and why.

Reference

Command Atlas 14 commands The invocations worth remembering for ss, lsof, strace, tcpdump, dig, openssl, find and the rest — with the mistake each one invites. File Investigator interactive Give it a path and it writes the whole forensic sequence for that file: timestamps, ownership, ACLs, open handles, checksums, recent changes and the processes touching it.