Production Incident Playbook

Method over tools

Under pressure, what matters most is rarely knowledge of a tool. It is method: mitigate first, then collect evidence before it disappears, form hypotheses that explain all of the evidence, and change one thing at a time. This page is a playbook for the incidents Go services actually have, built on the tools from the rest of this section. Each scenario lists the signals, the evidence to collect, the likely causes in order of frequency, and the fixes.

Before the incident: make evidence exist

Most of an incident's outcome is decided before it starts, by what the service exposes. A Go service in production should have:

Capability How
Profiles on demand net/http/pprof on an internal port
Continuous profiling Pyroscope / Parca / cloud profiler, so you can look at the profile from before the incident
Runtime metrics go_goroutines, go_gc_*, /sched/latencies:seconds, /gc/heap/live:bytes, /cpu/classes/gc/... (the Prometheus Go collector exports most)
RED metrics per endpoint Rate, errors, latency histograms
Dependency metrics DB pool WaitCount/WaitDuration, HTTP client errors and latency per backend
Build info Version, commit, Go version in a metric label and at /debug/version (debug.ReadBuildInfo)
Settings at startup Log GOMAXPROCS, GOMEMLIMIT, GOGC, cgroup limits
Crash evidence GOTRACEBACK=all (or crash with core dumps), logs shipped off the host
Flight recorder trace.FlightRecorder dumping on slow requests (Execution Tracer)
Unstripped binaries Kept for every release, for symbolizing profiles and cores

The general loop

  1. Mitigate. Roll back the last deploy, shift traffic, scale out, restart one instance (not all — keep one sick instance for evidence, out of rotation if possible). Restoring service comes before understanding it.
  2. Collect evidence from a sick instance before restarting it: goroutine dump (?debug=2), heap profile, 30 s CPU profile, 5 s trace, runtime/metrics snapshot, /proc and cgroup stats. A restart destroys all of it.
  3. Timeline: when did it start? What changed then — deploy, config flag, traffic pattern, dependency, certificate, data volume?
  4. Hypothesize and test: one hypothesis that explains all the signals; the cheapest observation that would disprove it.
  5. Fix, verify with the same signals, write it up — including what evidence was missing and how to have it next time.

Scenario: CPU is high

Signals: CPU near the limit, latency up, maybe throttling.

  1. CPU profile from a sick instance; compare to a healthy instance or to the continuous profiler's history (-diff_base).
  2. Is the hot code yours or the runtime's?
  3. Your function → what changed? A new code path, larger inputs (O(n²) on a list that grew), a cache that stopped hitting.
  4. runtime.mallocgc, gcBgMarkWorker, gcAssistAlloc, scanobject → GC. Check GC CPU fraction and allocation rate; find the allocations with the alloc_space profile (Reducing Allocations).
  5. runtime.findRunnable, futex, usleep dominating a busy process → scheduling overhead: too many goroutines waking for tiny units of work, or GOMAXPROCS far above the real CPU limit.
  6. syscall.Syscall → many small syscalls: missing buffering.
  7. Check nr_throttled in the cgroup and GOMAXPROCS vs. cpu.max.

Scenario: memory grows / OOMKilled

Signals: RSS climbing, OOMKilled in pod status, oom_kill in memory.events.

  1. Is the live heap growing? /gc/heap/live:bytes over hours, or successive heap profiles. If live heap is flat and RSS is high, it is headroom (GOGC) and fragmentation — set GOMEMLIMIT to ~90% of the limit and stop (Garbage Collector).
  2. If live heap grows: take two inuse_space heap profiles an hour apart and diff them (-diff_base). The growing allocation site is the leak's allocation point; ask who keeps it reachable.
  3. Check goroutine count: leaked goroutines retain everything they reference (Goroutine Leaks).
  4. Usual causes: unbounded caches/maps (and maps never shrink), sub-slices of big buffers kept alive (Memory Layout), goroutine leaks, unclosed iterators or response bodies, per-request data appended to a global slice.
  5. If the Go heap is fine but RSS grows: memory outside Go — cgo libraries, mmap, or a huge number of threads (each has a stack). /proc/$P/smaps_rollup, thread count.

Scenario: latency spikes, CPU looks fine

Signals: p99 up, p50 normal, CPU average moderate.

  1. Execution trace during a spike (or a flight-recorder dump):
  2. goroutines runnable but not running → CPU saturation at millisecond scale or throttling (scheduler latency metric);
  3. long sync block → lock contention (mutex profile) or pool exhaustion;
  4. long syscalls → disk or DNS;
  5. GC assist slices on request goroutines → allocation pressure.
  6. Dependency latency: is a downstream service or the DB slow? Check pool wait counts — a pool at its max with growing waits makes every request queue (Load Testing and Tail Latency).
  7. Correlate spikes with periodic events: GC cycles, cron jobs, compactions in an embedded store, log rotation, cache expiry storms (many keys with the same TTL expiring together).

Scenario: goroutine count explodes

Signals: go_goroutines rising monotonically, memory following.

  1. goroutine?debug=1: the group with the huge count and its stack is the leak site; debug=2 shows how long they have been blocked.
  2. Classify the wait state: chan send/chan receive → abandoned channel; IO wait → network call without deadline; sync.Mutex.Lock → lock held by something stuck (find the holder); select → missing ctx.Done() case.
  3. If instead the count rises with traffic and falls after: not a leak but unbounded concurrency — a burst queues work in goroutines. Add admission control (semaphore, bounded queue, 503 when full).

Scenario: "too many open files"

Signals: accept: too many open files, dial tcp: ... socket: too many open files.

  1. ls /proc/$P/fd | wc -l vs. the limit; ls -l /proc/$P/fd or lsof -p $P to see what they are: sockets to one backend? files?
  2. Sockets to a backend in ESTABLISHED: connections not reused — response bodies not closed, a new http.Client/Transport per request, MaxIdleConnsPerHost too low. In CLOSE_WAIT: we never closed after the peer did — a missing Close().
  3. Files: missing Close on an error path; defer f.Close() inside a long loop (defers run at function exit, not iteration end).

Scenario: the service hangs

Signals: requests time out, CPU near zero, health check may still pass.

  1. Goroutine dump. Look for many goroutines blocked in the same place, and for the one goroutine they are all waiting on (lock holder, the only receiver of a channel).
  2. Lock-ordering deadlocks show as pairs with swapped arguments (Panics, Stack Dumps, and Delve).
  3. A health check that returns OK without touching the hung path is why the orchestrator didn't restart it; make readiness checks exercise the critical dependency path (cheaply).

Scenario: regression right after a deploy

  1. Roll back first. Then investigate at leisure.
  2. Diff CPU and heap profiles between versions (the continuous profiler makes this one click).
  3. Check what else changed in the deploy: Go version (read release notes for runtime changes — e.g. GC or GOMAXPROCS defaults), dependency upgrades (go.sum diff), config, base image.
  4. Reproduce with a benchmark, bisect with git bisect run.

Post-incident: make it impossible next time

A good write-up answers: what happened (timeline), why (mechanism, not "human error"), how it was detected (and how long that took), what evidence was missing, and what systemic changes prevent the class of bug — a lint rule, a test (e.g. goleak in every package), an alert on a leading indicator (goroutine count slope, pool wait time, GC CPU fraction), a safer default (timeouts in a shared HTTP client constructor).

Common pitfall

Restarting every instance immediately. Under pressure, the fastest fix is a rolling restart — and it works, because most leaks and hangs reset with the process. But it also destroys every goroutine dump, heap profile, and trace that would have explained the cause, and the incident returns in three days at a worse time. Restart all but one instance; take that one out of the load balancer; collect the evidence listed above (it takes two minutes with pprof on an internal port); then restart it. If the bug cannot be studied in place, the incident is not over when the graphs recover — it is over when the cause is known.

Review questions

  1. Why should mitigation come before diagnosis, and what evidence must be collected before restarting?
  2. A CPU profile is dominated by runtime.scanobject and gcAssistAlloc. What do you look at next?
  3. RSS grows but the live heap is flat. What are the two most likely explanations?
  4. How do you tell a goroutine leak from unbounded concurrency under load?
  5. Many sockets are in CLOSE_WAIT. Whose bug is it, and what is it?