Panics, Stack Dumps, and Delve¶

Debugging is reading evidence¶

Most production Go bugs are diagnosed from three artifacts: a panic trace, a goroutine dump, or a core dump — and occasionally a live debugger session. Each is precise if you know how to read it. This page is about reading them quickly and setting up services so the evidence exists when you need it.

Outputs on this page are real, from Go 1.26.4 on darwin/arm64, with local paths shortened.

Anatomy of a panic¶

type Order struct{ Items []string }

func firstItem(o *Order) string { return o.Items[0] }

func main() {
    var o *Order
    fmt.Println(firstItem(o))
}

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x101124d40]

goroutine 1 [running]:
main.firstItem(...)
    panicx/main.go:27
main.main()
    panicx/main.go:33 +0x60

Part	Reading
`panic: runtime error: ...`	The panic value. Runtime errors name the class: nil dereference, index out of range, nil map write, failed type assertion, integer divide by zero
`[signal SIGSEGV ... addr=0x0]`	A nil pointer dereference is a real hardware fault, turned into a panic by the runtime. `addr=0x0` (or a small address like `0x18` = field offset from nil) confirms nil; a large random address suggests memory corruption via `unsafe`/cgo
`goroutine 1 [running]`	Which goroutine panicked and its state
`main.firstItem(...)`	`(...)` means the call was inlined — arguments unavailable
`main.go:33 +0x60`	File and line; `+0x60` is the offset of the PC within the function

The same function with an empty Items slice:

panic: runtime error: index out of range [0] with length 0

Runtime error messages include the actual index and length, and for type assertions the actual dynamic type (interface conversion: interface {} is string, not int). Read the message before reading the code.

Panics vs. fatal errors¶

	`panic`	`fatal error`
Examples	nil dereference, index out of range, explicit `panic(...)`	`concurrent map writes`, `all goroutines are asleep - deadlock!`, `out of memory`, stack overflow (`goroutine stack exceeds 1000000000-byte limit`)
`recover()`	Can stop it, in a deferred function on the same goroutine	Cannot be recovered: the process exits

A panic in any goroutine that is not recovered on that goroutine kills the whole process. net/http recovers panics per handler goroutine (and logs them), but a goroutine you start inside a handler is not covered. For background goroutines that must not take the process down, recover at the top of the goroutine and report the panic with its stack (debug.Stack()) — and treat each recovered panic as a bug, not as error handling.

Goroutine dumps¶

A dump of every goroutine's stack is the single most useful artifact for a hung or slow process. Ways to get one:

Method	Notes
`kill -QUIT <pid>` (or Ctrl-\ in a terminal)	Prints all goroutines to stderr and exits (exit code 2)
`curl host:6060/debug/pprof/goroutine?debug=2`	Same format, process keeps running — requires `net/http/pprof`
`?debug=1`	Grouped by identical stack, with counts — best for leaks (Goroutine Leaks)
`GOTRACEBACK=all` on a crash	All goroutines instead of only the crashing one

A real dump from a program where two goroutines transfer money between the same two accounts in opposite directions:

goroutine 8 [sync.Mutex.Lock]:
main.transfer(0x1cf9f1a9c240, 0x1cf9f1a9c250, 0x1)
    abba/main.go:19 +0xfc
main.main.func1()
    abba/main.go:29 +0x2c
created by main.main in goroutine 1
    abba/main.go:27 +0x90

goroutine 9 [sync.Mutex.Lock]:
main.transfer(0x1cf9f1a9c250, 0x1cf9f1a9c240, 0x1)
    abba/main.go:19 +0xfc
main.main.func2()
    abba/main.go:34 +0x2c

Everything needed is there:

State in brackets: [sync.Mutex.Lock] — both are waiting for a mutex. Other common states: [chan send], [chan receive], [select], [IO wait] (network, parked in the netpoller), [syscall], [sleep], [semacquire], [GC assist wait], [running]/[runnable]. Long waits include a duration: [chan receive, 42 minutes].
Arguments: transfer(0x...240, 0x...250) vs. transfer(0x...250, 0x...240) — the same two accounts in opposite order. Both hold the lock on their first argument (line 19 is the second Lock) and wait for the other's: a lock-ordering deadlock.
created by: where each goroutine was started — the fastest way to identify anonymous goroutines.

Note that the runtime did not report all goroutines are asleep - deadlock!: the HTTP server's goroutine was still alive, so from the runtime's view the program could make progress. The built-in deadlock detector only fires when every goroutine is blocked, which in real services is almost never — partial deadlocks must be found from dumps and from symptoms (requests hanging, goroutine counts rising). Fix and prevention are in Concurrency Bugs.

Argument values¶

Arguments in traces are raw words: pointers, integers, and for strings and slices their (pointer, length, capacity) words spread over several values. ... means elided; ? (e.g. 0x0?) means the value may be inaccurate because it was in a register that was reused. With the register ABI, arguments are best-effort — use them as hints (like the swapped account pointers above), confirm in a debugger or core dump.

GOTRACEBACK¶

Setting	On a fatal panic prints
`none`	Nothing
`single` (default)	The panicking goroutine
`all`	All user goroutines
`system`	All goroutines including runtime ones, with runtime frames
`crash`	Like `system`, then aborts with SIGABRT so the OS writes a core dump

Set it with the environment variable, or debug.SetTraceback("all") at startup. Production services usually want at least all, so a crash leaves a full picture of what every goroutine was doing.

Core dumps¶

A core dump is the entire process memory at the moment of the crash: every goroutine, every heap object, every variable. On Linux:

ulimit -c unlimited                     # or LimitCORE=infinity in the systemd unit
GOTRACEBACK=crash ./server              # a fatal panic now writes a core
# or take one from a live process without stopping it (gdb's gcore):
gcore -o /tmp/server.core <pid>

dlv core ./server /tmp/server.core      # needs the exact same binary (with DWARF)

Containers need the host's kernel.core_pattern to point somewhere the container can write, or a core handler such as systemd-coredump. Builds with -ldflags='-s -w' have no DWARF and make cores much less useful; keep an unstripped copy of every deployed binary (or ship debug info separately) so you can analyze cores and profiles later.

Inside dlv core, the same commands as a live session work, read-only: goroutines to list them, goroutine 42 to switch, bt for the stack, frame 3 then locals and print someVar to see values. This is how you see the contents of the map or the state of the struct that led to the crash — information no log line captured.

Delve¶

Delve (dlv) is the Go debugger. It understands goroutines, the Go runtime, and Go types, which gdb does not do well.

go install github.com/go-delve/delve/cmd/dlv@latest

dlv debug ./cmd/server -- -port=8080   # build with optimizations off, then run
dlv test ./pkg -- -test.run=TestX      # debug a test
dlv attach <pid>                       # attach to a running process (pauses it!)
dlv exec ./server                      # debug an existing binary
dlv core ./server core.1234            # post-mortem

dlv debug and dlv test compile with -gcflags='all=-N -l' (optimizations and inlining disabled) so every variable is visible and every line steppable. Debugging an optimized binary works, but variables may show as <optimized out> and inlined frames collapse.

Commands that cover most sessions:

Command	Does
`break pkg.(*Bank).Transfer` / `b file.go:42`	Breakpoint on a function or line
`condition 1 to == 3`	Stop at breakpoint 1 only when the condition holds — essential in hot code
`on 1 print amount`	Print something every time breakpoint 1 is hit
`trace pkg.Func`	Tracepoint: log calls without stopping
`continue`, `next`, `step`, `stepout`	Run, step over, step into, step out
`print expr`, `locals`, `args`	Inspect values (`print b.accounts[3]`, `print len(s)`)
`goroutines` / `goroutines -with user`	List goroutines with their current location
`goroutine 17 bt`	Stack of one goroutine
`frame 2 locals`	Locals in a caller's frame
`watch -w obj.field`	Stop when a memory location is written (hardware watchpoint)
`restart`	Rerun from the start, keeping breakpoints

IDE integration (VS Code's Go extension, GoLand) drives the same Delve engine. Two cautions for production: dlv attach stops the whole process while you look (every request stalls), and a breakpoint hit in a hot path pauses all goroutines. Prefer dumps, profiles, the execution tracer, and logs in production; use the debugger on a reproduction.

A debugging method, not a tool¶

Reproduce — ideally as a failing test. A bug you can trigger on demand is half fixed.
Read the evidence completely — the whole panic message, the whole dump, the time it started, what changed (deploy, config, traffic).
Form one hypothesis that explains all the evidence, and a test that would disprove it.
Bisect when the cause is a change: git bisect run go test -run TestRepro ./... finds the commit mechanically.
Fix the cause, not the symptom — and keep the reproduction as a regression test.

Common pitfall¶

Recovering panics to "keep the service up". A team wraps every goroutine in defer func() { recover() }() after a nil-pointer panic took down production. The crashes stop; the bug does not. A request that panicked halfway through a multi-step update leaves a mutex locked (a Lock without defer Unlock above the panic), a map half-modified, a transaction neither committed nor rolled back. Subsequent requests hang on the mutex or read inconsistent state, and there is no stack trace to say why — the panic was swallowed. Recover only at well-defined boundaries (a request, a job), log the stack, make sure everything above that boundary is released by defer, mark the request failed, and alert on the panic count. A panic is a bug report; silencing it throws the report away.

Review questions¶

In a nil-pointer panic, what does addr=0x18 tell you that addr=0x0 does not?
Which runtime failures can recover stop, and which always kill the process?
In the deadlock dump, which two details identify a lock-ordering deadlock?
Why did the runtime not print "all goroutines are asleep - deadlock!" for that program?
What do you need to keep from each release to analyze its core dumps later?