Linux Observability Tools for Go¶

Seeing what the Go runtime cannot¶

pprof and the execution tracer see the world from inside the Go runtime. Some answers only exist outside it: which syscalls are slow, whether the kernel is throttling the container, where time goes inside the kernel or a C library, whether packets are being retransmitted, whether the disk is saturated. Debugging a Go service well means moving between both views. This page is a field guide to the Linux tools, organized by question, with the Go-specific details that make them work (or not).

Commands are for Linux; most need root or CAP_SYS_ADMIN/CAP_BPF, and inside Kubernetes are run on the node or from a privileged debug container (kubectl debug node/... -it --image=...).

Start with the 60-second checklist¶

Before any specialized tool, get the whole-machine picture (adapted from Brendan Gregg's "Linux Performance Analysis in 60 seconds"):

uptime                  # load averages: trend over 1/5/15 min
dmesg -T | tail         # OOM kills, disk errors, TCP drops
vmstat 1                # r (run queue), free mem, si/so (swap), us/sy/wa/st CPU
mpstat -P ALL 1         # one hot core? (a single-threaded bottleneck)
pidstat 1               # per-process CPU
iostat -xz 1            # disk: r/s w/s, await (latency), aqu-sz (queue), %util
free -m                 # memory and page cache
sar -n DEV 1            # network throughput per interface
sar -n TCP,ETCP 1       # connections/s, retransmits
top                     # overview

Each answers a USE question — utilization, saturation, errors — for one resource (Load Testing and Tail Latency). st (steal) in vmstat means a noisy neighbour on the hypervisor; wa means CPUs idle waiting on I/O.

The process: `/proc`¶

P=$(pgrep -f my-service)
cat /proc/$P/status | egrep 'VmRSS|VmHWM|Threads|voluntary_ctxt'  # RSS, peak RSS, threads, context switches
ls /proc/$P/fd | wc -l                  # open file descriptors
cat /proc/$P/limits | grep 'open files' # fd limit
cat /proc/$P/smaps_rollup               # RSS broken down: anonymous vs file-backed
ls /proc/$P/task | wc -l                # OS threads (Go: Ms)
cat /proc/$P/sched                      # scheduler stats: wait time, migrations

Go-specific readings:

Thread count much larger than GOMAXPROCS + a few means blocking syscalls or cgo calls are holding threads (Runtime Scheduler); confirm with /debug/pprof/threadcreate.
RSS vs. Go heap: VmRSS well above /gc/heap/live + goroutine stacks is usually GC headroom and memory not yet returned to the OS (Garbage Collector); growth with a stable Go heap points at cgo/C memory, which Go's tools cannot see.
fd count creeping toward the limit means a leak of files or sockets — ls -l /proc/$P/fd shows what they are.

Containers: cgroup limits and throttling¶

cat /sys/fs/cgroup/cpu.max            # "200000 100000" = 2 CPUs (quota/period, cgroup v2)
cat /sys/fs/cgroup/cpu.stat           # nr_periods, nr_throttled, throttled_usec
cat /sys/fs/cgroup/memory.max         # memory limit
cat /sys/fs/cgroup/memory.current     # usage, INCLUDING page cache charged to the cgroup
cat /sys/fs/cgroup/memory.events      # oom, oom_kill counts

nr_throttled growing means the container used its whole CPU quota early in some periods and was frozen for the rest — every goroutine stalls for up to the remainder of the 100 ms period. It shows up as p99 latency spikes with no explanation in Go's own tools. Check that GOMAXPROCS matches the CPU limit (automatic since Go 1.25) and that the limit is realistic for the peak.

memory.current includes page cache from file I/O; the kernel reclaims it under pressure, but it makes "memory usage" dashboards look worse than the process's real footprint. memory.events oom_kill is the ground truth for "was it OOM-killed?" (also dmesg).

Syscalls: `strace` and `perf trace`¶

strace -f -tt -T -p $P -e trace=network          # -f: all threads; -T: time spent in each syscall
strace -f -c -p $P                                # summary table: count and time per syscall
perf trace -p $P --duration 10                     # only syscalls slower than 10 ms; much lower overhead

Go-specific notes:

Use -f: Go runs on many threads; without it you see one M.
Expect futex (parking/waking Ms), epoll_pwait (netpoller), nanosleep/usleep from sysmon and spinning Ms, and read/write returning EAGAIN (non-blocking sockets — normal).
strace is expensive (it stops the process on every syscall via ptrace); a busy service can slow down 10x or more. On production systems prefer perf trace or eBPF tools.
Typical discoveries: thousands of tiny writes (missing buffering — Network Programming in Go measured 5.8 µs vs. 8 ns per message), fsync taking 50 ms, repeated openat of the same config file, connect to a DNS server on every request.

CPU: `perf`¶

perf samples stacks using hardware counters, across user code, the Go runtime, cgo, and the kernel — which pprof cannot see.

perf top -p $P                                    # live hottest functions
perf record -F 99 -g -p $P -- sleep 30            # sample stacks at 99 Hz for 30 s
perf report --no-children                         # browse
perf script | stackcollapse-perf.pl | flamegraph.pl > cpu.svg   # flame graph (FlameGraph repo)

perf stat -e cycles,instructions,cache-misses,branch-misses -p $P -- sleep 10

Go works well with perf: Go emits frame pointers on amd64 and arm64 by default, so -g gets correct stacks, and the binary's symbol table gives function names (don't strip with -s if you want this). perf stat's IPC and cache-miss counts tell you whether a hot loop is compute-bound or memory-bound (CPU and Cache Efficiency). Time in kernel functions like copy_user_enhanced_fast_string, tcp_sendmsg, or __x64_sys_futex explains costs that pprof shows only as syscall.Syscall.

eBPF: `bpftrace` and BCC tools¶

eBPF runs small, verified programs in the kernel at tracepoints, kprobes, and uprobes, with low overhead — safe enough for production.

# Off-CPU / blocking: which code paths make the process wait, and for how long
offcputime-bpfcc -p $P 10

# Latency histograms of block I/O and of a specific syscall
biolatency-bpfcc 10
bpftrace -e 'tracepoint:syscalls:sys_enter_fsync /pid == '$P'/ { @s[tid] = nsecs }
             tracepoint:syscalls:sys_exit_fsync  /@s[tid]/ { @us = hist((nsecs - @s[tid]) / 1000); delete(@s[tid]); }'

# TCP: retransmits and connection lifetimes
tcpretrans-bpfcc
tcplife-bpfcc -p $P

# Every new process / every file opened
execsnoop-bpfcc
opensnoop-bpfcc -p $P

Go caveat for uprobes on Go functions: return probes (uretprobe) are unsafe on Go code because goroutine stacks move (the probe's trampoline return address gets copied), which can crash the process. Entry uprobes are fine; for return values and latency of Go functions, prefer the execution tracer, or tools built for Go's ABI. Kernel-side probes (syscalls, TCP, block I/O) have no such issue.

Network¶

ss -tanp state established '( sport = :8080 )' | wc -l   # connections to our port
ss -tmi dst 10.0.0.12                                     # per-socket: rtt, cwnd, retrans, buffer memory
ss -s                                                      # summary: estab, timewait, orphaned
ss -ltn                                                    # listen sockets: Recv-Q = accept backlog in use
nstat -az | egrep 'ListenOverflows|ListenDrops|TCPRetransSegs|TCPTimeouts'
tcpdump -i any -nn -s 200 port 5432 -w pg.pcap             # capture for Wireshark

Accept backlog full (ListenOverflows rising, Recv-Q at the backlog size in ss -ltn): the process is not calling Accept fast enough — usually because it is CPU-starved or blocked. Clients see connection timeouts while your logs are silent.
Thousands of TIME-WAIT on a client: new connection per request; fix connection reuse.
Retransmits (ss -ti per socket, tcpretrans): packet loss on the path; latency in multiples of the RTO (200 ms+).
Send-Q growing on an established socket: the peer is not reading — a slow consumer (Network Programming in Go).

Disk¶

iostat -xz 1        # per device: r_await/w_await (ms), aqu-sz (queue depth), %util
pidstat -d 1        # per-process read/write bytes
biolatency-bpfcc    # I/O latency histogram
cat /proc/$P/io     # bytes actually read/written by the process (vs. served from page cache)

await far above the device's normal latency with high aqu-sz means the device is saturated; %util near 100% is meaningful for HDDs but misleading for SSDs and NVMe, which serve many requests in parallel (Storage Hardware).

Choosing the tool: a map¶

Question	First tool	Then
Which of my functions use CPU?	pprof CPU profile	`perf` if time is in kernel/cgo
Why is a request slow but CPU low?	Execution tracer, block/mutex profiles	`offcputime`, `perf trace` for slow syscalls
Why is the process using so much memory?	pprof heap, `runtime/metrics`	`/proc/$P/smaps_rollup`, cgroup `memory.*` for non-Go memory
Is the container being throttled?	`cpu.stat` `nr_throttled`	Compare `GOMAXPROCS` to `cpu.max`
Are packets being lost?	`nstat`, `ss -ti`	`tcpretrans`, `tcpdump`
Is the disk the bottleneck?	`iostat -xz`	`biolatency`, `pidstat -d`
What is it doing right now?	goroutine dump	`perf top`, `strace -c` (briefly)

Common pitfall¶

Running strace on a production Go service to "just take a quick look". strace stops every thread on every syscall; a Go service doing tens of thousands of syscalls per second (netpoller wakeups, futexes, reads and writes) slows down by an order of magnitude, health checks time out, the load balancer marks the instance unhealthy, and traffic shifts onto the remaining instances during an incident that was already under way. Prefer the low-overhead tools in production (perf trace, eBPF, the Go tracer and profiles), reserve strace for reproductions or for a strictly filtered, short attach (-e trace= a few syscalls, a few seconds), and always know what a tool costs before pointing it at a struggling system.

Review questions¶

A Go service has 600 OS threads with GOMAXPROCS=8. What does that suggest, and how do you confirm it?
How do you prove a container is being CPU-throttled, and what does throttling look like from inside the Go process?
Why does perf produce correct stacks for Go binaries by default, and what can it show that pprof cannot?
Why are uretprobes dangerous on Go functions?
Clients report connection timeouts but the service's logs show nothing. Which two kernel counters do you check first?