Benchmarking Methodology

A benchmark is an experiment

Writing func BenchmarkX is easy. Getting a number that answers a real question — is the new version faster, by how much, and am I sure? — is a separate skill. A benchmark result is a measurement with noise, taken in an artificial environment, of code the compiler may have transformed. Treat it like an experiment: a hypothesis, a controlled setup, repeated measurements, and a statistical comparison.

All numbers on this page: Go 1.26.4, Apple M1 Pro, macOS, laptop on AC power.

The mechanics

func BenchmarkEncode(b *testing.B) {
    ev := makeEvent()          // setup: not timed (b.Loop resets the timer)
    b.ReportAllocs()           // same as -benchmem, for this benchmark
    b.SetBytes(int64(size))    // also report MB/s
    for b.Loop() {             // Go 1.24+
        encode(ev)
    }
}

go test -run='^$' -bench=Encode -benchmem -count=10 ./pkg | tee new.txt
Flag Purpose
-run='^$' Skip unit tests
-bench=regexp Which benchmarks; sub-benchmarks with /: -bench='Join/builder'
-benchmem Report B/op and allocs/op
-count=10 Repeat each benchmark 10 times — required for statistics
-benchtime=2s or =1000x Run time per measurement, or a fixed iteration count
-cpu=1,2,4,8 Run with several GOMAXPROCS values — scalability in one command
-cpuprofile, -memprofile, -blockprofile, -mutexprofile, -trace Profile exactly this benchmark

b.Loop() (Go 1.24) replaced the classic for i := 0; i < b.N; i++. It runs the benchmark function once, manages the timer automatically, and keeps the loop body from being optimized away. Use it for new code.

Other tools: b.Run(name, fn) for sub-benchmarks (sizes, variants); b.RunParallel to measure throughput with GOMAXPROCS goroutines; b.ReportMetric(v, "unit") to report domain metrics (cache hit rate, bytes compressed) next to ns/op; b.StopTimer/StartTimer for per-iteration setup (expensive themselves — avoid in tight loops).

Pitfall 1: the compiler deletes your benchmark

A pure function whose result is unused can be inlined and then removed entirely. Three ways to benchmark the same 64-bit hash mixer:

func BenchmarkMixBN(b *testing.B)   { for i := 0; i < b.N; i++ { mix(uint64(i)) } }
func BenchmarkMixSink(b *testing.B) { for i := 0; i < b.N; i++ { Sink64 = mix(uint64(i)) } }
func BenchmarkMixLoop(b *testing.B) { i := uint64(0); for b.Loop() { mix(i); i++ } }

BenchmarkMixBN-8     1000000000   0.3148 ns/op   ← loop body deleted: measures nothing
BenchmarkMixSink-8   1000000000   1.043 ns/op    ← result stored to a global: real work
BenchmarkMixLoop-8    536976820   2.237 ns/op    ← b.Loop keeps the call and result alive

0.31 ns is about one clock cycle: no hash function runs in one cycle. Any result under ~1 ns/op means "check the assembly". b.Loop and assigning results to a package-level sink both prevent elimination; they differ from each other because of the keep-alive bookkeeping b.Loop adds around the call (next pitfall). At the nanosecond scale, the harness is part of what you measure — compare variants measured the same way, never numbers from different harnesses.

Pitfall 2: code written directly inside b.Loop

b.Loop prevents elimination by wrapping every variable assigned syntactically inside the loop braces with a keep-alive. That is right for result = f(x), and wrong for a whole algorithm pasted into the loop:

for b.Loop() {                      // inner loop pasted into the body
    var s int64
    for _, x := range v { s += x }  // every `s +=` is kept alive
}
// vs.
for b.Loop() { sumInts(v) }         // the algorithm in its own function

BenchmarkSumIntsInline-8   1188   1008098 ns/op
BenchmarkSumIntsLoop-8     2696    439818 ns/op    ← same algorithm, 2.3x "faster"
BenchmarkSumIntsBN-8       2804    427966 ns/op

The inline version measured the keep-alive overhead on every iteration of the inner loop, not the algorithm. Put the code under test in a function; make the b.Loop body a call.

Pitfall 3: measuring setup, or measuring the cache

  • Setup inside the loop (building inputs, opening files) gets timed. With b.Loop, do setup before the loop; with b.N, call b.ResetTimer() after setup.
  • Benchmarks repeat the same operation on the same data, so the data is hot in CPU caches, branch predictors are trained, and the page cache holds every file. Production often sees cold data. If the realistic case is "a random key out of 10 million", the benchmark must use 10 million keys and random access — the storage engine's Get benchmark goes from 1.1 µs to 3.0 µs per operation when the working set stops fitting in the block cache.
  • Inputs must be realistic in size and distribution. A JSON encoder benchmarked on a 3-field struct says nothing about 3 KB payloads; a sort benchmarked on already-sorted input measures the best case.

Pitfall 4: noise

On a laptop, consecutive runs of the same benchmark easily differ by 5–10%. Sources: CPU frequency scaling and thermal throttling, other processes, efficiency vs. performance cores (Apple Silicon, Intel hybrid), memory layout (code alignment can shift a hot loop's performance by several percent between builds that change unrelated code), GC timing.

Controls, in rough order of value:

  1. Repeat (-count=10 or more) and compare with statistics, never single runs.
  2. Run old and new interleaved or back-to-back on the same machine, in the same thermal state. Never compare numbers from different machines or days.
  3. Quiet the machine: close browsers and IDE indexers; plug in the laptop. On a Linux benchmark box: disable turbo boost and frequency scaling (cpupower frequency-set -g performance), pin with taskset, consider isolated cores.
  4. For noisy CI environments, track instruction counts (perf stat -e instructions) alongside time: much more stable, though they miss cache and memory effects.

Comparing with benchstat

golang.org/x/perf/cmd/benchstat turns repeated runs into medians, confidence intervals, and significance tests.

go install golang.org/x/perf/cmd/benchstat@latest
git stash;     go test -run='^$' -bench=Join -benchmem -count=10 > old.txt
git stash pop; go test -run='^$' -bench=Join -benchmem -count=10 > new.txt
benchstat old.txt new.txt

Real result — strings concatenation with += versus a pre-grown strings.Builder, joining 100 short strings, 10 runs each:

Metric += (old.txt) strings.Builder (new.txt) Change
Time per op 14,631.5 ns ± 1% 590.3 ns ± 0% −95.97% (p=0.000, n=10)
Bytes allocated per op 81,864 B ± 0% 896 B ± 0% −98.91% (p=0.000, n=10)
Allocations per op 198 ± 0% 1 ± 0% −99.49% (p=0.000, n=10)

benchstat prints this as one block per metric in the terminal; the table above contains exactly the same numbers. How to read them:

  • The value is the median of the runs; ± 1% is the half-width of the 95% confidence interval. A wide interval (±10%) means the measurement is too noisy to support small conclusions.
  • p=0.000 n=10: a Mann-Whitney U test over 10 samples per side. If p is above 0.05, benchstat prints ~ instead of a percentage: no statistically significant difference — do not report one.
  • benchstat -col /impl results.txt compares sub-benchmarks against each other within one file (e.g. Join/impl=plus vs. Join/impl=builder). -format csv gives machine-readable output if you want to turn results into tables or charts.

The += version made 198 allocations (each concatenation copies the whole string so far: O(n²) bytes); the builder makes one. This is the level of explanation a benchmark result should come with: what changed, by how much, why.

Micro, macro, and system benchmarks

Level Measures Tool Answers
Micro One function in a loop testing.B "Is this implementation faster in isolation?"
Component One subsystem with realistic data testing.B with large setups, custom harnesses (a db_bench-style tool) "How many writes/s does the engine sustain? Where does it stall?"
System The whole service under load Load generators (Load Testing) "What latency at what throughput, and where does it break?"

A micro-benchmark win is a hypothesis about the system, not a result. A 40% faster encoder matters only if encoding is a significant fraction of the request's CPU profile; confirm with a profile of the real workload before and after.

A benchmarking checklist

  1. State the question and the expected result before running anything.
  2. Use realistic input sizes, distributions, and working-set sizes.
  3. Make the b.Loop body a function call; make sure results are used.
  4. -benchmem always: allocations predict GC cost under load.
  5. -count=10, same machine, back-to-back, benchstat. Ignore differences benchstat marks with ~.
  6. Sanity-check impossibly fast results against the assembly.
  7. Profile the benchmark (-cpuprofile) to confirm it spends its time where you think.
  8. Explain the result mechanically (allocations, cache misses, syscalls, contention). A number without an explanation is not done.
  9. Keep the benchmark in the repo; regressions are caught by rerunning it.

Common pitfall

Reporting a 3% improvement from one run each. An engineer optimizes a parser, runs the benchmark once before and once after, sees 1,020 ns → 990 ns, and writes "3% faster" in the PR. Ten runs each through benchstat would likely show ± 4% intervals and ~ (p=0.4): the difference is noise. Worse, if the change made code larger and shifted alignment, the next unrelated commit can flip the result. Small improvements need many samples and a significance test, and should be confirmed by an effect that makes sense mechanically (fewer instructions, fewer allocations, fewer cache misses in perf stat). When the honest answer is "no measurable difference", say that — it is a perfectly good result that keeps simpler code.

Review questions

  1. Why is a benchmark result under 1 ns/op suspicious, and how do you verify it?
  2. What exactly does b.Loop keep alive, and why does pasting an inner loop into its body distort the result?
  3. What do ± 1% and p=0.000 n=10 mean in benchstat output, and when does it print ~?
  4. Why can a micro-benchmark that fits in L1 cache mislead you about production performance?
  5. A change makes a function 30% faster in a micro-benchmark. What do you check before claiming it improves the service?