Load Testing and Tail Latency

What "fast" means for a service

A service is not fast or slow; it has a latency distribution at a given throughput. "p99 of 20 ms at 5,000 requests per second, with the knee at 7,000" is a statement about performance. "Average 8 ms" is not — averages hide exactly the requests users complain about. This page covers the queueing behaviour behind tail latency, how to measure it without fooling yourself, and how to find a system's real capacity. The background terms are in Latency vs. Throughput.

Measurements: Go 1.26.4, Apple M1 Pro.

Why tails matter more than medians

  • Fan-out amplifies tails. A page that calls 50 backends in parallel is as slow as the slowest of them. If each backend has a 1% chance of exceeding 100 ms, the page exceeds 100 ms with probability 1 − 0.99⁵⁰ ≈ 39.5%. A backend's p99 is the page's median.
  • Users make many requests. A session of 100 requests is likely to hit the p99 at least once (63%).
  • Tails are where systems fail: timeouts, retries, and retry storms start at the tail, not the median.

Report p50, p99, p99.9, and max, always at a stated throughput. Never average percentiles across instances or time windows (the average of two p99s is not a p99); merge histograms instead. Use log-linear histograms (HDR Histogram, Prometheus native histograms, runtime/metrics' own format) so tails are recorded with bounded relative error.

Queueing: why latency explodes near capacity

Any resource that serves requests — a CPU, a lock, a connection pool, a disk — behaves like a queue. For the simplest model (random arrivals, random service times, one server — M/M/1), the average time in the system is:

W = S / (1 − ρ)        S = service time, ρ = utilization (arrival rate × S)
Utilization ρ Latency as a multiple of service time
50% 2x
80% 5x
90% 10x
95% 20x
99% 100x

Latency is flat at low load and hyperbolic near saturation — the knee. Real systems differ in the constants, not the shape: the Execution Tracer experiment saw p99 go from 1.06 ms at 50% CPU to 1.93 ms at 90%, and to 124 ms at 110%. Two lessons follow:

  1. Run below the knee. Capacity planning targets something like 50–70% utilization of the bottleneck at peak, not 95%.
  2. Variability hurts as much as utilization. Bursty arrivals and variable service times (a GC pause, a slow query) both raise queueing delay at the same average load.

Little's Law — L = λ × W (items in the system = arrival rate × time in system) — is the back-of-the-envelope tool that connects them. 2,000 requests/s at 50 ms each means 100 requests in flight on average; if your connection pool holds 20 connections, that pool, not the CPU, is your bottleneck.

Coordinated omission: the load test that lies

Most naive load generators are closed loop: N workers each send a request, wait for the response, and send the next. When the server stalls, the workers stall with it — they stop sending, so the requests that would have arrived during the stall are never sent, and their latency is never recorded. The tool coordinates with the server to omit the bad data.

Experiment: a service does 1 ms of work, but every 2 seconds it freezes for 500 ms (as a GC pause, a failover, or a compaction stall would). Ten seconds of load measured two ways:

Load generator Requests p50 p99 p99.9 max
Closed loop: next request after the previous returns 6,852 1.1 ms 1.5 ms 5.5 ms 502 ms
Open loop: one request every 2 ms, latency from the scheduled send time 5,001 68.5 ms 494 ms 503 ms 507 ms

Same service, same stalls. The closed-loop test reports a p99 of 1.5 ms — each 500 ms stall delayed exactly one request, and the thousands of requests real users would have sent during the stall simply did not exist in the test. The open-loop test models users who keep arriving: each stall queues 250 requests, the queue then takes hundreds of milliseconds to drain, and more than half of all requests are affected. That is what production would see.

An open-loop generator in Go:

interval := time.Second / time.Duration(rps)
start := time.Now()
for i := 0; ; i++ {
    intended := start.Add(time.Duration(i) * interval)
    if intended.After(deadline) { break }
    time.Sleep(time.Until(intended))
    go func() {
        doRequest()
        record(time.Since(intended)) // from the scheduled time, not the actual send time
    }()
}

Tools that get this right: wrk2, vegeta, k6 (with arrival-rate executors), ghz for gRPC with a fixed rate. Tools and modes that do not (plain wrk, ab, hey, k6's default virtual-user executors) are fine for finding maximum throughput, not for latency.

Finding capacity: the procedure

  1. Define the SLO first: e.g. p99 < 50 ms with < 0.1% errors.
  2. Use a production-like environment: same instance types, same GOMAXPROCS and memory limits, realistic data volume (a database with 1,000 rows has no working-set problem), warm caches and a cold-start run.
  3. Use realistic traffic: the real mix of endpoints, payload sizes, and key distribution (Zipfian, not uniform — real access is skewed, which helps caches and hurts hot shards).
  4. Step the arrival rate open-loop: e.g. 10%, 20%, ... 150% of the expected peak, several minutes per step, recording the latency histogram and error rate at each.
  5. Find the knee — the rate where p99 starts rising steeply — and the breaking point where errors appear or latency diverges. Capacity is the highest rate that still meets the SLO, minus margin.
  6. Identify the bottleneck at the knee with the USE method (below), profiles, and a trace taken at that load.
  7. Test overload behaviour: at 2x capacity, does the service shed load and keep serving a fraction quickly, or does every request time out (congestion collapse)? Graceful degradation is designed, not accidental: bounded queues, admission control, deadlines propagated through context, and retry budgets (Resilience Patterns).
  8. Soak test: hours at expected peak to find leaks, fragmentation, and log/disk growth that short tests never reach.

Locating the bottleneck: USE and RED

USE (for every resource — CPU, memory, disk, network, but also locks, pools, and queues in your process): Utilization, Saturation (queue length or wait time), Errors.

Resource Utilization Saturation Go-specific signal
CPU process CPU / cores run-queue latency /sched/latencies:seconds, GOMAXPROCS vs. limit, cgroup throttling (nr_throttled)
Memory RSS vs. limit GC CPU fraction, page faults, swapping /gc/heap/live:bytes, gctrace, GOMEMLIMIT hits
Locks — wait time mutex profile
Connection pools in-use / max wait count and duration sql.DBStats.WaitCount, WaitDuration
Goroutines count blocked count by reason goroutine profile
Network bandwidth retransmits, accept-queue overflows ss -s, nstat, ListenOverflows
Disk busy time queue depth iostat -x aqu-sz, await

RED (for every service and endpoint): Rate, Errors, Duration (as a histogram). RED tells you that an endpoint is slow; USE tells you which resource is responsible.

Tail latency sources specific to Go services

Source Symptom Fix
CPU saturation / cgroup throttling All requests slow together in bursts; scheduler latency high Capacity, GOMAXPROCS matching the CPU limit (automatic since Go 1.25)
GC mark assists Slow requests are the ones that allocated during a GC cycle Reduce allocation; GOMEMLIMIT headroom (Garbage Collector)
Lock contention Requests serialized behind one mutex Mutex profile; shorter critical sections; sharding
Pool exhaustion Latency = pool wait; DB itself is idle Pool size from Little's Law; timeouts on acquire
Head-of-line blocking Slow requests delay fast ones on the same connection or queue Separate pools/queues per class of work
Retries Load amplification exactly when the system is struggling Retry budgets, jittered backoff, hedging only with care

Common pitfall

Load testing with a closed-loop tool, finding "p99 = 4 ms at 10,000 req/s", and shipping. In production, a 300 ms stall — a GC cycle under memory pressure, a database failover, a noisy neighbour — queues every request that arrives during it, and the real p99 is in the hundreds of milliseconds; the load test could never show it because its workers politely waited out every stall. Measure latency from the intended send time with an open-loop generator, include realistic stall sources (run the test long enough to include GC cycles, compactions, deployments), and compare the load test's histogram with production's before trusting either.

Review questions

  1. A page fans out to 50 backends. Each has a p99 of 100 ms. Roughly what fraction of page loads exceed 100 ms?
  2. Using W = S / (1 − ρ), how much does latency grow going from 80% to 95% utilization?
  3. Explain coordinated omission using the closed- vs. open-loop results. Why did the closed loop report a 1.5 ms p99?
  4. 3,000 requests/s each hold a DB connection for 20 ms. How many connections does the pool need at minimum?
  5. Which USE signals would distinguish CPU saturation from connection-pool exhaustion?