Latency vs Throughput

What is it: - Latency: time to complete one operation (end-to-end response time, measured in ms). A single user's experience. - Throughput: number of operations completed per unit time (requests per second, RPS). The system's capacity.

Why they are different: High throughput does not imply low latency. A system that queues requests and batches them has high throughput but every individual request waits in the queue (high latency). A system that processes each request immediately has low latency but may not scale to high load.

Little's Law: L = λ × W where L = average number of requests in the system (concurrency), λ = throughput (RPS), W = average latency (seconds). Rearranged: throughput = concurrency / latency. To double throughput at the same latency: double the number of parallel workers (goroutines, threads, connections).

Trade-offs: Optimizing for latency: reduce queue depth (process requests immediately), use in-process caching (avoid network round trips), colocate with data (low geographic latency). Consequence: lower utilization — resources idle between requests.

Optimizing for throughput: batch operations (send DB writes in bulk), increase parallelism, accept queue depth. Consequence: individual requests may wait in queue (higher latency at the tail).

graph LR a["Low queue depth<br/>process immediately"] --> lat["Low latency<br/>Lower throughput ceiling"] b["High queue depth<br/>batch and parallelize"] --> thr["High throughput<br/>Higher latency per request"]

p50 vs p99 latency: p50 (median) is the typical case. p99 is the worst case for 99% of requests. A service can have p50=10ms but p99=500ms if a small percentage of requests hit a slow code path (cache miss, GC pause, lock contention). SLAs are typically on p99 or p999 because outliers affect user experience most.

How to optimize latency: - Caching: eliminate DB round trips for hot data - Connection pooling: eliminate TCP+TLS setup overhead - Index optimization: eliminate full table scans - Async non-critical work: move email sending, analytics logging off the critical path - CDN: serve static assets from edge nodes near the user

How to optimize throughput: - Horizontal scaling: more instances = more parallel request handling - Async I/O (Go's goroutines, Node.js event loop): handle more concurrent requests without more threads - Batching: combine N small DB writes into one bulk INSERT - Load balancing: distribute load evenly across instances

Real-world usage: Proxel measures p99 latency of job dispatch (target: < 100ms) and throughput (target: 50 jobs/sec peak). When p99 spiked to 2s, the root cause was Redis connection pool exhaustion — all connections in use waiting for long-running XCLAIM queries. Fix: increase pool size and add timeout on XCLAIM.

Common pitfall

Optimizing average (mean) latency instead of p99 hides exactly the problem users notice most — a service can have an excellent 5ms average while 1% of users wait 2 seconds, and the average alone gives no hint that this tail exists. Averages get pulled down by the majority of fast requests; percentiles are the only way to see what the worst-served fraction of users actually experiences, which is usually what a latency SLA should be defined against, not the mean.