Load Balancing

graph TD subgraph l4["L4: routes the packet, doesn't look inside"] c1["client TCP stream"] -->|"forwarded as-is by IP:port"| b1["backend"] end subgraph l7["L7: terminates and re-establishes"] c2["client TCP+HTTP request"] --> term["LB terminates connection,<br/>reads the actual HTTP request"] term -->|"new connection, routed by path/header"| b2["backend"] end

L4 (transport layer): routes on IP + port, no payload inspection. Fast, protocol-agnostic. Cannot route on URL path, headers, cookies.

L7 (application layer): terminates the connection, inspects HTTP request, forwards to backend. Enables path-based routing, SSL termination, sticky sessions, request-level retry. Slightly higher latency (connection termination + re-establishment to backend).

Algorithms: - Round robin: even distribution; ignores backend capacity - Weighted round robin: proportional to backend capacity - Least connections: routes to backend with fewest active connections; better for variable-duration requests - IP hash: same client always to same backend (session affinity without cookies) - Consistent hashing: hash ring — adding/removing nodes only reassigns nearby keys

Health checks: LB probes backends periodically (TCP connect or HTTP GET). Unhealthy backends removed from rotation. Backend that passes health check but is slow: combine with circuit breaker at application level.

If a server dies: health check fails → LB removes it from the pool → all new requests go to remaining servers. Existing connections to the dead server get errors — clients should retry with exponential backoff. For zero-downtime: graceful shutdown (stop accepting new requests, drain existing, then exit).

Worked example — why "least connections" beats round robin under uneven load: three backends, all healthy. Backend A is mid-way through several requests that each take 2 seconds (a slow report-generation endpoint); backends B and C are only serving 50ms requests. Round robin sends the next request to A, B, C, A, B, C... in strict rotation regardless of how busy each one is — A keeps accumulating slow requests at the same rate as B and C accumulate fast ones, so A's queue grows without bound while B and C sit comparatively idle. Least-connections routing checks how many requests each backend is currently handling before choosing: once A has 5 requests in flight and B/C have 1 each, new requests go to B or C until the counts even out. The trade-off is that least-connections needs the load balancer to track live connection counts per backend, which round robin doesn't.

Worked example — consistent hashing under a node change: a 3-node hash ring (nodes at positions 0, 120, 240 on a 360-degree ring) is caching user sessions, each user ID hashed onto the ring and served by the next node clockwise. With plain hash(key) % N, removing one node (N: 3 → 2) changes the modulus for almost every key, so nearly all cache entries suddenly map to the "wrong" node — a near-total cache wipe. With consistent hashing, removing the node at position 120 only remaps the keys that were between position 0 and 120 (they now map to 240); keys anywhere else on the ring are unaffected, because their "next node clockwise" didn't change. Only the fraction of keys owned by the removed node gets redistributed, not the whole keyspace.