Resilience Patterns

stateDiagram-v2 Closed --> Open: failure rate exceeds threshold Open --> HalfOpen: cooldown period elapses HalfOpen --> Closed: probe request succeeds HalfOpen --> Open: probe request fails Closed --> Closed: request succeeds (normal operation)

Circuit breaker: Three states: Closed (normal, monitoring failure rate) → Open (all requests fail-fast, downstream recovers) → Half-Open (probe requests to test recovery, back to Closed on success or Open on failure). Prevents cascading failures — callers fail in milliseconds instead of waiting for timeouts.

Retry with exponential backoff + jitter: delay = base * 2^attempt + rand(0, base). Jitter prevents thundering herd — all clients retrying at the same time overwhelms the recovering service. Always set max retries and max delay cap. Retry only idempotent operations (or use idempotency keys for non-idempotent).

Timeout: Every outgoing call must have a deadline. Without it, a slow downstream causes goroutines to pile up, exhausting connection pools and memory. In Go: ctx, cancel := context.WithTimeout(ctx, 5*time.Second); defer cancel(). Cascading timeout: the deadline should propagate down the call chain and become shorter at each hop to ensure callers always get a response before their own deadline.

Bulkhead: Separate resource pools (goroutine pools, connection pools) per downstream. A slow downstream fills its own pool without affecting others.

Idempotency: Retries execute operations multiple times. Design all writes to be idempotent: natural idempotency (PUT with full state), idempotency key (UUID deduplication at server), conditional update (UPDATE WHERE current_state = expected_state).

Worked example — all four patterns in one incident: a checkout service calls a payment service, which starts responding slowly (2s instead of 50ms) because its database is under load. Without a timeout, every checkout request blocks for 2s waiting on payment, and goroutines/threads pile up holding open connections — the checkout service's own connection pool exhausts, and now checkout is failing for reasons that have nothing to do with payment's actual problem. Fix in order: (1) a timeout on the payment call (e.g. 500ms) bounds how long checkout waits per attempt; (2) a retry with exponential backoff + jitter handles transient blips without hammering the already-struggling payment service the instant it recovers; (3) once failures cross a threshold (say 50% of calls failing over 10 seconds), a circuit breaker opens and checkout fails fast for a cooldown period instead of still burning 500ms per request on calls that are very likely to fail anyway — giving payment's database room to recover instead of being retried into the ground; (4) a bulkhead keeps the connection pool used for payment calls separate from the pool used for, say, the shipping-rate service — so payment being slow never starves shipping's ability to get connections, even though both are called from the same checkout request. Every retried write also needs an idempotency key, since the timeout firing doesn't mean the payment didn't actually go through on payment's end — a retry without one risks charging the customer twice.