Case Study: Chat System¶

Step 1: Requirements¶

Functional: 1:1 and group messaging; online/offline presence; delivery confirmation (sent/delivered/read receipts).

Non-functional: low-latency delivery (a message should arrive in under a second when both parties are online); messages must not be lost, even if the recipient is offline when it's sent; message order within a conversation must be preserved.

Step 2: Capacity estimation¶

The number that shapes this design more than any other is concurrent connections, not requests/second — a chat system holds a persistent connection (WebSocket) open per online user, which is a fundamentally different capacity question than a stateless HTTP API's request rate. A single connection server handling, say, 50,000 concurrent connections means 100M concurrent online users needs on the order of 2,000 connection servers — a number that immediately raises the real question this case study is about: how does a message from a user on server A reach a recipient connected to server B?

Step 3: High-level design¶

graph LR a["User A"] -->|"WebSocket"| s1["Chat server 1"] b["User B"] -->|"WebSocket"| s2["Chat server 2"] s1 --> registry["Connection registry<br/>(user_id -> which server)"] s1 -->|"route via registry lookup"| s2 s1 --> mq["Message queue<br/>(for offline delivery)"] s1 --> store["Message store<br/>(durable history)"]

Step 4: Deep dive — routing a message across servers¶

User A and User B are each connected to a different chat server — neither server inherently knows where the other's connection lives. A connection registry (a fast key-value store, commonly Redis) maps user_id -> server_id, updated whenever a user connects or disconnects.

sequenceDiagram participant A as User A (on Server 1) participant S1 as Chat Server 1 participant Reg as Connection Registry participant S2 as Chat Server 2 participant B as User B (on Server 2) A->>S1: send message to User B S1->>Reg: where is User B connected? Reg->>S1: Server 2 S1->>S2: forward message S2->>B: deliver over WebSocket S1->>S1: write to durable message store (async)

Every message send requires this registry lookup — one Redis GET, sub-millisecond, and the actual message delivery hop between servers is server-to-server (a fast internal call), not routed back through the client. The durable write to the message store happens concurrently, not blocking delivery, since delivery latency is the higher-priority requirement and the durable write only needs to complete before the message is considered permanently safe, not before it's shown to the recipient.

Step 4 (continued): offline delivery¶

If the registry lookup finds no server for User B (they're offline), the message is written to a per-user queue (or directly to the durable store, marked undelivered) instead of being forwarded immediately. When User B reconnects, their chat server queries for undelivered messages and pushes them — the durable message store doubles as the offline-delivery mechanism, so no separate system is needed purely for "messages sent while offline." A message queue per recipient-style durable buffer, rather than a fire-and-forget send, is what makes "messages must not be lost" from step 1 actually hold.

Step 5: Bottlenecks and trade-offs¶

The connection registry is a critical shared dependency for every message send — if it's slow or unavailable, message routing stalls cluster-wide, the same single-shared-dependency trade-off the rate limiter case study runs into with Redis. Message ordering within a conversation needs explicit handling once a conversation's messages might be processed by different servers at different times — a per-conversation sequence number (assigned at write time, monotonically increasing) lets clients reorder messages correctly even if they arrive slightly out of order over the network, rather than relying on arrival order alone.

Common pitfall¶

Treating "message delivered to the recipient's WebSocket" as equivalent to "message safely persisted" skips the case where the chat server crashes between forwarding a message and writing it to durable storage — the recipient saw it, but a resend after a reconnect (or a second device syncing history) won't find it. The durable write and the real-time delivery are two separate guarantees that need to both be satisfied, and confusing "the user saw it" with "it's safely stored" is how message history quietly loses messages that were, briefly, successfully shown on screen.