Case Study: Notification System¶

Step 1: Requirements¶

Functional: send notifications triggered by application events (a comment on your post, a price drop, a security alert) through multiple channels — push notification, email, SMS; respect per-user, per-channel preferences (opted out of email, push only).

Non-functional: must absorb sudden bursts (a breaking-news push to millions of users at once); must not send duplicate notifications for the same event; some notifications are time-sensitive (a security alert) while others tolerate delay (a weekly digest) — these need different handling, not one uniform pipeline.

Step 2: Capacity estimation¶

The defining number here is the ratio between steady-state volume and peak burst volume — a system that sends 10,000 notifications/second on average but needs to absorb 1,000,000 in the same second during a viral event has to be sized for the burst, not the average, exactly the peak-vs-average pitfall from capacity estimation generally, just especially pronounced here because notification triggers are inherently bursty (one event, many recipients) rather than smoothly distributed.

Step 3: High-level design¶

graph LR event["Event producers<br/>(app services)"] --> mq["Message queue"] mq --> worker["Notification workers"] worker --> pref["Check user preferences"] pref --> apns["Push (APNs/FCM)"] pref --> smtp["Email (SMTP)"] pref --> sms["SMS (Twilio etc.)"]

A message queue sits between the event producers (the parts of the application that decide "this needs to notify someone") and the workers that actually send notifications — this decoupling is what lets the system absorb a burst: events pile up in the queue faster than they're processed, but nothing is lost, and workers drain the queue at a sustainable rate rather than every producer blocking on slow third-party notification APIs directly.

Step 4: Deep dive — burst absorption and priority¶

graph TD event["Incoming event"] --> classify{"Time-sensitive?"} classify -->|"yes: security alert"| highq["High-priority queue"] classify -->|"no: weekly digest"| lowq["Low-priority queue"] highworker["Workers drain high-priority first"] --> highq highworker -.->|"only when high-priority queue is empty"| lowq

A single undifferentiated queue treats a security alert the same as a bulk digest email — under burst load, the security alert can end up waiting behind millions of queued digest notifications, missing its actual latency requirement. Splitting into priority queues (or one queue per notification type, drained by dedicated worker pools sized per type) lets time-sensitive notifications skip ahead of bulk ones, directly satisfying the "different handling" requirement from step 1 rather than only implicitly hoping FIFO ordering happens to work out.

Step 4 (continued): deduplication and idempotency¶

Two failure modes push toward the same notification being sent twice: a producer accidentally publishing the same event twice, or a worker crashing after sending a notification but before marking the queue message as processed (causing a redelivery). An idempotency key (a deterministic hash of event ID + recipient + channel) checked against a short-TTL store (Redis) before sending — "have I already sent this exact notification recently?" — catches both cases with one mechanism, the same idempotency-key pattern already covered for payment retries applied here to notification delivery instead of a charge.

Step 5: Bottlenecks and trade-offs¶

Third-party providers (APNs, FCM, SMTP relays, SMS gateways) have their own rate limits, independent of how fast this system can produce notifications — worker pools per channel need their own retry with exponential backoff against provider-side throttling, and a provider outage should degrade that one channel, not back up the shared queue for every channel. This is a direct application of the bulkhead pattern: separate queues/worker pools per channel mean an SMS provider outage doesn't also stall push and email delivery.

Common pitfall¶

Sizing worker capacity for the steady-state average and relying on the queue to "absorb" any burst indefinitely ignores that a queue only buffers a delay, not a capacity shortfall — if sustained burst volume exceeds worker throughput for long enough, the queue grows without bound and notification delivery lag grows with it, eventually making even the time-sensitive queue arbitrarily late. Autoscaling worker count based on queue depth (not just CPU/memory) is what actually closes this gap, rather than a fixed worker pool sized for the average case with the queue treated as an unlimited shock absorber.