Design a notification system

Nearly every product sends notifications: "your order shipped", "your login code is 482913", "Ana commented on your post", "20% off this weekend". Teams start by calling an email API from wherever the event happens, and end up with duplicate messages, users who can't unsubscribe, login codes stuck behind a marketing campaign, and nobody able to say whether a message was delivered. This lesson designs a central notification service that every other service uses.

#Requirements

Functional:

• Channels: mobile push (iOS and Android), email, SMS and an in-app inbox.
• Other services send a notification by type and recipient, not by channel; the system decides how to deliver it from the user's preferences.
• Users can opt out per type and channel, and set quiet hours. Legally required messages (security alerts, receipts) bypass marketing opt-outs.
• Templates with localisation and variables.
• Scheduled sends and bulk campaigns to millions of users.
• Track each message's status: queued, sent, delivered, opened, failed.

Non-functional:

• Priority: a login code must arrive within seconds even while a 10-million-user campaign is going out.
• At-least-once delivery without visible duplicates: never lose a transactional message, and never send the same one twice.
• Survive the failure of any single external provider.
• Protect users from spam caused by bugs: no user should receive 50 notifications in a minute.

#Numbers

users:                     50 million
transactional:             ~5 per user per day → 250 million/day ≈ 3,000/s avg, ~10,000/s peak
campaigns:                 a few per week, up to 50 million recipients each
campaign send rate:        50 million over 1 hour → ~14,000/s on top of normal traffic
status events:             ~3 per message (sent, delivered, opened) → ~1 billion/day

The throughput itself is modest. The difficulty is in the bursts (campaigns), the external providers (each with its own rate limits and outages), and the rules (preferences, priorities, de-duplication).

#The API: intent, not channel

Producers describe what happened, not how to deliver it:

POST /v1/notifications
Idempotency-Key: order-8812-shipped

{
  "type": "order_shipped",
  "user_id": "u_123",
  "data": {"order_id": "8812", "carrier": "DHL", "eta": "2026-11-03"}
}

202 Accepted
{"notification_id": "n_7f3a", "status": "queued"}

The idempotency key matters here more than almost anywhere else. The order service will retry when the call times out; without the key, the user gets two "shipped" emails. The notification service stores keys for a few days and returns the original response on a repeat (idempotency keys).

Each notification type is registered once, with its category (security, transactional, social, marketing), priority, default channels and templates. That registry is what lets the service apply rules consistently, rather than each producer reinventing them.

#The pipeline

The stages:

1. API: authenticates the producer, validates the type and data, de-duplicates, writes the request durably and returns 202. Nothing slow happens here.
2. Router: loads the user's contact details and preferences, decides which channels apply, checks rate limits and quiet hours, renders the templates and enqueues one delivery per channel and device.
3. Channel queues: separate queues per channel and per priority. Separate queues mean a backlog of marketing email can never delay an SMS login code.
4. Channel workers: pull deliveries, call the provider (APNs and FCM for push, an email provider, an SMS provider), and record the result.
5. Tracking: provider callbacks (delivered, bounced, complained) and in-app opens update each delivery's status.

Splitting routing from sending lets each scale and fail independently. If the SMS provider is down, SMS deliveries wait in their queue while push and email continue.

#Preferences, quiet hours and rate limits

Preferences live in a table keyed by user, cached aggressively because every notification reads them:

user_preferences(user_id, category, channel, enabled)
user_settings(user_id, timezone, quiet_start, quiet_end, locale)

Rules the router applies, in order:

• Category rules first: security and legally required messages ignore marketing opt-outs but still respect "don't SMS me" if there's an alternative channel.
• Quiet hours: non-urgent notifications during the user's night are delayed to the end of quiet hours (scheduled), not dropped. Urgent ones go now.
• Per-user rate limits: for example at most 5 social notifications per hour, with the rest folded into a digest. This also contains bugs: a producer stuck in a loop can't send one user 1,000 messages (rate-limiting algorithms).
• Collapse duplicates: "Ana liked your photo" five times in a minute becomes "Ana and 4 others liked your photo". Push providers support a collapse key so a newer notification replaces an older one on the device.

#Talking to providers

Each external provider has rate limits, error codes and failure modes of its own, and workers must handle them carefully:

• Retries with backoff and jitter on temporary failures (timeouts, 429, 5xx). Respect Retry-After headers.
• Permanent failures stop retries: an invalid email address, or a push token the provider reports as unregistered (APNs returns 410, FCM returns UNREGISTERED). Mark the address or token invalid so you stop sending to it; repeatedly sending to dead addresses damages your sender reputation with email providers.
• Idempotent sends: a worker can crash after the provider accepted a message but before recording it, so the delivery will be retried. Give each delivery a stable ID, pass it as the provider's idempotency key or message ID where supported, and check the delivery's status before sending again (idempotent consumers).
• Failover: for SMS and email, contract a second provider. A circuit breaker on the primary switches traffic to the secondary when errors spike (circuit breakers). Push has no alternative to APNs and FCM, so push simply queues until they recover.
• Dead letters: deliveries that exhaust their retries go to a dead-letter queue and an alert, not into silence.

#Campaigns: fan-out without starving everything else

A 50-million-recipient campaign can't be one API call that fans out synchronously. Treat it as a job:

1. The campaign service stores the audience definition (a segment query) and schedules the campaign.
2. At send time, a batch job pages through the audience with a cursor (keyset pagination) and enqueues notifications in chunks of, say, 1,000 users, onto low-priority queues.
3. The send rate is throttled to what providers allow and what your own systems handle: clicks from 50 million emails arriving in ten minutes would overload the website the campaign links to.
4. Progress is checkpointed, so a crash resumes from the last chunk instead of restarting and sending duplicates.

Transactional traffic keeps flowing because it sits on separate high-priority queues with reserved worker capacity.

#Storage and tracking

Two kinds of data, with very different shapes:

• Notification and delivery records (who, what, which channel, status, timestamps): written once, updated a few times, queried by user ("show my inbox") and by ID. At around a billion status events a day, store them in a wide-column or time-partitioned store keyed by user_id and time, and expire them after 90 days.
• The in-app inbox is the same data viewed per user: the latest 100 notifications by user_id, newest first, with read/unread state. Cache the first page per user.

Aggregate metrics (delivery rate per provider, open rate per type) come from streaming the status events into an analytics store. Alert on drops in delivery rate per provider, which is often the first sign of a provider problem or a spam-filter block.

#Practice

Two incidents happen in the same week:

1. A bug in the social service calls the notify API in a loop, sending one user's 2,000 followers the same "new post" notification repeatedly.
2. During a 20-million-user email campaign, the primary email provider starts timing out on 40% of requests. Login codes by email are delayed by 15 minutes.

For each, explain which parts of the design should have prevented or contained it, and what you'd change.

#Key takeaways

• Producers send an intent (type, user, data) with an idempotency key; the notification service owns channels, preferences and templates.
• Split the pipeline into API, router and per-channel, per-priority queues and workers, so slow channels and bulk sends can't delay urgent messages.
• Apply category rules, quiet hours, per-user rate limits and collapsing to protect users from noise and from bugs.
• Treat providers as unreliable: retries with backoff, permanent-failure handling, idempotent sends, failover and dead-letter queues.
• Run campaigns as throttled, checkpointed jobs on low-priority queues with isolated capacity.