# Design a notification system > Design a service that sends push, email, SMS and in-app notifications reliably: preferences, priorities, rate limits, retries, provider failover and tracking. System Design roadmap · Case studies · 19 min read · https://levelcode.org/roadmaps/system-design/design-notification-system Nearly every product sends notifications: "your order shipped", "your login code is 482913", "Ana commented on your post", "20% off this weekend". Teams start by calling an email API from wherever the event happens, and end up with duplicate messages, users who can't unsubscribe, login codes stuck behind a marketing campaign, and nobody able to say whether a message was delivered. This lesson designs a central notification service that every other service uses. ## Requirements **Functional:** - Channels: mobile push (iOS and Android), email, SMS and an in-app inbox. - Other services send a notification by *type* and *recipient*, not by channel; the system decides how to deliver it from the user's preferences. - Users can opt out per type and channel, and set quiet hours. Legally required messages (security alerts, receipts) bypass marketing opt-outs. - Templates with localisation and variables. - Scheduled sends and bulk campaigns to millions of users. - Track each message's status: queued, sent, delivered, opened, failed. **Non-functional:** - **Priority**: a login code must arrive within seconds even while a 10-million-user campaign is going out. - **At-least-once delivery without visible duplicates**: never lose a transactional message, and never send the same one twice. - Survive the failure of any single external provider. - Protect users from spam caused by bugs: no user should receive 50 notifications in a minute. ## Numbers ```text users: 50 million transactional: ~5 per user per day → 250 million/day ≈ 3,000/s avg, ~10,000/s peak campaigns: a few per week, up to 50 million recipients each campaign send rate: 50 million over 1 hour → ~14,000/s on top of normal traffic status events: ~3 per message (sent, delivered, opened) → ~1 billion/day ``` The throughput itself is modest. The difficulty is in the bursts (campaigns), the external providers (each with its own rate limits and outages), and the rules (preferences, priorities, de-duplication). ## The API: intent, not channel Producers describe *what happened*, not how to deliver it: ```http POST /v1/notifications Idempotency-Key: order-8812-shipped { "type": "order_shipped", "user_id": "u_123", "data": {"order_id": "8812", "carrier": "DHL", "eta": "2026-11-03"} } 202 Accepted {"notification_id": "n_7f3a", "status": "queued"} ``` The **idempotency key** matters here more than almost anywhere else. The order service will retry when the call times out; without the key, the user gets two "shipped" emails. The notification service stores keys for a few days and returns the original response on a repeat ([idempotency keys](https://levelcode.org/concepts/idempotency-keys)). Each notification type is registered once, with its category (security, transactional, social, marketing), priority, default channels and templates. That registry is what lets the service apply rules consistently, rather than each producer reinventing them. ## The pipeline ```sequence title: An order-shipped notification participants: Order service, Notify API, Router, Push queue, Push worker, APNs Order service -> Notify API: POST /notifications (idempotency key) Notify API -> Notify API: Store request, dedupe on key Notify API --> Order service: 202 Accepted Notify API -> Router: Notification n_7f3a Router -> Router: Load prefs, quiet hours, rate limits, render template > The router decides channels: here push and in-app, not email. Router -> Push queue: Enqueue per device (high priority) Push worker -> APNs: Send to device token APNs --> Push worker: 200 OK Push worker -> Push worker: Record status "sent" ``` The stages: 1. **API**: authenticates the producer, validates the type and data, de-duplicates, writes the request durably and returns 202. Nothing slow happens here. 2. **Router**: loads the user's contact details and preferences, decides which channels apply, checks rate limits and quiet hours, renders the templates and enqueues one **delivery** per channel and device. 3. **Channel queues**: separate queues per channel and per **priority**. Separate queues mean a backlog of marketing email can never delay an SMS login code. 4. **Channel workers**: pull deliveries, call the provider (APNs and FCM for push, an email provider, an SMS provider), and record the result. 5. **Tracking**: provider callbacks (delivered, bounced, complained) and in-app opens update each delivery's status. Splitting routing from sending lets each scale and fail independently. If the SMS provider is down, SMS deliveries wait in their queue while push and email continue. ## Preferences, quiet hours and rate limits Preferences live in a table keyed by user, cached aggressively because every notification reads them: ```text user_preferences(user_id, category, channel, enabled) user_settings(user_id, timezone, quiet_start, quiet_end, locale) ``` Rules the router applies, in order: - **Category rules first**: security and legally required messages ignore marketing opt-outs but still respect "don't SMS me" if there's an alternative channel. - **Quiet hours**: non-urgent notifications during the user's night are *delayed* to the end of quiet hours (scheduled), not dropped. Urgent ones go now. - **Per-user rate limits**: for example at most 5 social notifications per hour, with the rest folded into a digest. This also contains bugs: a producer stuck in a loop can't send one user 1,000 messages ([rate-limiting algorithms](https://levelcode.org/concepts/rate-limiting-algorithms)). - **Collapse duplicates**: "Ana liked your photo" five times in a minute becomes "Ana and 4 others liked your photo". Push providers support a collapse key so a newer notification replaces an older one on the device. ## Talking to providers Each external provider has rate limits, error codes and failure modes of its own, and workers must handle them carefully: - **Retries with backoff and jitter** on temporary failures (timeouts, 429, 5xx). Respect `Retry-After` headers. - **Permanent failures stop retries**: an invalid email address, or a push token the provider reports as unregistered (APNs returns 410, FCM returns `UNREGISTERED`). Mark the address or token invalid so you stop sending to it; repeatedly sending to dead addresses damages your sender reputation with email providers. - **Idempotent sends**: a worker can crash after the provider accepted a message but before recording it, so the delivery will be retried. Give each delivery a stable ID, pass it as the provider's idempotency key or message ID where supported, and check the delivery's status before sending again ([idempotent consumers](https://levelcode.org/concepts/idempotent-consumers)). - **Failover**: for SMS and email, contract a second provider. A circuit breaker on the primary switches traffic to the secondary when errors spike ([circuit breakers](https://levelcode.org/concepts/circuit-breakers)). Push has no alternative to APNs and FCM, so push simply queues until they recover. - **Dead letters**: deliveries that exhaust their retries go to a [dead-letter queue](https://levelcode.org/concepts/dead-letter-queues) and an alert, not into silence. > [!WARNING] > Never put secrets or full personal details in push payloads or SMS text beyond what's needed. Lock screens show them, and SMS travels in plaintext through carriers. "You have a new message" plus an in-app fetch is safer than the message itself. ## Campaigns: fan-out without starving everything else A 50-million-recipient campaign can't be one API call that fans out synchronously. Treat it as a **job**: 1. The campaign service stores the audience definition (a segment query) and schedules the campaign. 2. At send time, a batch job pages through the audience with a cursor ([keyset pagination](https://levelcode.org/concepts/keyset-pagination)) and enqueues notifications in chunks of, say, 1,000 users, onto **low-priority** queues. 3. The send rate is throttled to what providers allow and what your own systems handle: clicks from 50 million emails arriving in ten minutes would overload the website the campaign links to. 4. Progress is checkpointed, so a crash resumes from the last chunk instead of restarting and sending duplicates. Transactional traffic keeps flowing because it sits on separate high-priority queues with reserved worker capacity. ## Storage and tracking Two kinds of data, with very different shapes: - **Notification and delivery records** (who, what, which channel, status, timestamps): written once, updated a few times, queried by user ("show my inbox") and by ID. At around a billion status events a day, store them in a wide-column or time-partitioned store keyed by `user_id` and time, and expire them after 90 days. - **The in-app inbox** is the same data viewed per user: the latest 100 notifications by `user_id`, newest first, with read/unread state. Cache the first page per user. Aggregate metrics (delivery rate per provider, open rate per type) come from streaming the status events into an analytics store. Alert on drops in delivery rate per provider, which is often the first sign of a provider problem or a spam-filter block. ## Practice Two incidents happen in the same week: 1. A bug in the social service calls the notify API in a loop, sending one user's 2,000 followers the same "new post" notification repeatedly. 2. During a 20-million-user email campaign, the primary email provider starts timing out on 40% of requests. Login codes by email are delayed by 15 minutes. For each, explain which parts of the design should have prevented or contained it, and what you'd change. **Solution** **1. The notification loop.** - *Idempotency keys* stop retries of the same request, but a loop generating *new* requests (each with a fresh key) gets through. The containment layer is **per-user rate limits and duplicate collapsing**: the same type for the same object (`post_id`) for the same user within a window should collapse into one notification. Make the collapse key part of the type registry (`new_post:{post_id}`). - Add a **per-producer rate limit** with an alert: the social service suddenly sending 50× its normal rate should be throttled automatically and page its owners. - A **kill switch per notification type** lets on-call stop a misbehaving type in seconds without a deploy. **2. Provider degradation during a campaign.** - Login codes should never share capacity with a campaign. Check that **priority queues are truly separate** all the way down: separate queues, *and* separate worker pools and provider connection limits. If high- and low-priority workers share one HTTP client with a fixed connection pool, the campaign's slow requests occupy every connection: that's a missing [bulkhead](https://levelcode.org/concepts/bulkheads). - The **circuit breaker** on the primary provider should have opened at a 40% error rate and failed over to the secondary for transactional email. If the failover only triggers on total failure, tune it to error rate and latency. - **Pause campaigns automatically** when provider health degrades: marketing email that arrives an hour late is fine, a login code that arrives 15 minutes late is a failed login. - Add an **SLO and alert** on time-to-send for the security category (say, p99 under 10 seconds), so this pages someone in minutes rather than being reported by users. ## Key takeaways - Producers send an intent (type, user, data) with an idempotency key; the notification service owns channels, preferences and templates. - Split the pipeline into API, router and per-channel, per-priority queues and workers, so slow channels and bulk sends can't delay urgent messages. - Apply category rules, quiet hours, per-user rate limits and collapsing to protect users from noise and from bugs. - Treat providers as unreliable: retries with backoff, permanent-failure handling, idempotent sends, failover and dead-letter queues. - Run campaigns as throttled, checkpointed jobs on low-priority queues with isolated capacity. ## Quiz 1. Why should SMS login codes and marketing emails use separate queues and worker pools? A. SMS needs a different message format B. Queues can only hold one channel C. So a backlog or slow provider on bulk traffic can never delay urgent messages D. Marketing email is cheaper to send **Answer: C.** Shared queues or shared worker capacity let a 50-million-message campaign sit in front of a login code. Separate queues and pools are a bulkhead between priorities. 2. APNs reports that a device token is unregistered. What should the push worker do? A. Mark the token invalid and stop sending to it, without retrying B. Retry with exponential backoff C. Send the notification by email instead, always D. Move the delivery to the dead-letter queue and alert **Answer: A.** An unregistered token is a permanent failure: the app was uninstalled or the token rotated. Retrying wastes capacity, and continuing to send hurts your standing with the provider. 3. The order service retries a POST /notifications after a timeout. What prevents a duplicate email? A. The router's per-user rate limit B. The Idempotency-Key header, which makes the API return the original result instead of creating a second notification C. The email provider's spam filter D. Quiet hours **Answer: B.** The retry carries the same idempotency key, so the API recognises it and doesn't enqueue a second notification. Rate limits are a backstop, not the mechanism. 4. A non-urgent notification is triggered at 2 a.m. in the user's time zone, inside their quiet hours. What should happen? A. Drop it B. Send it immediately but silently C. Convert it to SMS D. Schedule it for the end of quiet hours **Answer: D.** Quiet hours delay non-urgent notifications rather than dropping them; urgent categories such as security alerts can still go out immediately.