Design a chat system

Chat — WhatsApp, Slack, Discord, Messenger — feels simple to use and is demanding to build. Messages must arrive in well under a second, never be lost, appear in the same order for everyone in a conversation, and sync across a user's phone and laptop, while tens of millions of devices hold open connections at once. This lesson designs a chat backend, focusing on the parts that are specific to messaging: long-lived connections, ordering, delivery and sync.

#Requirements

Functional:

• One-to-one and group conversations (up to 500 members).
• Text messages and attachments (images, files).
• Delivery state per message: sent, delivered, read.
• Each user can have several devices; every device shows the full history.
• Presence ("online", "last seen") and push notifications for offline devices.
• Out of scope: voice and video calls, end-to-end encryption (which changes the server's role to relaying ciphertext, but not the delivery design).

Non-functional:

• p99 delivery under 500 ms between online users in the same region.
• No message loss once the server has acknowledged it.
• Consistent order within a conversation, the same on every device.
• High availability: users should be able to send even during partial failures.

#Numbers

daily active users:        100 million
messages:                  40 per user per day → 4 billion/day ≈ 46,000/s avg, ~150,000/s peak
concurrent connections:    ~30% of DAU online at peak → 30 million WebSockets
message size:              ~200 bytes of text and metadata → 800 GB/day, ~300 TB/year

Two numbers shape the design. 30 million concurrent connections means connection handling is its own tier. And 300 TB a year of append-mostly, time-ordered data read per conversation points to a wide-column store rather than a single relational database.

#Connections: keeping a line open

Chat needs the server to push messages to clients the moment they arrive. Polling every few seconds is too slow and wasteful at this scale, so clients hold a WebSocket open (with server-sent events or long polling as fallbacks). See polling, SSE and WebSockets.

That makes the edge of the system stateful: a dedicated tier of gateway servers terminates WebSockets. Each holds tens or hundreds of thousands of connections (an idle connection costs memory for buffers and TLS state, roughly 10–50 KB), so 30 million connections need a few hundred gateways. Gateways do as little as possible: authenticate the connection, pass messages to the chat service, and deliver messages back to devices.

To deliver a message to a device, the system must know which gateway holds its connection. A connection registry maps device_id → gateway_id, stored in Redis with a short TTL that each gateway refreshes with heartbeats. If a gateway dies, its entries expire and its clients reconnect elsewhere.

#Sending a message

Every message carries a client-generated ID so retries are safe: if the sender's connection drops before it hears the acknowledgement, it resends with the same ID and the server deduplicates (idempotency keys).

The order matters: the server acknowledges to the sender only after the message is durably stored. That's what "no loss once acknowledged" means. Delivery to recipients happens after, asynchronously; a recipient that's offline will get the message on its next sync.

For devices with no open connection, the chat service hands the message to a push notification service, which sends through Apple Push Notification service or Firebase Cloud Messaging. Push is best-effort and rate-limited by the platforms, so it's a wake-up signal ("you have a new message"), never the delivery mechanism of record.

#Ordering within a conversation

Clocks on phones are unreliable, and two servers' clocks differ (see clocks and ordering), so timestamps can't order messages. Instead, give each conversation a sequence number assigned by one place:

• Partition conversations by conversation_id across chat-service instances, so each conversation has exactly one owner at a time (via consistent hashing or a shard map, with a lease to prevent two owners — see leases and fencing tokens).
• The owner assigns seq = last_seq + 1 as it stores each message. Every device sorts by seq, so everyone sees the same order.
• A device that receives seq 1045 while its latest is 1042 knows it missed two messages and fetches them.

Ordering is per conversation only. Messages in different conversations have no meaningful relative order, which is what lets the system scale: conversations are independent.

#Storing messages

The dominant query is "the latest N messages in conversation C, and those before sequence S" — a range scan within one conversation. That's an ideal fit for a wide-column store such as Cassandra or ScyllaDB (Discord moved its message storage from Cassandra to ScyllaDB at trillions of messages):

-- CQL. One partition per conversation per bucket keeps partitions bounded in size.
CREATE TABLE messages (
  conversation_id bigint,
  bucket          int,          -- e.g. a 10-day window, so huge conversations don't make one giant partition
  seq             bigint,
  message_id      text,         -- the client-generated ID, for deduplication
  sender_id       bigint,
  body            text,
  attachment_key  text,
  created_at      timestamp,
  PRIMARY KEY ((conversation_id, bucket), seq)
) WITH CLUSTERING ORDER BY (seq DESC);

Reading the latest page is one partition read, sorted. Attachments go to object storage — the client uploads with a presigned URL and sends a message containing only the object key — so the message path stays small.

Per-user state lives separately: each user's conversation list (with the latest message and unread count for the sidebar), and per-user, per-conversation read pointers (last_read_seq).

#Multi-device sync and receipts

Every device keeps, per conversation, the highest seq it has. Sync is then simple and robust:

• On connect, the device sends its cursors; the server returns everything newer, a page at a time. A device that was offline for a week catches up the same way as one that missed a single message.
• Live messages arrive over the WebSocket; if one is missed, the gap in seq triggers a fetch.
• Delivery is effectively at least once (a message can arrive both live and via sync), and devices deduplicate by seq.

Receipts are just more state:

• Sent: the sender received the server's acknowledgement.
• Delivered: a recipient's device acknowledged the message. Report it to the sender as a small event.
• Read: the recipient's read pointer moved past the message. Store last_read_seq per user per conversation, and send it to the other members' devices — in large groups, as an aggregate ("read by 12") rather than one event per reader.

#Group fan-out and presence

For a group, the chat service stores each message once and then delivers it to every member's online devices through their gateways. With up to 500 members, fan-out on write is fine: up to a few thousand deliveries per message. Very large rooms (Discord servers or Slack channels with hundreds of thousands of members) switch to the pull model of the news-feed lesson: clients viewing the channel subscribe to it, and others fetch on open.

Presence is high-volume and low-value per event, so keep it cheap and eventually consistent:

• Devices send heartbeats every 30 seconds; a device is online while its heartbeat key in Redis hasn't expired.
• Don't broadcast every change to every contact. Clients subscribe to presence only for contacts currently on screen, and changes are batched.
• "Last seen" is written at most every few minutes.

#When things fail

#Practice

Design two features on top of this system:

1. Typing indicators ("Bob is typing…") in one-to-one and group chats.
2. Delete for everyone: a sender can delete a message up to an hour after sending, and it must disappear from every member's devices, including devices that are offline right now.

For each, say what's stored, what's sent over the connection, and what consistency is acceptable.

#Key takeaways

• Push delivery needs persistent connections; run a dedicated gateway tier and a connection registry that maps devices to gateways.
• Acknowledge a send only after the message is durably stored; deduplicate retries with client-generated IDs.
• Order messages with a per-conversation sequence number from a single owner, not timestamps; gaps tell clients what they missed.
• Store messages partitioned by conversation (and time bucket), clustered by sequence, in a wide-column store; attachments go to object storage.
• Multi-device sync is cursors per conversation; receipts are read pointers and acknowledgements.
• Treat push notifications as wake-ups, keep presence cheap and eventually consistent, and have clients reconnect with jitter.