# Queues & event streams

> Move work off the request path with queues and logs, understand at-least-once delivery, make consumers idempotent, keep order where it matters, and publish events reliably with the outbox pattern.

System Design roadmap · Communication & reliability · 21 min read · https://levelcode.org/roadmaps/system-design/queues-and-streams

A message queue lets one part of a system hand work to another without waiting for it. The web server enqueues "resize this image" and responds immediately; a worker picks the job up when it can. That decoupling absorbs bursts, isolates failures and lets each side scale on its own. It also introduces the hardest bugs in backend engineering: messages delivered twice, delivered out of order, or quietly never sent. This lesson covers the two kinds of messaging system, what their delivery guarantees really mean, and the patterns that make asynchronous work correct.

## Why go asynchronous

Put work on a queue when it:

- **doesn't need to finish before the response** — emails, thumbnails, search indexing, webhooks to partners;
- **is slow or unreliable** — a third-party API that takes 3 seconds or fails 1% of the time;
- **arrives in bursts** — 50,000 sign-ups after a TV ad become a backlog workers drain at a steady rate;
- **fans out** — one "order placed" event that billing, shipping, analytics and email each react to.

The cost is that the caller no longer knows when — or whether — the work happened. You need monitoring of queue depth and consumer lag, a way to handle messages that keep failing, and a UI that tolerates "processing…".

## Queues versus logs

Two designs go by the name "message broker", and they behave differently. See [message queues vs event logs](https://levelcode.org/concepts/queue-vs-log).

**Message queues** — RabbitMQ, Amazon SQS, Google Cloud Tasks. A message is delivered to **one** of the consumers, which processes it and **acknowledges** it; acknowledged messages are deleted. If a consumer dies before acknowledging, the message becomes visible again after a timeout (SQS's *visibility timeout*) and another consumer gets it. Add consumers to go faster. Ideal for **jobs**: units of work where each must be done once by someone.

**Event logs** — Apache Kafka, Amazon Kinesis, Redpanda, Pulsar. Messages are appended to a **partitioned, ordered log** and kept for a retention period (days, or forever), whether or not anyone has read them. Each **consumer group** tracks its own position (offset) in each partition. Any number of groups can read the same events independently, and a group can rewind to reprocess history. Ideal for **events**: facts that several systems react to, or that you want to replay.

| | Queue | Log |
|---|---|---|
| After processing | Message deleted | Message kept until retention expires |
| Several independent consumers | Needs a copy per consumer (fan-out exchange, SNS → SQS) | Built in: one consumer group each |
| Ordering | None, or per group in FIFO queues | Per partition |
| Replay | No | Yes, by resetting offsets |
| Scaling consumers | Add consumers freely | At most one consumer per partition per consumer group |

## Delivery guarantees

Every system offers one of three guarantees, and the choice comes down to **when the consumer acknowledges**. See [delivery guarantees](https://levelcode.org/concepts/delivery-semantics).

- **At most once:** acknowledge, then process. A crash mid-processing loses the message. Acceptable only for data you can lose, like metrics samples.
- **At least once:** process, then acknowledge. A crash after processing but before the acknowledgement means the message is redelivered and processed **again**. This is the default everywhere, and the one you should design for.
- **Exactly once:** not something a broker can give you end to end. A broker can't know whether your side effect — the database write, the email, the charge — happened before the crash. What you can build is **effectively once**: at-least-once delivery plus processing that's safe to repeat. (Kafka's transactions give exactly-once *within Kafka*, from one topic to another, but not to your database or an external API.)

So the rule is simple: **assume every message will arrive twice, and make that harmless.**

## Idempotent consumers

A consumer is idempotent if processing the same message twice has the same effect as processing it once. See [idempotent consumers](https://levelcode.org/concepts/idempotent-consumers). Some operations are naturally idempotent: "set the order status to shipped" can run twice. Others aren't: "add 10 credits", "send an email", "charge the card".

For those, record each processed message's ID **in the same database transaction as its effect**:

```sql
CREATE TABLE processed_messages (
  message_id   text PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);
```

```go
func (h *Handler) Handle(ctx context.Context, msg Message) error {
	tx, err := h.db.Begin(ctx)
	if err != nil {
		return err
	}
	defer tx.Rollback(ctx) // no-op after Commit

	// Claim the message. If the row already exists, we've done this work before.
	tag, err := tx.Exec(ctx,
		`INSERT INTO processed_messages (message_id) VALUES ($1) ON CONFLICT DO NOTHING`, msg.ID)
	if err != nil {
		return err
	}
	if tag.RowsAffected() == 0 {
		return nil // duplicate: acknowledge and move on
	}

	if _, err := tx.Exec(ctx,
		`UPDATE accounts SET credits = credits + $1 WHERE id = $2`, msg.Credits, msg.AccountID); err != nil {
		return err
	}
	return tx.Commit(ctx) // effect and dedup record commit together, or not at all
}
```

The caller acknowledges the message only after `Handle` returns `nil`. If the process dies before the commit, nothing happened and the redelivery does the work. If it dies after the commit but before the acknowledgement, the redelivery hits the existing row and does nothing.

For side effects outside your database — an email provider, a payment processor — pass a stable **idempotency key** derived from the message (see [idempotency keys](https://levelcode.org/concepts/idempotency-keys)), so the external service deduplicates for you.

## Ordering

Queues generally don't preserve order, and logs preserve it **only within a partition**. That's usually enough, if you choose the partition key well: send all events for one entity to the same partition by keying on its ID.

```text
topic "orders", 12 partitions, key = order_id
  order 981: created → paid → shipped   all in partition 7, consumed in order
  order 982: created → cancelled        all in partition 3
```

Events for different orders may be processed in any relative order, which is fine because they're independent. The trade-off is parallelism: a consumer group can use at most as many consumers as there are partitions, and one hot key can't be spread out. SQS FIFO queues offer the same idea with a *message group ID*.

Kafka 4.2 made **share groups** ("queues for Kafka") production-ready: several consumers share a partition, acknowledge records one by one, and a record that isn't acknowledged in time is redelivered, much like a queue. That lifts the one-consumer-per-partition limit, but gives up per-partition order, so use share groups for independent jobs and ordinary consumer groups for events whose order matters.

Even with per-key ordering, a retry can reorder things: message 1 fails, message 2 succeeds, message 1 succeeds on retry. If order matters, either block the partition until the failing message succeeds or is parked, or make consumers tolerate reordering by including a version number in each event and ignoring anything older than what you've already applied.

## When messages keep failing

Some messages fail every time: malformed data, a bug, a reference to a deleted record. Retried forever, a **poison message** blocks its partition or burns worker time indefinitely.

- **Retry with backoff**, a limited number of times (see [retries, backoff and jitter](https://levelcode.org/concepts/retries-with-backoff)).
- After the last attempt, move it to a **dead-letter queue** with the error attached, alert on it, and keep going. Fix the cause, then replay the dead letters. See [dead-letter queues](https://levelcode.org/concepts/dead-letter-queues).
- Distinguish **transient** errors (timeouts, 503s: retry) from **permanent** ones (validation failures: dead-letter immediately).

And watch **consumer lag** — how far behind the newest message a consumer group is. Rising lag means consumers can't keep up; it's the queue equivalent of CPU at 100%, and the earliest warning you get. See [backpressure](https://levelcode.org/concepts/backpressure).

## Publishing reliably: the transactional outbox

The most common messaging bug isn't in the consumer. It's in the producer:

```go
// Broken: two systems, no atomicity.
tx.Exec(ctx, `INSERT INTO orders …`)
tx.Commit(ctx)
broker.Publish(ctx, "order.created", order) // crash here → order exists, event never sent
```

Publish first and commit second, and a failed commit leaves an event for an order that doesn't exist. There's no ordering of a database write and a broker publish that's safe, because they can't share a transaction. The fix is the **transactional outbox**: write the event to a table **in the same transaction** as the change, and have a separate relay publish it. See [transactional outbox](https://levelcode.org/concepts/transactional-outbox).

```sql
CREATE TABLE outbox (
  id           bigserial PRIMARY KEY,
  topic        text NOT NULL,
  key          text NOT NULL,
  payload      jsonb NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz
);

BEGIN;
INSERT INTO orders (id, customer_id, total) VALUES ('ord_981', 'cus_7', 4200);
INSERT INTO outbox (topic, key, payload)
VALUES ('orders', 'ord_981', '{"type":"order.created","order_id":"ord_981","total":4200}');
COMMIT;
```

The relay loops, claiming a batch with `FOR UPDATE SKIP LOCKED` so several relays can run without double-claiming rows:

```sql
BEGIN;
SELECT id, topic, key, payload FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;
-- publish each row to the broker and wait for its acknowledgement, then:
UPDATE outbox SET published_at = now() WHERE id = ANY($1);
COMMIT;
```

If the relay crashes after publishing but before the update, it publishes those events again — at least once, which your idempotent consumers already handle. Instead of polling, you can feed the outbox (or the tables themselves) into Kafka with change data capture, using a tool like Debezium that reads the database's write-ahead log.

```sequence
title: Order placed, with an outbox
participants: API, Database, Relay, Broker, Email worker
API -> Database: INSERT order + outbox row (one tx)
Database --> API: committed
API -> API: respond 201
Relay -> Database: claim unpublished outbox rows
Relay -> Broker: publish order.created (key = order id)
Broker --> Relay: ack
Relay -> Database: mark rows published
Broker -> Email worker: order.created
Email worker -> Email worker: dedupe by event id, send email
```

## Practice

A SaaS app sends a webhook to customers' servers whenever an invoice is paid. Today the payment handler commits the invoice update, then calls each customer's webhook URL inline with a 10-second timeout. Problems: slow customer endpoints make payments slow; when the app crashes after the commit, webhooks are never sent; and when a customer's server is down for an hour, its webhooks are lost.

Redesign it with a queue or log. Cover how events are produced, delivered, retried and deduplicated, and what the customer must do.

**Solution**

**Produce reliably.** In the same transaction that marks the invoice paid, insert an `outbox` row: `{type: "invoice.paid", event_id: <uuid>, invoice_id, customer_id, paid_at}`. Commit. The payment request returns without waiting for any webhook. A relay (or CDC) publishes outbox rows to a topic keyed by `customer_id`, so each customer's events stay in order.

**Deliver asynchronously.** Webhook workers consume the topic and POST each event to the customer's URL with a short timeout (5 seconds) and an HMAC signature header so the customer can verify the sender. A slow customer no longer affects payments; per-customer concurrency limits stop one slow endpoint from tying up every worker (a bulkhead).

**Retry and park.** On a timeout or 5xx, schedule a retry with exponential backoff and jitter — for example after 1 minute, 5, 30, 2 hours, 12 hours — for up to 24 hours. Record each attempt. After the final attempt, move the event to a dead-letter store, mark the endpoint unhealthy, notify the customer, and offer a "resend" button. On a 4xx other than 429, stop retrying: the request itself is wrong.

**Deduplicate.** Delivery is at least once: the relay can republish, and a worker can crash after the customer received the POST but before recording success. Include `event_id` in every delivery and tell customers in the documentation to store processed IDs and ignore repeats. On your side, record successful deliveries by `event_id` so workers skip events already delivered.

**What the customer does:** verify the signature, respond 2xx quickly (and do their own processing asynchronously), and deduplicate by `event_id`.

## Key takeaways

- Use queues for jobs done once by some worker; use logs for events many consumers read, in order per partition, with replay.
- Delivery is at least once in practice. Exactly-once effects come from idempotent processing, not from the broker.
- Make consumers idempotent by recording message IDs in the same transaction as the effect, or by passing idempotency keys to external services.
- Order holds only per partition or message group: key by entity ID, and handle retries that reorder.
- Retry transient failures with backoff, dead-letter poison messages, and alert on consumer lag.
- Never dual-write to a database and a broker; use a transactional outbox or change data capture.

## Quiz

1. A consumer processes a message, commits its database change, and crashes before acknowledging. What happens next with at-least-once delivery?

   A. The message is lost
   B. The broker detects the commit and discards the message
   C. The message is redelivered and processed again unless the consumer detects the duplicate
   D. The broker rolls back the database change

   **Answer: C.** The broker never saw an acknowledgement, so it redelivers. Recording the message ID in the same transaction as the effect lets the second attempt see it's already done.

2. Why can't a service safely write to its database and then publish to Kafka in the same request handler?

   A. Kafka is too slow for request handlers
   B. The two writes can't share a transaction, so a crash between them leaves the database and the event stream disagreeing
   C. Kafka requires events to be published before the database write
   D. Databases block while Kafka is publishing

   **Answer: B.** Whichever order you choose, a failure between the steps loses or invents an event. The outbox puts the event in the same transaction as the change, and a relay publishes it at least once.

3. Events for each order must be processed in order, at high volume. What do you do in Kafka?

   A. Use one partition for the whole topic
   B. Key each event by order ID so all events for an order go to the same partition
   C. Add more consumers than partitions
   D. Rely on timestamps to sort events in the consumer

   **Answer: B.** Kafka orders messages within a partition. Keying by order ID keeps each order's events together while different orders spread across partitions for parallelism. One partition would serialise everything.

4. A message fails on every attempt because its payload references a deleted customer. What should the consumer do?

   A. Retry forever until the customer reappears
   B. Acknowledge it silently and drop it
   C. After a few attempts, or immediately for a permanent error, move it to a dead-letter queue and alert
   D. Restart the consumer process

   **Answer: C.** Retrying a poison message forever blocks or wastes capacity; dropping it silently loses information. A dead-letter queue keeps it for diagnosis and replay while the rest of the stream keeps flowing.