Queues & event streams

A message queue lets one part of a system hand work to another without waiting for it. The web server enqueues "resize this image" and responds immediately; a worker picks the job up when it can. That decoupling absorbs bursts, isolates failures and lets each side scale on its own. It also introduces the hardest bugs in backend engineering: messages delivered twice, delivered out of order, or quietly never sent. This lesson covers the two kinds of messaging system, what their delivery guarantees really mean, and the patterns that make asynchronous work correct.

#Why go asynchronous

Put work on a queue when it:

• doesn't need to finish before the response — emails, thumbnails, search indexing, webhooks to partners;
• is slow or unreliable — a third-party API that takes 3 seconds or fails 1% of the time;
• arrives in bursts — 50,000 sign-ups after a TV ad become a backlog workers drain at a steady rate;
• fans out — one "order placed" event that billing, shipping, analytics and email each react to.

The cost is that the caller no longer knows when — or whether — the work happened. You need monitoring of queue depth and consumer lag, a way to handle messages that keep failing, and a UI that tolerates "processing…".

#Queues versus logs

Two designs go by the name "message broker", and they behave differently. See message queues vs event logs.

Message queues — RabbitMQ, Amazon SQS, Google Cloud Tasks. A message is delivered to one of the consumers, which processes it and acknowledges it; acknowledged messages are deleted. If a consumer dies before acknowledging, the message becomes visible again after a timeout (SQS's visibility timeout) and another consumer gets it. Add consumers to go faster. Ideal for jobs: units of work where each must be done once by someone.

Event logs — Apache Kafka, Amazon Kinesis, Redpanda, Pulsar. Messages are appended to a partitioned, ordered log and kept for a retention period (days, or forever), whether or not anyone has read them. Each consumer group tracks its own position (offset) in each partition. Any number of groups can read the same events independently, and a group can rewind to reprocess history. Ideal for events: facts that several systems react to, or that you want to replay.

#Delivery guarantees

Every system offers one of three guarantees, and the choice comes down to when the consumer acknowledges. See delivery guarantees.

• At most once: acknowledge, then process. A crash mid-processing loses the message. Acceptable only for data you can lose, like metrics samples.
• At least once: process, then acknowledge. A crash after processing but before the acknowledgement means the message is redelivered and processed again. This is the default everywhere, and the one you should design for.
• Exactly once: not something a broker can give you end to end. A broker can't know whether your side effect — the database write, the email, the charge — happened before the crash. What you can build is effectively once: at-least-once delivery plus processing that's safe to repeat. (Kafka's transactions give exactly-once within Kafka, from one topic to another, but not to your database or an external API.)

So the rule is simple: assume every message will arrive twice, and make that harmless.

#Idempotent consumers

A consumer is idempotent if processing the same message twice has the same effect as processing it once. See idempotent consumers. Some operations are naturally idempotent: "set the order status to shipped" can run twice. Others aren't: "add 10 credits", "send an email", "charge the card".

For those, record each processed message's ID in the same database transaction as its effect:

CREATE TABLE processed_messages (
  message_id   text PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);

func (h *Handler) Handle(ctx context.Context, msg Message) error {
	tx, err := h.db.Begin(ctx)
	if err != nil {
		return err
	}
	defer tx.Rollback(ctx) // no-op after Commit

	// Claim the message. If the row already exists, we've done this work before.
	tag, err := tx.Exec(ctx,
		`INSERT INTO processed_messages (message_id) VALUES ($1) ON CONFLICT DO NOTHING`, msg.ID)
	if err != nil {
		return err
	}
	if tag.RowsAffected() == 0 {
		return nil // duplicate: acknowledge and move on
	}

	if _, err := tx.Exec(ctx,
		`UPDATE accounts SET credits = credits + $1 WHERE id = $2`, msg.Credits, msg.AccountID); err != nil {
		return err
	}
	return tx.Commit(ctx) // effect and dedup record commit together, or not at all
}

The caller acknowledges the message only after Handle returns nil. If the process dies before the commit, nothing happened and the redelivery does the work. If it dies after the commit but before the acknowledgement, the redelivery hits the existing row and does nothing.

For side effects outside your database — an email provider, a payment processor — pass a stable idempotency key derived from the message (see idempotency keys), so the external service deduplicates for you.

#Ordering

Queues generally don't preserve order, and logs preserve it only within a partition. That's usually enough, if you choose the partition key well: send all events for one entity to the same partition by keying on its ID.

topic "orders", 12 partitions, key = order_id
  order 981: created → paid → shipped   all in partition 7, consumed in order
  order 982: created → cancelled        all in partition 3

Events for different orders may be processed in any relative order, which is fine because they're independent. The trade-off is parallelism: a consumer group can use at most as many consumers as there are partitions, and one hot key can't be spread out. SQS FIFO queues offer the same idea with a message group ID.

Kafka 4.2 made share groups ("queues for Kafka") production-ready: several consumers share a partition, acknowledge records one by one, and a record that isn't acknowledged in time is redelivered, much like a queue. That lifts the one-consumer-per-partition limit, but gives up per-partition order, so use share groups for independent jobs and ordinary consumer groups for events whose order matters.

Even with per-key ordering, a retry can reorder things: message 1 fails, message 2 succeeds, message 1 succeeds on retry. If order matters, either block the partition until the failing message succeeds or is parked, or make consumers tolerate reordering by including a version number in each event and ignoring anything older than what you've already applied.

#When messages keep failing

Some messages fail every time: malformed data, a bug, a reference to a deleted record. Retried forever, a poison message blocks its partition or burns worker time indefinitely.

• Retry with backoff, a limited number of times (see retries, backoff and jitter).
• After the last attempt, move it to a dead-letter queue with the error attached, alert on it, and keep going. Fix the cause, then replay the dead letters. See dead-letter queues.
• Distinguish transient errors (timeouts, 503s: retry) from permanent ones (validation failures: dead-letter immediately).

And watch consumer lag — how far behind the newest message a consumer group is. Rising lag means consumers can't keep up; it's the queue equivalent of CPU at 100%, and the earliest warning you get. See backpressure.

#Publishing reliably: the transactional outbox

The most common messaging bug isn't in the consumer. It's in the producer:

// Broken: two systems, no atomicity.
tx.Exec(ctx, `INSERT INTO orders …`)
tx.Commit(ctx)
broker.Publish(ctx, "order.created", order) // crash here → order exists, event never sent

Publish first and commit second, and a failed commit leaves an event for an order that doesn't exist. There's no ordering of a database write and a broker publish that's safe, because they can't share a transaction. The fix is the transactional outbox: write the event to a table in the same transaction as the change, and have a separate relay publish it. See transactional outbox.

CREATE TABLE outbox (
  id           bigserial PRIMARY KEY,
  topic        text NOT NULL,
  key          text NOT NULL,
  payload      jsonb NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz
);

BEGIN;
INSERT INTO orders (id, customer_id, total) VALUES ('ord_981', 'cus_7', 4200);
INSERT INTO outbox (topic, key, payload)
VALUES ('orders', 'ord_981', '{"type":"order.created","order_id":"ord_981","total":4200}');
COMMIT;

The relay loops, claiming a batch with FOR UPDATE SKIP LOCKED so several relays can run without double-claiming rows:

BEGIN;
SELECT id, topic, key, payload FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;
-- publish each row to the broker and wait for its acknowledgement, then:
UPDATE outbox SET published_at = now() WHERE id = ANY($1);
COMMIT;

If the relay crashes after publishing but before the update, it publishes those events again — at least once, which your idempotent consumers already handle. Instead of polling, you can feed the outbox (or the tables themselves) into Kafka with change data capture, using a tool like Debezium that reads the database's write-ahead log.

#Practice

A SaaS app sends a webhook to customers' servers whenever an invoice is paid. Today the payment handler commits the invoice update, then calls each customer's webhook URL inline with a 10-second timeout. Problems: slow customer endpoints make payments slow; when the app crashes after the commit, webhooks are never sent; and when a customer's server is down for an hour, its webhooks are lost.

Redesign it with a queue or log. Cover how events are produced, delivered, retried and deduplicated, and what the customer must do.

#Key takeaways

• Use queues for jobs done once by some worker; use logs for events many consumers read, in order per partition, with replay.
• Delivery is at least once in practice. Exactly-once effects come from idempotent processing, not from the broker.
• Make consumers idempotent by recording message IDs in the same transaction as the effect, or by passing idempotency keys to external services.
• Order holds only per partition or message group: key by entity ID, and handle retries that reorder.
• Retry transient failures with backoff, dead-letter poison messages, and alert on consumer lag.
• Never dual-write to a database and a broker; use a transactional outbox or change data capture.