Dead-letter queues

Some messages fail no matter how often they're retried: malformed payloads, references to records that no longer exist, or a bug in the consumer. Retried forever, such a poison message blocks its partition (in ordered logs) or wastes worker capacity (in queues).

A dead-letter queue (DLQ) is where messages go after a limited number of failed attempts.

#How it works

1. A consumer fails to process a message.
2. It's retried with backoff, up to a maximum (say 5 attempts).
3. After the last attempt, it's moved to the DLQ with metadata: the error, the attempt count, timestamps, the original topic or queue.
4. Processing of the remaining messages continues.

SQS (maxReceiveCount with a redrive policy) and RabbitMQ (dead-letter exchanges) support this natively. With Kafka, consumers publish failures to a separate DLQ topic themselves, often with an intermediate "retry" topic for delayed retries.

#Classify errors

• Transient (timeouts, 503s, lock conflicts): retry with backoff.
• Permanent (validation errors, unknown message types, 4xx from a dependency): dead-letter immediately; retrying wastes time.

#Operating a DLQ

• Alert when messages arrive in it; a DLQ nobody watches is data loss with extra steps.
• Inspect the error, fix the cause (data or code), then replay the messages back into the main queue.
• Make sure replayed messages are safe to process — your consumer is idempotent, and messages carry enough context (versions, timestamps) to be applied late.
• Set retention long enough to cover a weekend.

Rising DLQ volume after a deploy is one of the clearest signals to roll back. See message queues vs event logs.