Dead‑letter queues

Backend Fundamentals · Messaging & Event-Driven Architecture · 8 min read

1. What problem does this solve?

When a message is processed by a consumer, it can fail for many reasons: transient network glitches, malformed payloads, downstream service outages, or business logic errors.
If the consumer simply discards the message or keeps retrying forever, two things happen:

1. Lost data – the message never reaches its destination, and the system has no record of the failure.
2. Back‑pressure – repeated retries can flood the queue, slow down healthy consumers, and consume resources unnecessarily.

A dead‑letter queue (DLQ) is a safety net that captures messages that cannot be processed after a configurable number of attempts. It allows you to:

• Preserve the failed message for later inspection or re‑processing.
• Prevent a single problematic message from blocking the entire pipeline.
• Separate “good” traffic from “bad” traffic so that monitoring and alerting can focus on real issues.

Without a DLQ, you risk silent data loss, hard‑to‑debug failures, and degraded system performance.

Real‑world systems:

• AWS SQS and SNS support DLQs.
• Kafka can route to a separate topic for failed records.
• RabbitMQ has a “dead‑letter exchange” feature.
• Azure Service Bus offers DLQ support per queue/subscription.

2. Mental model

Analogy
Imagine a mail sorting center. Most letters go straight to the post office. Occasionally a letter is damaged or has an invalid address. Instead of throwing it away, the center places it in a “problematic mail” box. A human operator later reviews that box, fixes the address or discards the letter. The rest of the mail flow continues unhindered.

Engineering model
A DLQ is just another queue (or topic) that receives messages that failed processing after a defined retry policy. The consumer that processes the main queue is responsible for:

1. Detecting a failure.
2. Incrementing a retry counter (often stored in message headers).
3. Deciding whether to re‑enqueue or send to the DLQ.

The DLQ can be inspected, re‑processed, or archived independently.

3. Deep explanation

4. How it works

1. Producer sends a message to orders.queue.
2. Consumer pulls the message.
3. Consumer processes; if it throws an error, it increments x-retry-count.
4. If x-retry-count < maxRetries, the consumer re‑queues the message (or lets it become visible again).
5. If x-retry-count ≥ maxRetries, the broker moves the message to orders.dlq.
6. DLQ consumer (separate process) reads from orders.dlq, logs the failure, and may attempt a manual fix or alert engineers.

sequenceDiagram
    participant Producer
    participant Broker
    participant Consumer
    participant DLQConsumer

    Producer->>Broker: Send message
    Broker->>Consumer: Deliver message
    Consumer->>Consumer: Process
    alt Success
        Consumer->>Broker: Ack
    else Failure
        Consumer->>Broker: Nack (retry)
        loop Retry attempts
            Broker->>Consumer: Deliver message
            Consumer->>Consumer: Process
            alt Success
                Consumer->>Broker: Ack
                break
            else Failure
                Consumer->>Broker: Nack (retry)
            end
        end
        opt Max retries exceeded
            Broker->>DLQConsumer: Move to DLQ
        end
    end
    DLQConsumer->>DLQConsumer: Inspect / alert

5. Practical implementation

Node.js + TypeScript + AWS SQS example

// src/sqsConsumer.ts
import { SQSClient, ReceiveMessageCommand, DeleteMessageCommand, SendMessageCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });

const MAIN_QUEUE_URL = process.env.MAIN_QUEUE_URL!;
const DLQ_URL = process.env.DLQ_QUEUE_URL!;

// Max attempts before dead‑lettering
const MAX_RETRIES = 5;

// Helper to get retry count from message attributes
function getRetryCount(msg: any): number {
  return Number(msg.MessageAttributes?.RetryCount?.StringValue ?? "0");
}

async function processMessage(body: any) {
  // Simulate business logic
  if (Math.random() < 0.3) throw new Error("Random failure");
  // ... real processing
}

async function poll() {
  const res = await sqs.send(new ReceiveMessageCommand({
    QueueUrl: MAIN_QUEUE_URL,
    MaxNumberOfMessages: 10,
    WaitTimeSeconds: 20,
    MessageAttributeNames: ["All"],
  }));

  if (!res.Messages) return;

  for (const msg of res.Messages) {
    const retryCount = getRetryCount(msg);
    try {
      await processMessage(JSON.parse(msg.Body!));
      // Success: delete from queue
      await sqs.send(new DeleteMessageCommand({
        QueueUrl: MAIN_QUEUE_URL,
        ReceiptHandle: msg.ReceiptHandle!,
      }));
    } catch (err) {
      if (retryCount + 1 >= MAX_RETRIES) {
        // Move to DLQ
        await sqs.send(new SendMessageCommand({
          QueueUrl: DLQ_URL,
          MessageBody: msg.Body!,
          MessageAttributes: {
            OriginalQueue: { DataType: "String", StringValue: MAIN_QUEUE_URL },
            RetryCount: { DataType: "Number", StringValue: String(retryCount + 1) },
            Error: { DataType: "String", StringValue: (err as Error).message },
          },
        }));
        // Delete from main queue
        await sqs.send(new DeleteMessageCommand({
          QueueUrl: MAIN_QUEUE_URL,
          ReceiptHandle: msg.ReceiptHandle!,
        }));
      } else {
        // Increment retry count and re‑queue
        await sqs.send(new SendMessageCommand({
          QueueUrl: MAIN_QUEUE_URL,
          MessageBody: msg.Body!,
          MessageAttributes: {
            RetryCount: { DataType: "Number", StringValue: String(retryCount + 1) },
          },
        }));
        // Delete original
        await sqs.send(new DeleteMessageCommand({
          QueueUrl: MAIN_QUEUE_URL,
          ReceiptHandle: msg.ReceiptHandle!,
        }));
      }
    }
  }
}

setInterval(poll, 5000);

Key points

• RetryCount is stored as a message attribute so the broker can’t lose it.
• After MAX_RETRIES attempts, the message is sent to the DLQ.
• The DLQ consumer can be a separate Lambda or Node process that logs or retries manually.

Kafka example (topic per DLQ)

// src/kafkaConsumer.ts
import { Kafka, logLevel } from "kafkajs";

const kafka = new Kafka({
  clientId: "orders-service",
  brokers: ["kafka:9092"],
  logLevel: logLevel.INFO,
});

const consumer = kafka.consumer({ groupId: "orders-group" });
const producer = kafka.producer();

const MAIN_TOPIC = "orders";
const DLQ_TOPIC = "orders-dlq";
const MAX_RETRIES = 5;

async function run() {
  await consumer.connect();
  await producer.connect();
  await consumer.subscribe({ topic: MAIN_TOPIC, fromBeginning: false });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const retry = Number(message.headers?.retry ?? "0");
      try {
        // business logic
        if (Math.random() < 0.2) throw new Error("Kafka fail");
      } catch (err) {
        if (retry + 1 >= MAX_RETRIES) {
          await producer.send({
            topic: DLQ_TOPIC,
            messages: [
              {
                key: message.key,
                value: message.value,
                headers: {
                  ...message.headers,
                  retry: Buffer.from(String(retry + 1)),
                  error: Buffer.from((err as Error).message),
                },
              },
            ],
          });
        } else {
          await producer.send({
            topic: MAIN_TOPIC,
            messages: [
              {
                key: message.key,
                value: message.value,
                headers: {
                  ...message.headers,
                  retry: Buffer.from(String(retry + 1)),
                },
              },
            ],
          });
        }
      }
    },
  });
}

run().catch(console.error);

6. Production considerations

7. Common mistakes

1. No retry counter – moving the same failed message endlessly back to the main queue causes infinite loops.
2. Using the same queue for DLQ – conflating DLQ with the main queue makes it hard to separate good from bad traffic.
3. Ignoring DLQ retention – leaving DLQ messages forever can consume storage and hide new failures.
4. Not preserving metadata – losing original queue, timestamp, or error details makes debugging impossible.
5. Treating DLQ as a “fire‑and‑forget” – failing to set up alerts or manual inspection leads to silent data loss.
6. Over‑aggressive back‑off – waiting too long before retrying can delay critical processing.
7. Not handling DLQ consumer failures – if the DLQ consumer crashes, you lose the ability to inspect or re‑process messages.

8. Senior engineer perspective

When used at scale

• Architecture decisions

  • Separate DLQ per service or per logical domain.
  • Use broker‑native DLQ features (SQS, Kafka) rather than custom queues to reduce operational overhead.
  • Consider dead‑letter topics in Kafka for high‑throughput scenarios.

• Trade‑offs

  • Cost vs. safety: DLQ storage adds cost; balance retention with the need to audit failures.
  • Latency: Adding DLQ hops can increase end‑to‑end latency; design for acceptable SLA.

• Bottlenecks

  • A single DLQ consumer can become a bottleneck if many messages accumulate. Scale it horizontally.
  • Broker retention limits may truncate DLQ messages before they’re inspected.

• Failure scenarios

  • Broker outage: DLQ may be inaccessible; design for graceful degradation.
  • Message size limits: DLQ may reject oversized messages; enforce size checks early.

• Monitoring

  • Dashboards: DLQ size, rate, and age.
  • Alerts: >10% of messages in DLQ, or >5% increase over baseline.

• Cost/performance

  • Use compression for DLQ messages to reduce storage.
  • Batch DLQ writes to improve throughput.

• When NOT to use

  • For idempotent, transient failures that can be retried indefinitely without side effects.
  • In low‑volume systems where manual inspection is acceptable without a DLQ.

9. Interview questions

10. Hands‑on task

Task: Add a dead‑letter queue to the existing orders service in this repository.

1. Create a new SQS queue named orders-dlq.
2. Update the consumer code to move messages to orders-dlq after 3 retries.
3. Deploy the updated consumer to a staging environment.
4. Produce a message that will fail (e.g., send a payload with an invalid field).
5. Verify the message lands in orders-dlq and inspect its attributes.

Goal: Demonstrate end‑to‑end DLQ flow and observe metrics in CloudWatch.

11. Quick revision

• DLQ captures messages that fail after a retry policy.
• Store retry count in message attributes or headers.
• Broker‑native DLQs (SQS, Kafka) simplify implementation.
• Separate DLQ per domain prevents mixing good/bad traffic.
• Monitor DLQ size, rate, and age; alert on spikes.
• Retention policies prevent storage bloat.
• Preserve original metadata for debugging.
• Use idempotent processing when re‑trying from DLQ.
• Scale DLQ consumers independently.
• Encrypt DLQ messages and restrict access.

12. References

• AWS SQS Dead Letter Queues
• Kafka Dead Letter Queue Pattern
• RabbitMQ Dead Letter Exchanges
• Azure Service Bus Dead‑Letter Queues
• Kafkajs – Dead Letter Queue Example