Message queues

Backend Fundamentals · Messaging & Event-Driven Architecture · 9 min read

1. What problem does this solve?

Message queues are the backbone of many modern backend systems: order processing pipelines, event sourcing, log aggregation, background job queues, and microservice communication.

2. Mental model

Analogy: The Postal Service

• Producers are people who write letters (messages).
• Queues are post offices that hold letters until a courier (consumer) picks them up.
• Consumers are couriers that deliver letters to recipients.
• Broker is the national postal network that routes letters between post offices.

Just as the postal system decouples the sender from the receiver, a message queue decouples the producer from the consumer. The sender can finish its work immediately after dropping the letter, while the receiver processes it at its own pace.

Engineering model

Producer  →  Broker (queue)  →  Consumer

• Producer publishes a message to a topic or queue.
• Broker stores the message durably and makes it available to consumers.
• Consumer pulls (or is pushed) the message, processes it, and acknowledges receipt.

3. Deep explanation

Internals

Important terminology

• Producer – entity that sends messages.
• Consumer – entity that receives messages.
• Broker – the message‑queue server (e.g., RabbitMQ, Kafka, Redis Streams).
• Queue – a FIFO buffer (RabbitMQ, SQS).
• Topic – a publish/subscribe log (Kafka, Redis Streams).
• Partition – a slice of a topic that allows parallel consumption.
• Consumer Group – a set of consumers that share a topic’s partitions.
• Acknowledgement (Ack) – consumer confirms successful processing.
• At‑least‑once – message may be delivered multiple times but never lost.
• At‑most‑once – message delivered once or not at all (risk of loss).
• Exactly‑once – hard to guarantee; often achieved via idempotent consumers.

Trade‑offs

Failure modes

• Broker crash – messages may be lost if not persisted or replicated.
• Consumer crash – unacked messages re‑queued; can lead to duplicates.
• Network partition – split‑brain; some consumers may see stale data.
• Message corruption – malformed payloads can crash consumers.

Limitations

• Eventual consistency – consumers may lag behind producers.
• No built‑in transaction across services – requires application‑level choreography.
• Limited queryability – cannot easily search past messages without external indexing.

Where it fits

• Microservice communication – async request/response or event bus.
• Background jobs – offload CPU‑heavy work.
• Event sourcing – store domain events in a log.
• Data pipelines – stream data between ingestion, processing, and storage layers.

Interaction with related concepts

4. How it works

1. Producer serializes a payload and sends it to the broker’s queue/topic.
2. Broker writes the message to disk (or memory) and assigns a unique ID/offset.
3. Consumer pulls the message (pull) or receives it via push.
4. Consumer processes the payload.
5. Consumer sends an ack back to the broker.
6. Broker removes the message from the queue (or marks it as consumed).
7. If ack is not received within a timeout, the broker re‑queues the message.

sequenceDiagram
    participant P as Producer
    participant B as Broker
    participant C as Consumer

    P->>B: Publish(message)
    B->>B: Persist(message)
    B->>C: Deliver(message)
    C->>C: Process(message)
    C->>B: Ack(message)
    B->>B: Remove(message)

5. Practical implementation

5.1 RabbitMQ with amqplib (Node.js)

// rabbitmq.ts
import amqp from 'amqplib';

const RABBIT_URL = 'amqp://localhost';
const QUEUE = 'orders';

export async function publishOrder(order: any) {
  const conn = await amqp.connect(RABBIT_URL);
  const ch = await conn.createChannel();

  // Ensure the queue exists
  await ch.assertQueue(QUEUE, { durable: true });

  // Serialize and send
  const msg = Buffer.from(JSON.stringify(order));
  ch.sendToQueue(QUEUE, msg, { persistent: true });

  console.log('Order queued:', order.id);
  await ch.close();
  await conn.close();
}

// consumer.ts
import amqp from 'amqplib';

const RABBIT_URL = 'amqp://localhost';
const QUEUE = 'orders';

async function consume() {
  const conn = await amqp.connect(RABBIT_URL);
  const ch = await conn.createChannel();

  await ch.assertQueue(QUEUE, { durable: true });

  // Prefetch 1 to avoid overloading the consumer
  ch.prefetch(1);

  console.log('Waiting for orders...');
  ch.consume(
    QUEUE,
    async (msg) => {
      if (!msg) return;
      const order = JSON.parse(msg.content.toString());
      try {
        // Process order (e.g., charge payment, update inventory)
        await processOrder(order);
        ch.ack(msg); // Success
      } catch (err) {
        console.error('Processing failed', err);
        ch.nack(msg, false, false); // Drop or requeue based on policy
      }
    },
    { noAck: false }
  );
}

async function processOrder(order: any) {
  // Simulate async work
  await new Promise((r) => setTimeout(r, 200));
  console.log('Processed order', order.id);
}

consume().catch(console.error);

Key points

• persistent: true ensures the message survives broker restarts.
• prefetch(1) limits unacked messages to one, preventing a slow consumer from blocking others.
• ack/nack control message removal or requeue.

5.2 Redis Streams (ioredis)

// redis-stream.ts
import Redis from 'ioredis';

const redis = new Redis(); // defaults to localhost:6379
const STREAM = 'orders';
const GROUP = 'order-processor';
const CONSUMER = 'worker-1';

async function produce(order: any) {
  await redis.xadd(
    STREAM,
    '*',
    'id',
    order.id,
    'amount',
    order.amount.toString()
  );
  console.log('Order added to stream', order.id);
}

async function consume() {
  // Create consumer group if not exists
  try {
    await redis.xgroup('CREATE', STREAM, GROUP, '
  

, 'MKSTREAM');
  } catch (e) {
    if (!e.message.includes('BUSYGROUP')) throw e;
  }

  while (true) {
    const msgs = await redis.xreadgroup(
      'GROUP',
      GROUP,
      CONSUMER,
      'COUNT',
      1,
      'BLOCK',
      5000,
      'STREAMS',
      STREAM,
      '>'
    );

    if (!msgs) continue;

    const [stream, entries] = msgs[0];
    for (const [id, fields] of entries) {
      const order = parseFields(fields);
      try {
        await processOrder(order);
        await redis.xack(STREAM, GROUP, id);
      } catch (err) {
        console.error('Failed', err);
        // Optionally move to a DLQ stream
      }
    }
  }
}

function parseFields(fields: string[]) {
  const obj: any = {};
  for (let i = 0; i < fields.length; i += 2) {
    obj[fields[i]] = fields[i + 1];
  }
  return obj;
}

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

Architectural decisions

• Broker choice: RabbitMQ for complex routing, Kafka for high‑throughput logs, Redis Streams for lightweight queues.
• Partitioning strategy: Key‑based partitioning to preserve ordering per entity.
• Consumer group design: Balance parallelism vs ordering guarantees.
• Retention policy: Long‑term logs vs short‑lived jobs.
• Multi‑region deployment: Latency vs consistency trade‑off.

Bottlenecks & mitigations

• Broker I/O: Use SSDs, tune page cache, enable compression.
• Network: Deploy brokers close to consumers, use VPC peering.
• Consumer lag: Monitor lag, add workers, optimize processing.

Failure scenarios

• Broker split‑brain: Use quorum or consensus protocols (Raft, Zookeeper).
• Consumer crash: Implement graceful shutdown, graceful rebalancing.
• Duplicate messages: Enforce idempotency, use unique message IDs.

Monitoring

• Throughput: Messages per second, per consumer.
• Latency: Producer‑to‑consumer time, consumer processing time.
• Lag: Offset difference between producer and consumer.
• Error rates: DLQ count, retry failures.

Cost/performance

• Managed services (e.g., AWS SQS, Azure Service Bus, GCP Pub/Sub) reduce ops but may limit tuning.
• Self‑hosted gives full control but requires cluster management.

When NOT to use

• Simple background jobs that fit in a database queue.
• Low‑volume, low‑latency use cases where a lightweight in‑memory queue suffices.
• When you need strong transactional guarantees across services (use a saga or two‑phase commit instead).

9. Interview questions

10. Hands‑on task

Build a simple order‑processing pipeline

1. Producer – A Next.js API route (/api/orders) that accepts an order payload and publishes it to a RabbitMQ queue.
2. Consumer – A Node.js worker that consumes the queue, simulates inventory check and payment, then writes the result to a PostgreSQL table.
3. Observability – Expose Prometheus metrics (queue_depth, processing_latency) and log each step.
4. Testing – Write a Jest test that enqueues an order and verifies the database record after a short delay.

Goal: Understand the full flow from HTTP request → queue → background worker → persistence, and observe how the queue decouples the API from the heavy processing.

11. Quick revision

• Decoupling – Producers and consumers operate independently.
• Load leveling – Queues buffer bursts and smooth traffic.
• Durability – Persistent storage protects against broker crashes.
• Ack/Nack – Controls message removal and retry.
• Consumer groups – Parallel consumption with partitioning.
• Dead‑letter queue – Handles failed messages.
• Ordering – Guaranteed within a partition; global ordering is hard.
• Exactly‑once – Requires idempotent consumers or transactional patterns.
• Scaling – Horizontal scaling of brokers and consumers.
• Observability – Metrics: throughput, latency, lag; logs; tracing.
• Security – TLS, ACLs, authentication.
• Common pitfalls – No persistence, high prefetch, no DLQ, duplicate handling.

12. References

• RabbitMQ: https://www.rabbitmq.com/documentation.html
• Kafka: https://kafka.apache.org/documentation/
• Redis Streams: https://redis.io/docs/manual/data-types/streams/
• AMQP 0-9-1 spec: https://www.rabbitmq.com/amqp-0-9-1-reference.html
• Prometheus metrics for RabbitMQ: https://github.com/kbudde/rabbitmq_exporter
• OpenTelemetry for Node.js: https://opentelemetry.io/docs/instrumentation/node/
• AWS SQS documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html
• Azure Service Bus docs: https://learn.microsoft.com/en-us/azure/service-bus-messaging/
• GCP Pub/Sub docs: https://cloud.google.com/pubsub/docs/overview