Pub/Sub — Decoupling producers and consumers In a monolithic app a function that writes to a database often calls another function that sends an email. If the email service crashes, the…
Message queues — Message queues are the backbone of many modern backend systems: order processing pipelines, event sourcing, log aggregation, background job queues, and microservice communication.
RabbitMQ — Motivation In a distributed backend, services often need to talk to each other. If every service calls another synchronously (e.g., HTTP request), the caller must wait for the…
Kafka — Imagine a large bulletin board in an office. Employees (producers) post notes (messages) on the board. Anyone (consumers) can read the notes later. The board is divided into…
Redis Streams — Problem Traditional in‑memory queues or simple Redis lists (LPUSH/RPOP) lack built‑in consumer groups, pending‑message tracking, or automatic rebalancing. They also provide only…
Consumer groups — Imagine a group of workers (consumers) sharing a pile of tasks (messages). Each task is a job card. The workers form a “task squad” (consumer group). When a new job card appears…
Message acknowledgment — When a backend service consumes a message from a queue or stream, it must confirm that it has processed the message successfully. Without this confirmation the system cannot…
Retry mechanisms — When a backend service calls another service, database, or external API, the call can fail for many reasons: transient network glitches, temporary unavailability of the target…
Dead‑letter queues — Without a DLQ, you risk silent data loss, hard‑to‑debug failures, and degraded system performance.
Event‑Driven Architecture — Event‑driven architecture (EDA) addresses these by decoupling producers and consumers through an asynchronous message bus. Producers publish events; consumers subscribe and react…
Event sourcing — Motivation In a traditional CRUD system the current state of an entity (e.g., a bank account balance) is stored in a table and updated in place. If you need to audit, replay, or…
CQRS — Motivation In many applications, the same data store is used for both reading (queries) and writing (commands). When the workload grows, the read side can become a bottleneck for…
At‑least‑once delivery — When a system sends a message (e.g., an event, a task, a notification) it wants the receiver to process it at least once. The motivation is that in real‑world distributed systems…
At-most-once delivery — When a backend system receives a message (e.g., a user signup event, a payment notification, or a sensor reading), it must decide how many times to process that message.
Exactly‑once semantics — Below is a minimal Node.js example using Kafka (kafkajs) and PostgreSQL (pg). It demonstrates an outbox pattern with a deduplication table.
Ordering — Motivation In distributed systems, operations often happen in parallel and messages can arrive out of the order they were sent. For many business scenarios the sequence of events…
Backpressure — Motivation In a distributed backend, services often produce data faster than downstream consumers can consume it. Think of a high‑traffic API that writes to a queue, but the…