Exactly‑once semantics

Backend Fundamentals · Messaging & Event-Driven Architecture · 9 min read

1. What problem does this solve?

• Duplicate processing – A message can be delivered more than once (network retries, broker replication, consumer crashes).
• Lost processing – A message can be dropped if the consumer fails before acknowledging it.
• State corruption – Re‑processing the same event can corrupt aggregates, double‑charge a user, or create duplicate records.

Why it matters

• Financial services, inventory, booking systems, and any domain where idempotency is not enough need guarantees that an operation happens once.
• Without it, downstream systems may see inconsistent data, violating business rules and user expectations.

What happens if it’s not used

• Duplicate charges, double‑created orders, inventory over‑commitment, or stale data.
• Hard‑to‑debug bugs that appear only under load or failure conditions.

Where it appears

• Message brokers (Kafka, RabbitMQ, SQS) – “exactly‑once delivery” flags.
• Event‑sourced systems – each event must be applied once.
• Payment gateways – a payment must be captured only once.
• Microservice orchestration – a saga step must not repeat.

2. Mental model

Analogy – The Postal Service

Engineering model

1. Producer writes a message with a unique id (e.g., UUID).
2. Broker stores the message and may replicate it.
3. Consumer reads the message, performs the side‑effect, and atomically records the id as “processed”.
4. If the consumer crashes before recording the id, the message is retried.
5. If the consumer records the id before performing the side‑effect, a deduplication check prevents re‑execution.

3. Deep explanation

Internals

Important terminology

• Idempotent operation – Re‑executing it yields the same result.
• Exactly‑once delivery – The broker guarantees a message is delivered once to each consumer group.
• Exactly‑once processing – The consumer guarantees the side‑effect occurs once.
• Transactional outbox – Decouples event publishing from domain updates while preserving atomicity.
• Two‑phase commit (2PC) – Classic distributed transaction protocol (rarely used in modern microservices).

Trade‑offs

Failure modes

1. Consumer crash after processing but before recording id – Duplicate processing on restart.
2. Consumer crash before processing – Message is retried; safe if idempotent.
3. Broker partition – Duplicate messages may be emitted.
4. Deduplication store failure – Cannot confirm uniqueness; may lead to duplicates or missed events.

Limitations

• No distributed system can guarantee true exactly‑once in the presence of network partitions without sacrificing availability (CAP theorem).
• Requires additional storage and transaction boundaries.
• Adds latency: the consumer must wait for the deduplication write to commit.

Where it fits

• Event‑driven architecture – Each event is processed once to maintain aggregate consistency.
• Microservice communication – Service A publishes an event; Service B consumes it exactly once.
• Database writes – Avoid duplicate rows or double‑charged rows.

Interaction with related concepts

• Idempotency – Exactly‑once is a stronger guarantee; idempotent handlers can be used as a fallback.
• Event sourcing – Each event is an immutable record; duplicates break the event log.
• Saga patterns – Compensation actions must be idempotent; exactly‑once ensures the saga step is not retried unnecessarily.
• CQRS – Command handlers must be idempotent; query side can rely on event ordering.

4. How it works

sequenceDiagram
    participant Producer
    participant Broker
    participant Consumer
    participant DB
    participant Dedup

    Producer->>Broker: send(message, id)
    Broker->>Consumer: deliver(message, id)
    Consumer->>DB: begin transaction
    Consumer->>DB: perform business logic
    Consumer->>Dedup: insert(id)  // atomic
    Dedup-->>Consumer: success
    Consumer->>DB: commit
    Consumer->>Broker: ack(id)

• Step 1 – Producer sends a message with a unique id.
• Step 2 – Broker delivers the message.
• Step 3 – Consumer starts a transaction that includes both the business logic and the deduplication record.
• Step 4 – If the transaction commits, the id is stored and the broker is acknowledged.
• Step 5 – If the consumer crashes before commit, the broker will retry; the dedup check prevents re‑execution.

5. Practical implementation

Below is a minimal Node.js example using Kafka (kafkajs) and PostgreSQL (pg).
It demonstrates an outbox pattern with a deduplication table.

// src/consumer.ts
import { Kafka, EachMessagePayload } from 'kafkajs';
import { Pool } from 'pg';

const kafka = new Kafka({ brokers: ['localhost:9092'] });
const consumer = kafka.consumer({ groupId: 'orders' });

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
});

async function run() {
  await consumer.connect();
  await consumer.subscribe({ topic: 'order-events', fromBeginning: true });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }: EachMessagePayload) => {
      const msgId = message.key?.toString(); // unique id
      const payload = JSON.parse(message.value?.toString() || '{}');

      // 1️⃣ Start a transaction
      const client = await pool.connect();
      try {
        await client.query('BEGIN');

        // 2️⃣ Check deduplication
        const { rowCount } = await client.query(
          'SELECT 1 FROM processed_events WHERE event_id = $1',
          [msgId]
        );
        if (rowCount > 0) {
          // Already processed – skip
          await client.query('COMMIT');
          return;
        }

        // 3️⃣ Perform business logic (e.g., create order)
        await client.query(
          'INSERT INTO orders (id, user_id, amount) VALUES ($1, $2, $3)',
          [payload.orderId, payload.userId, payload.amount]
        );

        // 4️⃣ Record the event id
        await client.query(
          'INSERT INTO processed_events (event_id, processed_at) VALUES ($1, NOW())',
          [msgId]
        );

        // 5️⃣ Commit transaction
        await client.query('COMMIT');

        // 6️⃣ Ack the message
        await consumer.commitOffsets([
          { topic, partition, offset: (parseInt(message.offset) + 1).toString() },
        ]);
      } catch (err) {
        await client.query('ROLLBACK');
        console.error('Processing failed', err);
        // Let the consumer retry (no ack)
      } finally {
        client.release();
      }
    },
  });
}

run().catch(console.error);

Key points

• The processed_events table holds the unique event_id.
• The entire operation is wrapped in a single PostgreSQL transaction, ensuring atomicity.
• If the consumer crashes after the business logic but before the dedup record, the transaction aborts and the message is retried.
• If the consumer crashes after the dedup record but before the business logic, the transaction aborts, but the dedup record is not committed, so the message will be retried.
• The consumer only acknowledges the message after the transaction commits, guaranteeing at‑least‑once delivery from Kafka and exactly‑once processing.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale?

• Architecture decisions – Choose between Transactional outbox + CDC (Kafka Connect) or Kafka exactly‑once with idempotent producers.
• Bottlenecks – The deduplication store becomes a hotspot; consider sharding or a distributed cache (Redis).
• Failure scenarios – Network partitions can cause duplicate messages; design for graceful degradation.
• Monitoring – Track lag, duplicate rates, dedup table growth, and transaction commit times.
• Cost/performance – Extra storage for dedup keys, extra DB connections, and potential lock contention.
• When NOT to use – Simple CRUD services with low traffic; idempotent handlers may suffice.

Design checklist

1. Unique event id – UUID or deterministic hash.
2. Atomic write – Same transaction for business logic + dedup.
3. Idempotent side‑effects – Where possible, make operations naturally idempotent.
4. Retention policy – Clean up dedup table after a safe window.
5. Testing – Simulate crashes after each step to ensure recovery.

9. Interview questions

10. Hands‑on task

Task:
Implement a simple “order‑processor” service that consumes order-created events from a Kafka topic and writes orders to PostgreSQL, ensuring exactly‑once semantics.

1. Create a PostgreSQL table orders and processed_events.
2. Write a Node.js consumer (using kafkajs) that:
   • Reads messages with a unique event_id.
   • Starts a transaction, inserts the order, inserts the event_id into processed_events.
   • Commits the transaction and acknowledges the message.
3. Test by sending the same message twice and verifying only one order row is created.
4. Add a simple metric (e.g., using prom-client) that counts duplicate attempts.

11. Quick revision

• Exactly‑once = message delivered and processed only once.
• Requires unique event ids and deduplication.
• Use transactions to atomically write business data + dedup record.
• Kafka’s exactly‑once delivery is broker‑side; consumer must still ensure side‑effect idempotency.
• Outbox + CDC decouples publishing from business logic.
• Idempotent handlers are a fallback but not a replacement.
• Failure modes: consumer crash before commit → retry; after commit → duplicate if dedup missing.
• Performance: index processed_events.event_id; consider TTL.
• Monitoring: track duplicate events, consumer lag, transaction commit times.
• Common pitfall: auto‑committing offsets before DB commit.
• When not to use: low‑traffic CRUD services; idempotent operations suffice.

12. References

• Kafka – Exactly‑once semantics: https://kafka.apache.org/documentation/#exactly_once
• KafkaJS documentation: https://kafka.js.org/docs
• PostgreSQL – Transactions: https://www.postgresql.org/docs/current/transaction-iso.html
• Outbox pattern: https://microservices.io/patterns/data/transactional-outbox.html
• Prometheus client for Node.js: https://github.com/siimon/prom-client
• Node.js pg module: https://node-postgres.com/

(All URLs are official or widely accepted resources.)