At-most-once delivery

Backend Fundamentals · Messaging & Event-Driven Architecture · 7 min read

1. What problem does this solve?

When a backend system receives a message (e.g., a user signup event, a payment notification, or a sensor reading), it must decide how many times to process that message.

• Problem: If the same message is processed twice, side‑effects (e.g., charging a user twice, sending duplicate emails, or inserting duplicate rows) can corrupt data or break business rules.
• Why it matters: In financial, medical, or e‑commerce systems, duplicate processing can lead to monetary loss, regulatory violations, or user frustration.
• What happens if it’s not used:
  • Duplicate records in databases.
  • Double charges or double notifications.
  • Inconsistent state across services.
• Real‑world appearance:
  • Payment gateways (e.g., Stripe) guarantee at‑most‑once for charge events.
  • Email delivery services (SendGrid) use at‑most‑once to avoid duplicate emails.
  • IoT platforms often drop duplicate sensor readings to keep storage costs low.

2. Mental model

Analogy: Think of a single‑use ticket for a concert.

• You can use it once (enter the venue).
• If you lose the ticket, you cannot re‑enter (ticket is gone).
• If you try to use it twice, the system will reject the second attempt.

Engineering model:

• Consume → Mark as processed → Execute business logic.
• If the system crashes after marking but before executing, the message is lost (at‑most‑once).
• If it crashes before marking, the message will be re‑queued and may be processed again (at‑least‑once).
• At‑most‑once deliberately sacrifices reliability for simplicity and speed.

3. Deep explanation

4. How it works

1. Consumer polls broker for a batch of messages.
2. Immediately commit the offsets of the batch to the broker.
3. Process each message in the batch.
4. If processing fails, the message is discarded (lost).
5. If processing succeeds, move to the next batch.

sequenceDiagram
    participant C as Consumer
    participant B as Broker
    participant DB as Database

    C->>B: Poll for messages
    B->>C: Return batch [msg1, msg2]
    C->>B: Commit offsets for msg1, msg2
    C->>DB: Process msg1
    C->>DB: Process msg2
    Note over C: If crash after commit, msg1/2 lost

5. Practical implementation

Below is a minimal Node.js consumer using KafkaJS that demonstrates at‑most‑once semantics.
It commits offsets before processing and uses a Redis set to deduplicate in case the consumer crashes before processing.

// atMostOnceConsumer.ts
import { Kafka, EachMessagePayload } from 'kafkajs';
import Redis from 'ioredis';

const kafka = new Kafka({
  clientId: 'order-service',
  brokers: ['kafka:9092'],
});

const consumer = kafka.consumer({ groupId: 'orders' });
const redis = new Redis(); // default localhost:6379

/**
 * Idempotent handler: writes order to DB only if not already processed.
 */
async function handleOrder(msg: EachMessagePayload) {
  const orderId = msg.message.key?.toString() ?? '';
  const orderData = JSON.parse(msg.message.value?.toString() ?? '{}');

  // Check Redis for deduplication key
  const alreadyProcessed = await redis.sismember('orders:processed', orderId);
  if (alreadyProcessed) {
    console.log(`Order ${orderId} already processed – skipping`);
    return;
  }

  // Persist to DB (pseudo‑code)
  await db.query('INSERT INTO orders (id, data) VALUES ($1, $2)', [orderId, orderData]);

  // Mark as processed
  await redis.sadd('orders:processed', orderId);
}

async function run() {
  await consumer.connect();
  await consumer.subscribe({ topic: 'orders', fromBeginning: false });

  await consumer.run({
    // `eachBatchAutoResolve` is true by default; we set it to false to control commit timing
    eachBatchAutoResolve: false,
    eachBatch: async ({ batch, resolveOffset, heartbeat, commitOffsetsIfNecessary }) => {
      // Commit offsets *before* processing
      await consumer.commitOffsets([
        { topic: batch.topic, partition: batch.partition, offset: (Number(batch.lastOffset()) + 1).toString() },
      ]);

      // Process each message
      for (const message of batch.messages) {
        try {
          await handleOrder({ topic: batch.topic, partition: batch.partition, message });
        } catch (err) {
          console.error('Processing failed', err);
          // In at-most-once we simply skip; the message is lost
        }
        // Resolve offset so consumer knows it was handled
        resolveOffset(message.offset);
      }

      await heartbeat();
      await commitOffsetsIfNecessary();
    },
  });
}

run().catch(console.error);

Key lines

• consumer.commitOffsets([...]) – commits before processing.
• Redis set orders:processed – protects against accidental re‑processing if the consumer crashes after commit but before processing.
• eachBatchAutoResolve: false – gives us explicit control over when offsets are committed.

Tip: In production, replace the Redis set with a durable deduplication store (e.g., a PostgreSQL table with a unique constraint on order_id).

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

When scaling at‑most‑once delivery:

• Architecture decisions
  • Use deduplication tables with unique constraints instead of in‑memory sets.
  • Partition topics to parallelize consumers.
  • Consider a dead‑letter queue for messages that fail processing after commit.
• Trade‑offs
  • Acceptable data loss vs. complexity of exactly‑once.
  • Cost of additional storage for dedup keys vs. risk of duplicates.
• Bottlenecks
  • Offset commit latency if broker is slow.
  • Dedup store contention under high throughput.
• Failure scenarios
  • Broker outage → consumer stops, messages accumulate.
  • Consumer crash after commit → lost messages.
• Monitoring
  • Lag metrics (consumer_lag).
  • Ratio of committed vs. processed messages.
  • Error rates in processing.
• Cost/performance
  • Batching commits reduces network overhead.
  • Using Redis vs. Postgres for dedup: Redis is faster but volatile.
• When NOT to use
  • Financial transactions where loss is unacceptable.
  • Regulatory data that must be preserved.
  • Systems requiring audit trails of every event.

9. Interview questions

10. Hands‑on task

Task: Build a small Node.js service that consumes a Kafka topic user-registrations using at‑most‑once semantics.

• Commit offsets before processing.
• Store processed user IDs in Redis to avoid accidental re‑processing if the consumer crashes after commit.
• Log each processed user and the time it was processed.
• Expose a /metrics endpoint (Prometheus) that reports the number of processed vs. committed messages.

Deliverables:

• consumer.ts implementing the logic.
• Dockerfile to run Kafka, Redis, and the consumer.
• A simple script to produce test messages to user-registrations.
• A README with instructions to run the stack locally.

11. Quick revision

• At‑most‑once: commit before processing → no duplicates, possible loss.
• Offset commit: broker‑side marker of “delivered”.
• Deduplication: idempotent handlers or persistent dedup keys.
• Trade‑off: speed & simplicity vs. data loss.
• Common pitfall: committing after processing → duplicates.
• Use cases: analytics, logs, non‑critical events.
• Monitoring: consumer lag, processed/committed ratio, error rates.
• Mitigation: dead‑letter queue, durable dedup store, graceful shutdown.
• When to avoid: financial, regulatory, audit‑required data.
• Key libraries: KafkaJS, ioredis, Prometheus client.
• Deployment: Docker/K8s with health probes, auto‑restart.

12. References

• KafkaJS – Consumer
• Kafka – Exactly‑Once Semantics
• Redis – Sets
• Prometheus – Client for Node.js
• Node.js – Streams and Event Loop
• Kafka – Consumer Offsets