Idempotency

Backend Fundamentals · Core Backend Fundamentals · 9 min read

1. What problem does this solve?

Motivation
When a client sends a request that changes state (e.g., create an order, charge a card, update a profile), the client often wants to be sure that the operation is performed exactly once. Network glitches, retries, or duplicate submissions can cause the same request to hit the server multiple times, leading to duplicate records, double charges, or inconsistent data.

What problem exists?

• Duplicate side‑effects (two orders, two charges).
• Inconsistent state (partial updates applied twice).
• Harder error handling (client doesn’t know if the first attempt succeeded).

Why the problem matters

• Financial systems: double charging a customer is catastrophic.
• Inventory: duplicate orders can deplete stock incorrectly.
• User experience: duplicate emails or notifications.

What happens if idempotency is not used?

• Clients must implement complex retry logic that can’t distinguish “already processed” from “failed”.
• Backend must guard against duplicates manually (unique constraints, checks), which is error‑prone.
• Auditing becomes difficult because you can’t reliably trace a single logical operation.

Where this concept appears in real backend systems

• Payment gateways (Stripe, PayPal) expose an Idempotency-Key header.
• Cloud APIs (AWS S3 PUT, Google Cloud Storage write).
• Microservice orchestration (Kafka message deduplication).
• RESTful APIs that create resources (POST /orders).

2. Mental model

Analogy
Imagine a vending machine that accepts a coin and dispenses a snack. If you accidentally press the button twice, you might get two snacks. An idempotent vending machine would detect the second press and refuse to dispense again, ensuring you only get one snack per coin.

Engineering model

• Key: A unique identifier supplied by the client (e.g., UUID).
• State machine: The server stores the key and the result of the operation.
• Idempotent operation: When the same key is seen again, the server returns the stored result instead of re‑executing the side‑effect.

3. Deep explanation

Internals

1. Idempotency key – a client‑generated value (UUID, hash of payload, etc.).
2. Storage – persist the key + outcome (status, response body, error).
3. Lookup – before performing the operation, check if the key exists.
4. Execution – if not found, run the operation, store the result, and return it.
5. Return – if found, return the stored result (or error) immediately.

Important terminology

• Idempotent: Performing an operation multiple times has the same effect as performing it once.
• Idempotency key: The unique identifier that ties a request to its outcome.
• Idempotency store: The persistence layer that holds keys and results.
• TTL (Time‑to‑Live): Optional expiration for keys to avoid unbounded growth.

Trade‑offs

Failure modes

• Store failure: If the idempotency store is down, duplicates may slip through.
• Race conditions: Two concurrent requests with the same key may both think they’re the first. Use atomic upsert or locking.
• Key collision: Client re‑uses a key for a different operation – must be avoided.

Limitations

• Only protects against exactly the same request.
• Does not prevent different requests that have the same effect (e.g., two distinct orders).
• Requires careful key generation strategy (e.g., include payload hash).

Where it fits in a backend architecture

• API gateway: Often the first place to validate and store the key.
• Service layer: Performs the actual business logic after key lookup.
• Database: Stores the key + result, often in a separate table or key‑value store (Redis, PostgreSQL).

Interaction with related concepts

• Transactions: Idempotency key storage should be part of the same transaction that performs the side‑effect to avoid “key stored but operation failed”.
• Caching: Idempotency store can be a cache (Redis) for fast lookup, with a fallback to persistent DB.
• Event sourcing: Idempotent commands become idempotent events; duplicate events are ignored.

4. How it works

1. Client generates a UUID idempotencyKey.
2. Client sends request with header Idempotency-Key: <key>.
3. API gateway checks the key store.
   • If key exists → return stored response.
   • If key missing → proceed.
4. Begin a database transaction.
5. Perform the business operation.
6. Store the key + outcome in the idempotency table (atomic upsert).
7. Commit the transaction.
8. Return the response to the client.

sequenceDiagram
    participant Client
    participant Gateway
    participant Service
    participant DB

    Client->>Gateway: POST /orders (Idempotency-Key: X)
    Gateway->>DB: SELECT * FROM idempotency WHERE key = X
    alt Key exists
        Gateway->>Client: 200 OK (cached response)
    else
        Gateway->>Service: processOrder()
        Service->>DB: BEGIN
        Service->>DB: INSERT INTO orders (...)
        Service->>DB: INSERT INTO idempotency (key, status, response) VALUES (X, 'SUCCESS', ...)
        Service->>DB: COMMIT
        Service->>Gateway: 201 Created
        Gateway->>Client: 201 Created
    end

5. Practical implementation

Below is a minimal Node.js/TypeScript example using Express, PostgreSQL (via pg), and Redis for fast lookup.
The idempotency table is in PostgreSQL; Redis is optional for performance.

// src/idempotency.ts
import { Pool } from 'pg';
import { createClient } from 'redis';

const pgPool = new Pool(); // assumes PG env vars
const redis = createClient(); // assumes REDIS_URL

/**
 * Store or retrieve idempotency result.
 * @param key - Idempotency key
 * @param fn - Function that performs the operation if key is missing
 */
export async function withIdempotency<T>(
  key: string,
  fn: () => Promise<T>
): Promise<T> {
  // 1. Fast path: check Redis
  const cached = await redis.get(key);
  if (cached) {
    return JSON.parse(cached) as T;
  }

  // 2. Lock to avoid race conditions
  const lockKey = `lock:${key}`;
  const lockAcquired = await redis.set(lockKey, '1', {
    NX: true,
    EX: 30, // 30s lock
  });

  if (!lockAcquired) {
    // Another request is processing; wait until it finishes
    await new Promise((res) => setTimeout(res, 100));
    return withIdempotency(key, fn); // retry
  }

  try {
    // 3. Check DB in case Redis missed it
    const client = await pgPool.connect();
    try {
      const res = await client.query(
        'SELECT status, response FROM idempotency WHERE key = $1',
        [key]
      );
      if (res.rows.length) {
        return JSON.parse(res.rows[0].response) as T;
      }

      // 4. Perform the operation inside a transaction
      await client.query('BEGIN');
      const result = await fn(); // e.g., createOrder()
      const responseBody = JSON.stringify(result);

      // 5. Persist idempotency record
      await client.query(
        `INSERT INTO idempotency (key, status, response, created_at)
         VALUES ($1, $2, $3, NOW())`,
        [key, 'SUCCESS', responseBody]
      );
      await client.query('COMMIT');

      // 6. Cache in Redis
      await redis.set(key, responseBody, { EX: 60 * 60 * 24 }); // 24h TTL

      return result;
    } catch (err) {
      await client.query('ROLLBACK');
      // Store failure so subsequent retries get the same error
      const errorBody = JSON.stringify({ error: err.message });
      await client.query(
        `INSERT INTO idempotency (key, status, response, created_at)
         VALUES ($1, $2, $3, NOW())
         ON CONFLICT (key) DO UPDATE SET status = EXCLUDED.status,
           response = EXCLUDED.response`,
        [key, 'FAILURE', errorBody]
      );
      await redis.set(key, errorBody, { EX: 60 * 60 * 24 });
      throw err;
    } finally {
      client.release();
    }
  } finally {
    // Release lock
    await redis.del(lockKey);
  }
}

Usage in an Express route

// src/routes/orders.ts
import express from 'express';
import { withIdempotency } from '../idempotency';
import { createOrder } from '../services/orderService';

const router = express.Router();

router.post(
  '/orders',
  async (req, res, next) => {
    const key = req.header('Idempotency-Key');
    if (!key) {
      return res.status(400).json({ error: 'Missing Idempotency-Key' });
    }

    try {
      const order = await withIdempotency(key, async () => {
        // Business logic
        return await createOrder(req.body);
      });
      res.status(201).json(order);
    } catch (err) {
      next(err);
    }
  }
);

export default router;

createOrder example

// src/services/orderService.ts
import { Pool } from 'pg';
const pool = new Pool();

export async function createOrder(data: any) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const insert = await client.query(
      'INSERT INTO orders (user_id, amount, status) VALUES ($1, $2, $3) RETURNING *',
      [data.userId, data.amount, 'PENDING']
    );
    const order = insert.rows[0];
    await client.query('COMMIT');
    return order;
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale?

• Distributed cache: Use Redis Cluster or DynamoDB with partitioning.
• Atomicity: Leverage INSERT ... ON CONFLICT or SELECT FOR UPDATE to avoid race conditions.
• Batching: For high‑volume services, batch idempotency checks to reduce round‑trips.
• Observability: Instrument key hit/miss ratios, lock wait times, and error rates.
• Cost: Redis memory cost scales with key count; consider TTL policies or pruning strategies.

Design decisions

• Key format: UUID v4 vs. hash of payload + user ID.
• Storage choice: Redis for speed vs. PostgreSQL for durability.
• Locking strategy: Distributed lock (Redlock) vs. database row lock.
• TTL: 24h for most ops; longer for critical ones.
• Idempotency per service: Each microservice maintains its own store to avoid cross‑service contamination.

Bottlenecks

• Cache saturation: Redis memory limits.
• Lock contention: High concurrency on the same key (rare but possible).
• Database write amplification: Every operation writes to the idempotency table.

Failure scenarios

• Cache outage: Fallback to DB lookup; may increase latency.
• DB replication lag: A read from a replica might miss a recent key; use primary or consistent reads.

Monitoring

• Key hit/miss ratio
• Lock acquisition time
• TTL expiration rate
• Error rate per key

When NOT to use

• Read‑only operations – idempotency is unnecessary.
• Idempotent by nature (e.g., GET /status).
• Low‑volume, low‑risk services – the overhead may outweigh benefits.

9. Interview questions

10. Hands‑on task

Task: Add idempotency support to the existing /orders endpoint in the repository.

1. Create an idempotency table in PostgreSQL with columns: key (PK), status, response, created_at.
2. Implement the withIdempotency helper as shown above.
3. Modify the /orders route to require an Idempotency-Key header and use the helper.
4. Write a test that sends the same request twice with the same key and asserts that the database contains only one order record.
5. (Optional) Add a Redis cache layer and measure the difference in latency between cache hit and miss.

11. Quick revision

• Idempotency guarantees a single side‑effect for repeated identical requests.
• Client supplies a unique Idempotency-Key.
• Server checks a fast store (Redis) → if found, return cached response.
• If not found, execute operation inside a transaction, persist key + result.
• Use atomic upsert or distributed lock to avoid race conditions.
• Store failures so retries return the same error.
• TTLs prevent unbounded growth; monitor hit/miss ratios.
• Common pitfalls: non‑unique keys, missing failure storage, race conditions.
• At scale: choose Redis vs. DB, lock strategy, observability, cost.
• Interview focus: key purpose, implementation, race handling, trade‑offs.

12. References

• HTTP Idempotency-Key Header – MDN
• PostgreSQL Upsert – INSERT … ON CONFLICT
• Redis SET NX / Redlock
• Express.js Request Handling
• FastAPI Idempotency Example
• Stripe Idempotency Keys