Distributed transactions

Backend Fundamentals · Distributed Systems · 8 min read

1. What problem does this solve?

When a business operation touches multiple independent data stores or services (e.g., an order service, inventory service, payment gateway, and shipping service), we need to guarantee that all parts either succeed together or fail together.
Without a distributed transaction mechanism:

• Inconsistency: One service commits while another rolls back, leaving stale or orphaned data.
• Business errors: A customer may be charged but not receive the product, or inventory may be deducted without a payment.
• Hard to debug: Inconsistent state is difficult to trace across services.

In real backends, this problem appears in:

• E‑commerce checkout flows
• Banking transfers across accounts in different regions
• Multi‑service orchestration in SaaS platforms
• Microservice architectures where each service owns its own database

2. Mental model

Analogy – The “Three‑Way Handshake”

Imagine a group of friends deciding to go out together.

1. Proposal: One friend suggests a plan.
2. Commit: All friends agree (send “yes”).
3. Finalize: Everyone confirms they’re ready (send “ready”).

If any friend says “no” at any step, the whole plan is aborted.
This is similar to a two‑phase commit (2PC): a prepare phase where all participants agree, followed by a commit phase where the decision is finalized.

Engineering model

In distributed systems, a coordinator (or orchestrator) sends a prepare request to all participants.

• If every participant replies prepared, the coordinator sends commit.
• If any participant replies abort or times out, the coordinator sends abort to all.

Because the coordinator must wait for all participants, 2PC is blocking and can suffer from long‑running locks.
Modern microservices often use the Saga pattern instead, which breaks the transaction into a series of independent steps with compensating actions.

3. Deep explanation

Failure modes

• Coordinator crash: In 2PC, participants may be left in a prepared state, leading to deadlocks.
• Participant crash: If a participant fails after preparing, it must recover and inform the coordinator.
• Network partition: Can cause timeouts and ambiguous states.
• Compensation failure: In a Saga, if a compensating action fails, the system may end up partially rolled back.

Limitations

• 2PC/3PC are not suitable for high‑throughput, low‑latency microservices.
• Sagas provide eventual consistency but not strict ACID guarantees.
• Designing compensations can be non‑trivial, especially for external systems (e.g., payment gateways).

Where it fits

Distributed transactions sit at the intersection of:

• Database layer (ACID vs. BASE)
• Service orchestration (Saga orchestrator vs. choreography)
• Messaging (Kafka, RabbitMQ, Redis Streams)
• Observability (distributed tracing, logs)

They are often implemented as a separate transaction manager service or embedded in the application logic.

4. How it works

Two‑Phase Commit (simplified)

sequenceDiagram
    participant C as Coordinator
    participant A as Service A
    participant B as Service B

    C->>A: Prepare
    C->>B: Prepare
    A-->>C: Prepared
    B-->>C: Prepared
    C->>A: Commit
    C->>B: Commit

Saga Orchestration (simplified)

sequenceDiagram
    participant O as Orchestrator
    participant I as Inventory Service
    participant P as Payment Service
    participant S as Shipping Service

    O->>I: ReserveStock
    I-->>O: Success
    O->>P: ChargeCard
    P-->>O: Success
    O->>S: CreateShipment
    S-->>O: Success
    Note right of O: Order completed

    %% Failure path
    O->>P: ChargeCard
    P-->>O: Failure
    O->>I: ReleaseStock   %% Compensation

5. Practical implementation

Below is a Saga orchestrator written in TypeScript/Node.js using Redis Streams for messaging.
It coordinates an order flow across an inventory service and a payment service.

// saga-orchestrator.ts
import { createClient } from 'redis';
import axios from 'axios';

const redis = createClient();
await redis.connect();

const ORDER_STREAM = 'orders';
const INVENTORY_STREAM = 'inventory';
const PAYMENT_STREAM = 'payment';

interface Order {
  id: string;
  productId: string;
  quantity: number;
  userId: string;
  status: 'pending' | 'completed' | 'failed';
}

// Helper to publish a message to a stream
async function publish(stream: string, message: any) {
  await redis.xAdd(stream, '*', message);
}

// Orchestrator logic
async function handleOrder(order: Order) {
  try {
    // 1️⃣ Reserve stock
    await publish(INVENTORY_STREAM, { orderId: order.id, action: 'reserve', ...order });

    // 2️⃣ Charge payment
    await publish(PAYMENT_STREAM, { orderId: order.id, action: 'charge', ...order });

    // 3️⃣ Mark order completed
    order.status = 'completed';
    await publish(ORDER_STREAM, { orderId: order.id, status: order.status });
    console.log(`Order ${order.id} completed`);
  } catch (err) {
    // 4️⃣ Compensation path
    await publish(INVENTORY_STREAM, { orderId: order.id, action: 'release', ...order });
    order.status = 'failed';
    await publish(ORDER_STREAM, { orderId: order.id, status: order.status });
    console.error(`Order ${order.id} failed:`, err);
  }
}

// Example usage
const newOrder: Order = {
  id: 'ord_123',
  productId: 'prod_456',
  quantity: 2,
  userId: 'user_789',
  status: 'pending',
};

handleOrder(newOrder);

Inventory Service (simplified)

// inventory-service.ts
import { createClient } from 'redis';
const redis = createClient();
await redis.connect();

const INVENTORY_STREAM = 'inventory';

redis.xReadGroup('inventory-group', 'consumer-1', { [INVENTORY_STREAM]: '>' }, async (messages) => {
  for (const msg of messages) {
    const { orderId, action, productId, quantity } = msg.message;
    if (action === 'reserve') {
      // Reserve logic (e.g., decrement stock)
      console.log(`Reserving ${quantity} of ${productId} for order ${orderId}`);
      // On success, publish to next step
      await redis.xAdd('payment', '*', { orderId, action: 'ready' });
    } else if (action === 'release') {
      // Release logic
      console.log(`Releasing stock for order ${orderId}`);
    }
  }
});

Payment Service (simplified)

// payment-service.ts
import { createClient } from 'redis';
import axios from 'axios';
const redis = createClient();
await redis.connect();

const PAYMENT_STREAM = 'payment';

redis.xReadGroup('payment-group', 'consumer-1', { [PAYMENT_STREAM]: '>' }, async (messages) => {
  for (const msg of messages) {
    const { orderId, action } = msg.message;
    if (action === 'ready') {
      // Charge card via external API
      try {
        await axios.post('https://payment-gateway.example.com/charge', { orderId });
        console.log(`Charged order ${orderId}`);
        // Notify orchestrator of success
        await redis.xAdd('orders', '*', { orderId, status: 'completed' });
      } catch (e) {
        console.error(`Payment failed for order ${orderId}`);
        // Notify orchestrator of failure
        await redis.xAdd('orders', '*', { orderId, status: 'failed' });
      }
    }
  }
});

Note: In a real system you would add idempotency keys, retries, and proper error handling. The example is intentionally minimal to illustrate the flow.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale?

• Architecture: Adopt a Saga orchestrator cluster with leader election; use a dedicated message broker per domain.
• Trade‑offs: Accept eventual consistency for higher throughput; avoid 2PC unless you own all databases.
• Bottlenecks: Coordinator throughput, message broker latency, and compensating action cost.
• Failure scenarios: Network partitions, broker outages, or compensating action failures.
• Monitoring: Distributed tracing (e.g., Jaeger), stream lag dashboards, SLA metrics per step.
• Cost/performance: Running multiple orchestrator replicas, message broker clusters, and persistent storage.
• When NOT to use: For simple, single‑database transactions; for real‑time, low‑latency operations where eventual consistency is unacceptable.

Design decisions

• Orchestrator vs. Choreography: Orchestrator gives central control but adds a single point of failure; choreography is more resilient but harder to reason about.
• Compensation strategy: Prefer idempotent, side‑effect‑free compensations; use domain events to trigger them.
• State persistence: Store saga state in a durable store to survive orchestrator restarts.
• Idempotency keys: Generate per step to avoid duplicate processing.
• Timeouts: Set per‑step timeouts to avoid indefinite waiting.
• Testing: Use chaos engineering to simulate failures in each step.

9. Interview questions

10. Hands‑on task

Implement a simple Saga orchestrator for a “book‑purchase” flow:

1. Create three services: book-service, payment-service, and order-service.
2. Use Redis Streams to coordinate the flow.
3. Each service should expose an HTTP endpoint to trigger its step.
4. The orchestrator should:
   • Reserve a book (decrement stock).
   • Charge the user (simulate with a delay).
   • Create an order record (store in a JSON file or in-memory map).
5. Add a compensation step that releases the book if payment fails.
6. Write unit tests that simulate a payment failure and verify the book stock is restored.

Goal: Demonstrate understanding of message‑driven orchestration, idempotency, and compensation.

11. Quick revision

• Distributed transactions guarantee atomicity across services.
• 2PC: prepare → commit; blocking, single coordinator.
• 3PC: adds pre‑commit; reduces blocking but still fragile.
• Saga: orchestrated or choreographed steps with compensations; eventual consistency.
• Idempotency is essential for retries.
• Failure modes: coordinator crash, participant crash, network partition.
• Production concerns: scalability, reliability, observability, security, cost.
• Common mistakes: using 2PC in microservices, skipping idempotency, ignoring compensation failures.
• Senior decisions: orchestrator vs. choreography, state persistence, monitoring, cost trade‑offs.

12. References

• Two‑Phase Commit (2PC) – Wikipedia
• Saga Pattern – Microsoft Docs
• Redis Streams – Official Documentation
• TypeORM – Transactions
• OpenTelemetry – Distributed Tracing
• Kafka – Distributed Commit Protocol