Strong Consistency

Backend Fundamentals · Distributed Systems · 8 min read

1. What problem does this solve?

In a distributed backend, data is often replicated across multiple nodes or services.
When a client writes a value, that write must be visible to all subsequent reads, no matter which node the read hits.
Strong consistency guarantees that every read after a write returns the most recent value.

Why it matters

  • Business correctness – e.g., a bank transfer must not allow overdrafts or double‑spending.
  • User experience – a user updating a profile should immediately see the change.
  • Data integrity – stale data can lead to cascading failures (e.g., inventory counts, seat reservations).

Consequences of not using it

  • Inconsistent state – different services may act on stale data, causing errors or data corruption.
  • Hard-to‑debug bugs – race conditions become invisible until a read occurs on a stale replica.
  • Security risks – stale authorization data can grant unintended access.

Where it appears in real systems

  • Financial services – transaction processing, ledger updates.
  • E‑commerce – inventory, order status, cart updates.
  • Social platforms – likes, follows, real‑time feeds.
  • Databases – PostgreSQL, MySQL, MongoDB (with replica sets), Redis (with SET/GET in a single node).

2. Mental model

Analogy: The “One‑Way Street”

Imagine a one‑way street that only allows cars to move forward.
When a car (write) enters the street, it can’t be overtaken by another car (read) that hasn't yet passed it.
Every car that follows will see the car that just entered before it.

In distributed systems, strong consistency is like that one‑way street: once a write is committed, all subsequent reads must see that write, regardless of which node they hit.

Engineering model

  • Linearizability – the strongest form of consistency: operations appear to occur instantaneously at some point between their invocation and completion.
  • Atomicity – a transaction’s changes are all applied or none.
  • Isolation – concurrent transactions don’t interfere.
  • Durability – once committed, the data survives failures.

3. Deep explanation

Internals

ConceptWhat it isHow it works
Two‑Phase Commit (2PC)Protocol to coordinate a transaction across multiple resources.Phase 1 (Prepare): each participant votes “commit” or “abort”.
Phase 2 (Commit/Abort): coordinator sends commit if all voted yes, else abort.
Write‑ahead loggingLog changes before applying them.Allows rollback and crash recovery.
Consensus (Paxos/Raft)Agreement on the order of operations.Ensures all replicas apply operations in the same order, preserving linearizability.
Quorum reads/writesMajority of replicas must acknowledge.Guarantees that a read sees the latest committed write.

Important terminology

  • Linearizability – operations appear instantaneously.
  • Serializability – transactions appear as if executed in some order.
  • Read‑your‑writes – a client sees its own writes immediately.
  • Strong vs eventual consistency – eventual consistency relaxes guarantees for higher availability.

Trade‑offs

Trade‑offImpact
Latency2PC and consensus add round‑trips; strong consistency often means higher latency.
AvailabilityCAP theorem: you can’t have Consistency + Partition tolerance + Availability simultaneously. Strong consistency sacrifices availability during partitions.
ComplexityImplementing 2PC or consensus is non‑trivial; bugs can be subtle.
ThroughputLocking and coordination reduce concurrency.

Failure modes

  • Coordinator failure in 2PC → deadlock; participants may be left in uncertain state.
  • Network partitions → some nodes can’t reach the coordinator; they must abort or block.
  • Clock skew → ordering of events can be misinterpreted; consensus protocols mitigate this.

Limitations

  • Not suitable for high‑write, low‑latency workloads (e.g., real‑time analytics).
  • Requires synchronous communication; not ideal for geographically distributed services.
  • Hard to scale horizontally without sophisticated consensus layers.

Where it fits

  • Monolithic services – a single database with ACID guarantees.
  • Microservices – use a distributed transaction coordinator or event sourcing with strong ordering.
  • Cache layers – write‑through or write‑back caches that flush to the backing store before acknowledging the client.
  • Caching – strong consistency often forces cache invalidation or write‑through patterns.
  • Event sourcing – events are appended to a log; strong consistency ensures the log order is preserved.
  • Idempotency – necessary to safely retry writes in the face of failures.

4. How it works

Below is a simplified 2PC flow for a bank transfer that updates two PostgreSQL accounts and a Redis cache.

sequenceDiagram
    participant Client
    participant API
    participant Postgres1
    participant Postgres2
    participant Redis
    participant Coordinator

    Client->>API: POST /transfer {from, to, amount}
    API->>Coordinator: beginTransaction()
    Coordinator->>Postgres1: PREPARE tx1
    Coordinator->>Postgres2: PREPARE tx2
    Coordinator->>Redis: WATCH key
    Coordinator->>Redis: MULTI
    Coordinator->>Redis: SET key newBalance
    Coordinator->>Redis: EXEC
    alt all PREPARE OK
        Coordinator->>Postgres1: COMMIT
        Coordinator->>Postgres2: COMMIT
        Coordinator->>API: success
    else any PREPARE FAIL
        Coordinator->>Postgres1: ROLLBACK
        Coordinator->>Postgres2: ROLLBACK
        Coordinator->>API: failure
    end
  • Step 1 – Client initiates transfer.
  • Step 2 – API starts a distributed transaction.
  • Step 3 – Each participant prepares (writes to log, locks rows).
  • Step 4 – Coordinator decides to commit or abort.
  • Step 5 – All participants commit or rollback atomically.

5. Practical implementation

Below is a Node.js example using pg (PostgreSQL) and ioredis.
It demonstrates a two‑phase commit for a transfer between two accounts with a Redis cache update.

// transfer.ts
import { Client as PgClient } from 'pg';
import Redis from 'ioredis';

const pg = new PgClient({ connectionString: process.env.DATABASE_URL });
const redis = new Redis(process.env.REDIS_URL);

async function transfer(fromId: number, toId: number, amount: number) {
  await pg.connect();
  try {
    // 1️⃣ Begin a local transaction
    await pg.query('BEGIN');

    // 2️⃣ Lock rows to prevent concurrent updates
    const resFrom = await pg.query(
      'SELECT balance FROM accounts WHERE id = $1 FOR UPDATE',
      [fromId]
    );
    const resTo = await pg.query(
      'SELECT balance FROM accounts WHERE id = $2 FOR UPDATE',
      [toId]
    );

    const fromBalance = resFrom.rows[0].balance;
    const toBalance = resTo.rows[0].balance;

    if (fromBalance < amount) {
      throw new Error('Insufficient funds');
    }

    // 3️⃣ Update balances
    await pg.query(
      'UPDATE accounts SET balance = $1 WHERE id = $2',
      [fromBalance - amount, fromId]
    );
    await pg.query(
      'UPDATE accounts SET balance = $1 WHERE id = $2',
      [toBalance + amount, toId]
    );

    // 4️⃣ Write‑through cache update (optimistic)
    const cacheKeyFrom = `account:${fromId}`;
    const cacheKeyTo = `account:${toId}`;

    // Use WATCH/MULTI to ensure cache consistency
    await redis.watch(cacheKeyFrom, cacheKeyTo);
    const multi = redis.multi();
    multi.set(cacheKeyFrom, (fromBalance - amount).toString());
    multi.set(cacheKeyTo, (toBalance + amount).toString());
    const execResult = await multi.exec();

    if (!execResult) {
      // Cache was modified concurrently; abort the whole transaction
      throw new Error('Cache conflict');
    }

    // 5️⃣ Commit the DB transaction
    await pg.query('COMMIT');
    console.log('Transfer succeeded');
  } catch (err) {
    // Rollback on any error
    await pg.query('ROLLBACK');
    console.error('Transfer failed:', err);
    throw err;
  } finally {
    await pg.end();
  }
}

Key points

  • FOR UPDATE locks the rows, preventing concurrent updates.
  • WATCH + MULTI ensures the Redis cache is updated atomically with the DB transaction.
  • If the cache changes between WATCH and EXEC, the transaction aborts, guaranteeing strong consistency.
  • In production, you’d replace the manual 2PC with a proper transaction coordinator or use a database that supports distributed transactions (e.g., CockroachDB).

6. Production considerations

ConcernWhat to watch forMitigation
Scalability2PC adds latency; high write volume may saturate the coordinator.Use sharding, partitioning, or move to a strongly consistent distributed DB (e.g., CockroachDB).
ReliabilityCoordinator failure → deadlock.Implement timeouts, retry logic, and a recovery manager.
SecurityExposing transaction details over the network.Encrypt traffic (TLS), use IAM roles for DB access.
ObservabilityHard to trace distributed commits.Add distributed tracing (OpenTelemetry), log transaction IDs.
PerformanceLock contention on rows.Use optimistic concurrency control or version columns.
Error handlingPartial commits.Ensure all participants can roll back; use idempotent operations.
Resource usageExtra memory for logs.Tune log retention, use WAL archiving.
Operational concernsMonitoring coordinator health.Alert on coordinator unavailability, use health checks.
Failure recoveryNetwork partitions.Design for graceful degradation; fallback to eventual consistency if acceptable.
DeploymentVersion compatibility between services.Use semantic versioning, contract tests.

7. Common mistakes

MistakeWhy it’s problematic
Assuming a single‑node DB gives strong consistency across servicesA cache or another microservice may still read stale data.
Using READ UNCOMMITTED isolationAllows dirty reads; violates strong consistency.
Ignoring cache invalidationClients may read stale cache values after a write.
Over‑lockingLocking too many rows or using table locks reduces concurrency.
Not handling coordinator failuresLeads to deadlocks and stalled transactions.
Using eventual consistency for critical opsBusiness logic may rely on immediate visibility.
Failing to propagate transaction IDsMakes debugging distributed failures impossible.

8. Senior engineer perspective

What changes at scale?

  • Architecture – You may replace a hand‑rolled 2PC with a distributed DB (CockroachDB, YugabyteDB) or a consensus layer (Raft) to simplify logic.
  • Trade‑offs – You’ll weigh consistency vs latency; sometimes you’ll opt for read‑your‑writes with a local cache and eventual consistency for other reads.
  • Bottlenecks – The transaction coordinator becomes a single point of contention; horizontal scaling requires sharding or partitioning.
  • Failure scenarios – Network partitions can halt the entire system; you must design for graceful degradation or use split‑brain handling.
  • Monitoring – Track commit latency, abort rates, and lock wait times. Use distributed tracing to surface slow paths.
  • Cost/performance – Strong consistency often means more replicas, higher write amplification, and more expensive hardware. Evaluate ROI against business risk.
  • When NOT to use – For analytics pipelines, real‑time metrics, or high‑frequency trading where latency dominates, eventual consistency or a hybrid approach may be preferable.

9. Interview questions

1BeginnerWhat is strong consistency and how does it differ from eventual consistency?
Answer
Strong consistency guarantees that all reads after a write see the latest value (linearizability). Eventual consistency allows temporary divergence; all replicas converge eventually.
2BeginnerWhy might a distributed system choose eventual consistency over strong consistency?
Answer
To increase availability and reduce latency, especially during network partitions (CAP theorem).
3SeniorExplain how two‑phase commit ensures atomicity across multiple databases. What are its main failure points?
Answer
2PC has a prepare phase where participants vote, then a commit/abort phase. Failures: coordinator crash (deadlock), network partition (participants may be left in uncertain state), or participant crash after commit.
4SeniorDesign a system that needs to transfer money between accounts in a microservice architecture while guaranteeing strong consistency. Discuss trade‑offs and alternatives.
Answer
Use a distributed transaction coordinator, or a strongly consistent distributed DB. Alternatives: event sourcing with a single source of truth, or using a saga pattern with compensating actions (but sacrifices strong consistency).
5Scenario/DebuggingYou notice that after a transfer, some services still read the old balance for a few seconds. What could be causing this and how would you investigate?
Answer
Likely a cache stale issue (write‑through not flushed), or a read hitting a replica that hasn't caught up. Investigate by checking cache invalidation logic, replication lag metrics, and transaction commit logs.

10. Hands‑on task

Task: Add a “transfer” endpoint to the existing express app in this repository that:

  1. Accepts POST /transfer with JSON { fromId, toId, amount }.
  2. Uses a PostgreSQL transaction to debit fromId and credit toId.
  3. Updates a Redis cache (account:{id}) with the new balances.
  4. Guarantees strong consistency: if the cache update fails, the DB transaction must roll back.
  5. Logs the transaction ID and status.

Hints: Use pg for DB, ioredis for Redis, and uuid for transaction IDs. Wrap the DB operations in BEGIN/COMMIT/ROLLBACK. Use WATCH/MULTI/EXEC for Redis. Add error handling and retry logic.


11. Quick revision

  • Strong consistency = linearizability; reads see latest write.
  • CAP theorem: Consistency + Partition tolerance → Availability sacrificed.
  • 2PC: Prepare + Commit/Abort; coordinator‑driven.
  • Consensus (Raft/Paxos): Order operations across replicas.
  • Quorum: Majority must acknowledge for read/write.
  • Write‑through cache: Cache updated before acknowledging client.
  • Locking: SELECT … FOR UPDATE prevents concurrent updates.
  • Watch/Multi: Redis optimistic concurrency control.
  • Failure modes: Coordinator crash, network partition, lock contention.
  • Trade‑offs: Latency vs availability; complexity vs simplicity.
  • Monitoring: Commit latency, abort rates, replication lag.

12. References