PACELC

Backend Fundamentals · System Design · 9 min read

1. What problem does this solve?

In a distributed system, data is spread across multiple nodes. When a network partition occurs (some nodes become unreachable), the system must decide how to behave.
The PACELC rule formalizes the trade‑offs that arise both during a partition and during normal operation.

• Problem: How to balance consistency, availability, partition tolerance, latency, and consistency when the network is healthy.
• Why it matters: A wrong balance can lead to stale reads, lost writes, or service outages—critical for user‑facing APIs, financial systems, or real‑time analytics.
• Consequences of ignoring PACELC:
  • Too much consistency: users see stale data or experience high latency.
  • Too much availability: data diverges and later requires costly reconciliation.
  • Ignoring latency: users get frustrated, and the system may be throttled by clients.
• Real‑world appearance:
  • Cassandra (AP + L): favors availability and low latency, sacrificing consistency.
  • PostgreSQL with synchronous replication (CP + C): favors consistency and partition tolerance, sacrificing availability.
  • MongoDB with read/write concerns (P + A + C + L): configurable trade‑offs per operation.

2. Mental model

Analogy – The Restaurant Order System

Engineering model

• Partition (P): Network split.
• Availability (A): Service continues to respond.
• Consistency (C): All nodes see the same data.
• Latency (L): Time to complete a request.
• Elasticity (E): When no partition, you still choose between Latency and Consistency.

PACELC says:

During a partition: choose between P + A vs. P + C.
During normal operation: choose between E + L vs. E + C.

3. Deep explanation

Internals

1. Replication: Data is copied across nodes.
2. Quorum: A majority of replicas must agree for a write to succeed.
3. Read/Write concerns: Clients specify how many replicas must respond.
4. Conflict resolution: When replicas diverge, a strategy (last‑write‑wins, vector clocks, CRDTs) resolves differences.

Important terminology

Trade‑offs

Failure modes

• Split‑brain: Two partitions both think they are primary → data divergence.
• Write‑skew: Concurrent writes on different nodes lead to inconsistent state.
• Read‑after‑write anomalies: Client reads stale data immediately after a write.

Limitations

• PACELC is a model, not a prescription. Real systems use hybrid strategies (e.g., MongoDB’s read/write concerns).
• It assumes binary choices; many systems offer graded consistency (e.g., read‑majority).
• It does not cover durability or security trade‑offs.

Where it fits

• Data layer: Replication protocols, sharding, and consistency settings.
• API layer: Choosing read/write concerns per endpoint.
• Infrastructure: Network design, load balancers, and failover mechanisms.

Interaction with related concepts

• Event sourcing: Guarantees eventual consistency via event logs.
• CQRS: Separates read/write models, often tuning each side differently.
• Cache invalidation: Affects consistency vs. latency trade‑offs.
• Microservices: Each service may adopt its own PACELC balance.

4. How it works

sequenceDiagram
    participant Client
    participant API
    participant Replicas
    participant DB

    Client->>API: Write request
    API->>Replicas: Replicate to 3 nodes
    Replicas->>Replicas: Quorum check (2/3)
    alt Quorum achieved
        Replicas->>DB: Persist
        DB-->>Replicas: Ack
        Replicas-->>API: Ack
        API-->>Client: Success
    else Quorum failed
        API-->>Client: Error (Unavailable)
    end

    Client->>API: Read request
    API->>Replicas: Read from 2 nodes
    alt Read majority
        Replicas-->>API: Data
        API-->>Client: Data
    else Read single
        Replicas-->>API: Data
        API-->>Client: Data
    end

• During partition: If two replicas are unreachable, the system may still respond (A) or refuse (C).
• During normal operation: Reads can be served from a single replica (low L) or from a majority (higher C).

5. Practical implementation

Below is a minimal Node.js example using Redis Cluster for eventual consistency and PostgreSQL for strong consistency. The API exposes two endpoints: one that favors availability/latency, another that enforces consistency.

// src/server.ts
import express from 'express';
import { createClient } from 'redis';
import { Pool } from 'pg';

const app = express();
app.use(express.json());

// Redis cluster client (eventual consistency)
const redis = createClient({ url: 'redis://localhost:7000' });
redis.on('error', err => console.error('Redis error', err));

// PostgreSQL client (strong consistency)
const pgPool = new Pool({
  connectionString: process.env.DATABASE_URL,
});

// ---------- 1️⃣ Availability + Low Latency ----------
app.post('/cache', async (req, res) => {
  const { key, value } = req.body;
  try {
    // Fire‑and‑forget write to Redis
    redis.set(key, value);
    // Also persist to Postgres asynchronously
    pgPool.query('INSERT INTO kv(key, value) VALUES($1,$2) ON CONFLICT(key) DO UPDATE SET value=$2', [key, value]);
    res.status(202).send({ status: 'queued' });
  } catch (e) {
    res.status(500).send({ error: e.message });
  }
});

app.get('/cache/:key', async (req, res) => {
  const key = req.params.key;
  try {
    // Read from Redis first (low latency)
    const cached = await redis.get(key);
    if (cached !== null) return res.send({ value: cached, source: 'redis' });

    // Fallback to Postgres (strong consistency)
    const { rows } = await pgPool.query('SELECT value FROM kv WHERE key=$1', [key]);
    if (rows.length) return res.send({ value: rows[0].value, source: 'postgres' });

    res.status(404).send({ error: 'Not found' });
  } catch (e) {
    res.status(500).send({ error: e.message });
  }
});

// ---------- 2️⃣ Consistency + Partition Tolerance ----------
app.post('/kv', async (req, res) => {
  const { key, value } = req.body;
  try {
    // Wait for Postgres commit (strong consistency)
    await pgPool.query('INSERT INTO kv(key, value) VALUES($1,$2) ON CONFLICT(key) DO UPDATE SET value=$2', [key, value]);

    // Replicate to Redis synchronously
    await redis.set(key, value);

    res.send({ status: 'ok' });
  } catch (e) {
    res.status(500).send({ error: e.message });
  }
});

app.get('/kv/:key', async (req, res) => {
  const key = req.params.key;
  try {
    const { rows } = await pgPool.query('SELECT value FROM kv WHERE key=$1', [key]);
    if (!rows.length) return res.status(404).send({ error: 'Not found' });

    // Ensure Redis is up‑to‑date
    await redis.set(key, rows[0].value);

    res.send({ value: rows[0].value });
  } catch (e) {
    res.status(500).send({ error: e.message });
  }
});

app.listen(3000, () => console.log('Server listening on :3000'));

Key points

• /cache endpoint: A + L – writes are queued, reads are fast.
• /kv endpoint: C + P – writes wait for PostgreSQL, reads are strongly consistent.
• The same data is stored in both stores; the choice of endpoint dictates the PACELC trade‑off.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

Architecture decisions

• Choose a primary store (PostgreSQL, Cassandra, etc.) based on consistency needs.
• Add a cache layer (Redis, Memcached) for low‑latency reads.
• Define per‑endpoint consistency: e.g., /orders uses strong consistency; /feed uses eventual.
• Implement conflict resolution: CRDTs for counters, vector clocks for documents.

Trade‑offs

• Cost vs. consistency: Strong consistency requires more replicas, higher latency, and more network traffic.
• Operational complexity: Multiple consistency levels mean more configuration and monitoring.
• Latency budgets: For real‑time dashboards, you may accept eventual consistency; for payment APIs, you cannot.

Bottlenecks

• Primary node: All writes must reach it; becomes a choke point.
• Quorum calculation: With many replicas, quorum checks can become expensive.
• Network: Partition detection and reconnection can delay writes.

Failure scenarios

• Split‑brain: Mitigate with anti‑entropy protocols and read/write concerns that enforce majority.
• Replica lag: Detect with metrics; trigger alerts if lag exceeds threshold.
• Node failure: Ensure automatic failover and re‑replication.

Monitoring

• Quorum success rate per operation.
• Read/write latency split by consistency level.
• Replica lag (time difference between primary and followers).
• Partition detection (e.g., via heartbeat failures).

Cost/performance

• More replicas → higher storage and network costs but better availability.
• Strong consistency → higher CPU and I/O due to coordination.
• Caching → reduces load on primary but adds memory cost.

When NOT to use

• Simple CRUD services where eventual consistency is acceptable and you want minimal latency.
• Highly regulated domains (HIPAA, PCI) that require strict audit trails and strong consistency.
• Systems with very low write volume where the overhead of strong consistency outweighs benefits.

9. Interview questions

10. Hands‑on task

Task: Extend the /cache endpoint to support a read‑majority mode that reads from at least 2 Redis replicas before returning data.

• Add a query parameter ?consistency=majority.
• If the majority cannot be reached within 200 ms, return a 503 Service Unavailable.
• Log the latency and success/failure of the majority read.

Hints:

• Use ioredis cluster client’s nodes('master') to get replica addresses.
• Use Promise.allSettled to wait for responses.
• Measure time with process.hrtime().

11. Quick revision

• PACELC extends CAP: Partition + Availability + Consistency; Elasticity + Latency + Consistency.
• During partition: choose A (available) or C (consistent).
• During normal operation: choose L (low latency) or C (strong consistency).
• Read/write concerns (quorum) control the trade‑off.
• Eventual consistency → fast, but may return stale data.
• Strong consistency → safe, but higher latency and possible downtime.
• Conflict resolution: CRDTs, vector clocks, last‑write‑wins.
• Monitoring: quorum success rate, replica lag, latency per consistency level.
• Common pitfalls: assuming eventual consistency is always fine; ignoring partition testing.
• Senior‑level: design per‑endpoint consistency, balance cost vs. safety, implement reconciliation.

12. References

• CAP Theorem – Wikipedia
• PACELC – Wikipedia
• Redis Cluster – Official Docs
• PostgreSQL Replication – Official Docs
• ioredis – GitHub
• CRDTs – Wikipedia
• Kafka – Official Docs