Network Partitions

Backend Fundamentals · Distributed Systems · 8 min read

1. What problem does this solve?

Motivation
In a distributed system, nodes communicate over a network that can become unreliable. A network partition happens when a subset of nodes can no longer reach another subset. This is a real‑world problem: a fiber cut, a router failure, or a software bug can split your cluster into isolated islands.

What problem exists?

• Nodes lose visibility of each other’s state.
• Operations that require coordination (e.g., writes to a replicated database) may be blocked or duplicated.
• Clients may receive stale or inconsistent data.

Why the problem matters

• Users expect strong consistency or at least predictable behavior.
• In financial or health‑care systems, stale data can cause regulatory violations.
• A partition can cascade into a full outage if not handled correctly.

What happens if this concept is not used?

• Systems may crash or become unresponsive.
• Data corruption or loss can occur if writes are applied on both sides of a partition without reconciliation.
• Service level objectives (SLOs) are violated, leading to SLA penalties.

Where this concept appears in real backend systems

• Distributed databases (Cassandra, CockroachDB, MongoDB replica sets).
• Message brokers (Kafka, RabbitMQ).
• Service meshes (Istio, Linkerd).
• Multi‑region deployments of microservices behind load balancers.

2. Mental model

Analogy: A city with two neighborhoods separated by a broken bridge

• The bridge is the network link.
• Residents (nodes) can still talk within their neighborhood but cannot reach the other side.
• If a delivery truck (write) is sent to both sides, it may end up in two separate warehouses (duplicate data).
• The city must decide: keep the bridge open (partition tolerance) and accept that some deliveries may be duplicated, or close the bridge (avoid partition) and risk a traffic jam (availability loss).

Engineering model

• Partition tolerance: The system continues to operate despite network splits.
• Consistency vs. Availability: According to the CAP theorem, you must trade one for the other when a partition occurs.
• Conflict resolution: Strategies like last‑write‑wins, vector clocks, or application‑level merge are used to reconcile divergent states.

3. Deep explanation

Internals

• Heartbeat / Gossip protocols: Nodes periodically exchange liveness information.
• Consensus algorithms (Raft, Paxos): Ensure a single leader can commit logs even when some nodes are unreachable.
• Replication: Data is copied across nodes; during a partition, replicas may diverge.

Important terminology

Trade‑offs

Failure modes

• Split‑brain: Two leaders think they are the master, leading to divergent writes.
• Write starvation: Clients cannot write because quorum cannot be reached.
• Data loss: If a node crashes during a partition and its data is not replicated elsewhere.

Limitations

• No algorithm can guarantee both consistency and availability and partition tolerance simultaneously (CAP).
• Conflict resolution can be complex and application‑specific.
• Monitoring partitions requires sophisticated tooling (e.g., tracing, metrics).

Where this concept fits in a backend architecture

• Data layer: Distributed databases, caches, message queues.
• Service layer: Microservices that coordinate via RPC or event streams.
• Infrastructure layer: Load balancers, service meshes, network policies.

Interaction with related concepts

• Consensus vs. leader‑less replication: Raft vs. Dynamo.
• Idempotency: Helps mitigate duplicate writes during reconciliation.
• Circuit breakers: Prevent cascading failures when a partition is detected.
• Back‑pressure: Throttle clients when quorum cannot be reached.

4. How it works

1. Nodes start and join a cluster.
2. Heartbeat messages are exchanged every t seconds.
3. If a node misses k heartbeats from another, it marks that node as suspect.
4. After m additional heartbeats, the suspect node is considered dead.
5. Quorum calculation: For a write, at least floor(n/2)+1 nodes must acknowledge.
6. If quorum is not reachable, the write is blocked or queued (depending on policy).
7. When the partition heals, nodes exchange write logs.
8. Conflict resolution is applied (e.g., last‑write‑wins).
9. State is reconciled and all nodes converge to the same view.

sequenceDiagram
    participant Client
    participant NodeA
    participant NodeB
    participant NodeC

    Client->>NodeA: write(key, value)
    NodeA->>NodeB: replicate(key, value)
    NodeA->>NodeC: replicate(key, value)
    NodeB-->>NodeA: ack
    NodeC-->>NodeA: ack
    NodeA->>Client: success

    Note over NodeA,NodeB: Network partition occurs
    NodeA->>NodeB: replicate(key, newValue)
    NodeB--x>NodeA: timeout
    NodeA->>Client: blocked (no quorum)

5. Practical implementation

Below is a minimal Node.js example using Redis as a distributed key‑value store with Redlock (a distributed lock implementation) to illustrate handling partitions.

// src/partition-demo.ts
import { createClient } from 'redis';
import Redlock from 'redlock';

const redisUrls = [
  'redis://localhost:6379',   // Node 1
  'redis://localhost:6380',   // Node 2
  'redis://localhost:6381',   // Node 3
];

// Create a Redis client for each node
const clients = redisUrls.map(url => createClient({ url }));

// Connect all clients
await Promise.all(clients.map(c => c.connect()));

// Redlock requires an array of clients
const redlock = new Redlock(clients, {
  // Retry every 200ms
  retryDelay: 200,
  // Retry up to 10 times
  retryCount: 10,
});

async function writeWithLock(key: string, value: string) {
  try {
    // Acquire a lock on the key
    const lock = await redlock.acquire([`locks:${key}`], 1000);

    // Perform the write
    await clients[0].set(key, value);

    // Release the lock
    await lock.release();
    console.log(`Wrote ${key}=${value} successfully`);
  } catch (err) {
    console.error('Failed to acquire lock or write:', err);
  }
}

// Simulate a write
await writeWithLock('user:123', 'Alice');

Key points

• Redlock ensures that only one node can write to a key at a time, even across a partition.
• If a partition isolates a node, the lock acquisition will fail after the retry window, preventing a split‑brain write.
• The write is retried when the partition heals.

Tip: In production, use a dedicated Redis cluster or a managed service (e.g., AWS ElastiCache) that automatically handles failover and partition detection.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale?

• Cluster size: More nodes mean higher quorum thresholds; consider sharding or hierarchical quorum to keep latency low.
• Network topology: Multi‑region deployments introduce higher latency; use geo‑replication with read‑only replicas.
• Observability: Centralized tracing (Jaeger, Zipkin) and metrics (Prometheus) become essential.

Design decisions

• Consensus algorithm: Raft is easier to reason about; Paxos offers stronger guarantees but is more complex.
• Conflict resolution: Application‑specific merge functions vs. generic last‑write‑wins.
• Quorum strategy: Majority quorum vs. dynamic quorum (e.g., read‑only replicas).
• Failover handling: Automatic leader election vs. manual intervention.

Bottlenecks

• Network bandwidth: Replication traffic can saturate links.
• Disk I/O: Log persistence on every write.
• CPU: Consensus algorithms are CPU‑heavy.

Failure scenarios

• Split‑brain: Mitigated by split‑brain detection (e.g., using a shared lease).
• Leader loss: Rapid re‑election to avoid downtime.
• Data loss: Use write‑ahead logs and snapshots to recover.

Monitoring

• Heartbeat lag: Detect partitions early.
• Lock acquisition rates: High failure rates indicate contention.
• Replication lag: Ensure replicas catch up.

Cost/performance

• More replicas → higher cost but better availability.
• Sharding reduces per‑node load but adds complexity.

When NOT to use

• Small, single‑region services: Overkill to implement full consensus.
• Read‑heavy workloads: Use eventual consistency with read replicas.
• Stateless services: No need for partition tolerance at the data layer.

9. Interview questions

10. Hands‑on task

Task: Implement a simple distributed counter that uses Redis and Redlock to safely increment a value across multiple Node.js instances.

• Spin up three Redis containers (ports 6379, 6380, 6381).
• Write a script that starts three Node.js processes, each attempting to increment the counter 100 times.
• Verify that the final counter value equals 300 and that no duplicate increments occur even if you manually stop one Redis instance mid‑run.

Goal: Observe how Redlock prevents split‑brain writes and how the system recovers when a node is removed.

11. Quick revision

• A network partition is a loss of connectivity between subsets of nodes.
• CAP theorem: Partition tolerance + either consistency or availability.
• Quorum: Minimum nodes that must agree for an operation.
• Consensus algorithms (Raft, Paxos) elect leaders and ensure agreement.
• Heartbeat/gossip protocols detect node liveness.
• Conflict resolution: last‑write‑wins, vector clocks, application merge.
• Redlock provides distributed locking over Redis to avoid split‑brain writes.
• Observability: heartbeat lag, lock acquisition failures, replication lag.
• Scaling: shard data, use hierarchical quorum, monitor network latency.
• Common pitfalls: ignoring quorum, over‑tuning heartbeats, not handling lock failures.

12. References

• CAP Theorem – Wikipedia
• Raft Consensus Algorithm – The Raft Paper
• Redlock – Redis Distributed Lock
• Redis Sentinel – High Availability
• Kafka – Handling Partitions
• Istio – Service Mesh and Resilience