Failover

Backend Fundamentals · System Design · 9 min read

1. What problem does this solve?

Motivation
In a production backend you rarely have a single point of truth. Services, databases, caches, and message brokers all run on multiple machines or containers. If any of those components goes down, the whole system can become unusable. Failover is the mechanism that detects a failure and automatically switches traffic to a healthy replica, keeping the service available.

What problem exists

• A single instance of a database or cache can crash, become unreachable, or suffer from a hardware fault.
• Network partitions can isolate a node from the rest of the cluster.
• A service may become overloaded and start rejecting requests.

Why the problem matters

• Availability: Users expect 99.99 % uptime. A single outage can break that guarantee.
• Data integrity: If a primary database fails and no failover occurs, writes may be lost or become inconsistent.
• Business continuity: For e‑commerce, finance, or health‑care, downtime can mean lost revenue or safety risks.

What happens if this concept is not used

• The entire application may crash or become unresponsive.
• Users see errors or timeouts.
• Data may be lost or corrupted if the primary node fails without a backup.

Where this concept appears in real backend systems

• Database clusters (PostgreSQL streaming replication, MongoDB replica sets, MySQL Group Replication).
• Cache layers (Redis Sentinel, Amazon ElastiCache Multi‑AZ).
• Message brokers (Kafka controller election, RabbitMQ mirrored queues).
• Web servers behind load balancers (NGINX health checks, AWS ALB).
• Cloud services with built‑in failover (AWS RDS Multi‑AZ, GCP Cloud SQL HA).

2. Mental model

Analogy
Think of a city’s power grid. The main power plant (primary) supplies electricity. If it goes offline, a backup generator (secondary) kicks in automatically, so residents keep using power. The switch happens without anyone noticing.

Engineering model

• Primary: The node that currently serves traffic or holds the master copy of data.
• Replica/Secondary: A standby node that stays in sync with the primary.
• Health check: A probe that periodically verifies the node is alive and functional.
• Election: A deterministic process that decides which replica becomes the new primary when the current one fails.
• Failover: The act of redirecting traffic to the new primary and updating any routing or configuration.

3. Deep explanation

Internals

• Replication: Data is copied from primary to replicas. In PostgreSQL this is streaming replication; in Redis it’s asynchronous replication; in Kafka the controller replicates topic metadata.
• Heartbeat / Health checks: Simple TCP/HTTP pings or more sophisticated checks (e.g., query a known key).
• Leader election: Algorithms like Raft, Paxos, or simpler “first‑come‑first‑serve” based on timestamps or node IDs.
• Routing: Load balancers or DNS failover (e.g., Route 53 health checks) redirect traffic.

Important terminology

• Primary / Master: The node that accepts writes.
• Replica / Slave: Nodes that copy data from the primary.
• Quorum: Minimum number of nodes that must agree to elect a leader.
• Split‑brain: When two replicas think they are primary simultaneously, leading to data divergence.
• Read‑only replica: A replica that only serves read traffic.

Trade‑offs

Failure modes

• Network partition: A node can be isolated but still think it’s connected.
• Hardware failure: Disk crash, memory corruption.
• Software bug: Deadlock, memory leak causing unresponsiveness.
• Mis‑configuration: Wrong replication slots, missing quorum.

Limitations

• Latency: Replication lag can cause stale reads.
• Consistency: In asynchronous setups, writes may be lost if the primary dies before replication.
• Complexity: Managing elections, backups, and monitoring adds operational overhead.

Where this concept fits in a backend architecture

┌───────────────────────┐
│  Client / Frontend    │
└────────────┬──────────┘
             │
             ▼
   ┌───────────────────────┐
   │  Load Balancer / LB   │
   └───────┬───────┬───────┘
           │       │
           ▼       ▼
   ┌───────────────┐   ┌───────────────┐
   │  Service A    │   │  Service B    │
   └───────┬───────┘   └───────┬───────┘
           │               │
           ▼               ▼
   ┌───────────────────────┐
   │  Database Cluster     │
   │  (Primary + Replicas) │
   └───────────────────────┘

Failover sits between the load balancer and the database cluster, ensuring the LB always points to a healthy primary.

Interaction with related concepts

• Load balancing: LB health checks trigger failover.
• Caching: Redis Sentinel can promote a replica to master; clients must reconnect.
• CI/CD: Deployment pipelines must preserve replication topology.
• Observability: Metrics (replication lag, election time) inform alerting.

4. How it works

1. Health check: Load balancer or monitoring system pings the primary every N seconds.
2. Detection: If the primary fails to respond or returns an error, the health check marks it unhealthy.
3. Election: Replicas run an election algorithm (e.g., Raft) to pick a new primary.
4. Promotion: The elected replica promotes itself to primary (writes enabled).
5. Routing update: Load balancer updates its routing table to point to the new primary.
6. Client reconnection: Clients automatically retry or reconnect to the new primary.
7. Re‑sync: The old primary (if it comes back) is demoted to replica and re‑synchronizes.

sequenceDiagram
    participant LB as Load Balancer
    participant P as Primary DB
    participant R1 as Replica 1
    participant R2 as Replica 2
    participant Client

    Client->>LB: Request
    LB->>P: Forward
    P->>Client: Response

    Note over LB,P: Primary crashes

    LB->>R1: Health Check
    R1-->>LB: Unhealthy
    LB->>R2: Health Check
    R2-->>LB: Healthy

    R2->>R2: Run election
    R2-->>R2: Promote to Primary

    LB->>R2: Update routing
    Client->>LB: Request
    LB->>R2: Forward
    R2->>Client: Response

5. Practical implementation

Below is a minimal Node.js example using Redis Sentinel for cache failover.
The app connects to Redis via the ioredis client, which automatically follows Sentinel.

// redisClient.ts
import Redis from 'ioredis';

const sentinelNodes = [
  { host: 'sentinel-1.local', port: 26379 },
  { host: 'sentinel-2.local', port: 26379 },
  { host: 'sentinel-3.local', port: 26379 },
];

const redis = new Redis({
  sentinels: sentinelNodes,
  name: 'mymaster',          // name of the master in Sentinel config
  // Optional: retry strategy, timeouts, etc.
});

redis.on('connect', () => console.log('Connected to Redis via Sentinel'));
redis.on('error', (err) => console.error('Redis error:', err));

export default redis;

// server.ts
import express from 'express';
import redis from './redisClient';

const app = express();
const PORT = 3000;

// Simple key/value endpoint
app.get('/cache/:key', async (req, res) => {
  try {
    const value = await redis.get(req.params.key);
    if (value === null) {
      return res.status(404).send('Not found');
    }
    res.send(value);
  } catch (err) {
    console.error(err);
    res.status(500).send('Redis error');
  }
});

app.listen(PORT, () => console.log(`Server listening on ${PORT}`));

How it works

1. The ioredis client connects to the Sentinel cluster.
2. Sentinel monitors the master and replicas; if the master fails, it promotes a replica.
3. The client automatically reconnects to the new master without code changes.

Tip: In production, add retryStrategy, maxRetriesPerRequest, and proper error handling to avoid silent failures.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

Architecture decisions

• Replication strategy: Synchronous vs asynchronous based on consistency requirements.
• Quorum size: For a 3‑node cluster, a 2‑node quorum prevents split‑brain.
• Geographic distribution: Multi‑AZ or multi‑region for disaster recovery vs latency trade‑off.
• Load balancer choice: DNS failover vs L7 LB with health checks.

Trade‑offs

• Latency vs consistency: Synchronous replication increases write latency but guarantees no data loss.
• Cost vs availability: More replicas mean higher infrastructure cost but higher uptime.

Bottlenecks

• Network bandwidth: Replication traffic can saturate links.
• Disk I/O: Writing WALs to disk on the primary can become a bottleneck.
• Election latency: Slow elections delay failover.

Failure scenarios

• Network partition: Use synchronous_commit=remote_write to mitigate.
• Replica overload: Monitor CPU/memory; scale replicas horizontally.

Monitoring

• Replication lag (pg_stat_replication in PostgreSQL).
• Sentinel metrics (redis_master_last_io_seconds_ago).
• Health check logs from load balancers.

Cost/performance considerations

• Read replicas reduce load on primary but add replication cost.
• Multi‑region increases latency but improves disaster recovery.

When NOT to use the approach

• Stateless services: No need for database failover.
• Low‑traffic apps: A single instance may suffice; cost of replicas outweighs benefit.
• Highly mutable data with strict consistency: Consider synchronous replication or a different consistency model.

9. Interview questions

10. Hands‑on task

Task: Add automatic failover to a local Redis setup using Sentinel.

1. Spin up three Redis containers: one master, two replicas.
2. Configure Sentinel on each replica to monitor the master.
3. Write a Node.js script that connects via ioredis and sets a key.
4. Simulate a master failure (stop the master container).
5. Verify that the script automatically reconnects to the promoted replica and can still set/get keys.

Hints: Use Docker Compose for orchestration; expose Sentinel ports; set name: mymaster in Sentinel config.

11. Quick revision

• Failover switches traffic from a failed primary to a healthy replica.
• Replication keeps replicas in sync; asynchronous = lower latency, possible data loss.
• Health checks (TCP/HTTP) trigger failover.
• Leader election (Raft/Paxos) selects new primary.
• Split‑brain occurs if quorum is not enforced.
• Load balancers must update routing on failover.
• Monitoring replication lag is essential.
• Security: TLS, auth, and proper ACLs on replicas.
• Testing failover in staging prevents production surprises.
• Synchronous replication guarantees consistency but adds latency.
• Use ioredis or pg with Sentinel/replication support for automatic failover.

12. References

• ioredis Sentinel documentation
• PostgreSQL Streaming Replication
• Redis Sentinel
• Kafka Controller Election
• AWS RDS Multi‑AZ
• Prometheus Redis Exporter
• Patroni – PostgreSQL HA
• pg_auto_failover – PostgreSQL failover tool