Failover

Backend Fundamentals · Reliability Engineering · 8 min read

1. What problem does this solve?

Motivation
In a distributed backend, a single point of failure (SPOF) can bring the whole service down. Failover is the mechanism that detects a failure and automatically switches traffic to a healthy replica, keeping the system available.

What problem exists

• A database node crashes, a cache instance goes offline, or a microservice instance dies.
• Clients experience timeouts or errors, leading to poor user experience and SLA violations.

Why the problem matters

• Availability: Users expect 99.99 % uptime. Even a few seconds of outage can erode trust.
• Business continuity: E‑commerce, payments, or real‑time analytics cannot tolerate downtime.
• Regulatory compliance: Some industries mandate high availability and disaster recovery.

What happens if this concept is not used

• Manual intervention is required to restore service.
• Downtime increases, leading to revenue loss and potential penalties.
• Scaling becomes difficult because you cannot rely on automatic recovery.

Where this concept appears in real backend systems

• Database clusters (PostgreSQL streaming replication, MongoDB replica sets).
• Cache layers (Redis Sentinel, Amazon ElastiCache).
• Load balancers (NGINX, HAProxy, AWS ELB).
• Service meshes (Istio, Linkerd) with circuit breakers.
• Cloud provider managed services (RDS Multi-AZ, Cloud SQL HA).

2. Mental model

Analogy
Think of a city’s water supply. There’s a main pipeline (primary) that delivers water to homes. If the main pipeline bursts, a backup pipeline (secondary) automatically takes over, ensuring residents still get water. The switch happens without the residents noticing.

Engineering model

• Primary: The main instance that receives traffic.
• Secondary: A standby instance that stays in sync (replication, health checks).
• Failover trigger: Health check failure, heartbeat loss, or explicit promotion.
• Failover action: Route traffic to secondary, promote it to primary, optionally demote the old primary.

3. Deep explanation

Internals

• Health checks: Periodic probes (HTTP, TCP, custom) that report status to a load balancer or orchestrator.
• Replication: Data is asynchronously or synchronously replicated from primary to secondary (e.g., PostgreSQL streaming replication, Redis replication).
• Promotion: The secondary becomes primary by changing its role and updating routing rules.
• State synchronization: For stateful services, ensure the secondary has a consistent snapshot (e.g., using WAL replay).

Important terminology

Trade‑offs

Failure modes

• Network partition: Nodes lose connectivity; need to avoid split‑brain.
• Hardware failure: Disk crash, power loss.
• Software bugs: Deadlocks, memory leaks causing unresponsiveness.
• Configuration errors: Wrong replication slots, missing failover scripts.

Limitations

• Latency: Failover can introduce a brief outage (seconds to minutes).
• Consistency: In asynchronous setups, data may be lost or out‑of‑sync.
• Cost: Running standby instances increases infrastructure spend.
• Complexity: Requires careful orchestration and monitoring.

Where this concept fits in a backend architecture

• Database layer: PostgreSQL HA, MongoDB replica sets.
• Cache layer: Redis Sentinel, Amazon ElastiCache.
• Service layer: Load balancers, Kubernetes readiness probes.
• Infrastructure layer: Cloud provider managed failover (RDS Multi‑AZ).

Interaction with related concepts

• Load balancing: Failover often relies on a load balancer to redirect traffic.
• Circuit breakers: Detect failures at the application level before failover.
• Health checks: The heartbeat that triggers failover.
• Disaster recovery: Failover is a subset of DR; DR includes backup restoration.

4. How it works

Step‑by‑step flow

1. Primary receives traffic
   Clients send requests to the primary node.

2. Health checks run
   A monitoring system (e.g., Kubernetes liveness probe, custom script) pings the primary every N seconds.

3. Failure detected
   If the primary fails to respond or returns an error, the health check marks it as unhealthy.

4. Failover trigger
   The orchestrator (e.g., Kubernetes controller, HAProxy) initiates failover.

5. Promotion
   The secondary is promoted to primary (e.g., pg_ctl promote, redis-cli SLAVEOF NO ONE).

6. Routing update
   Traffic is redirected to the new primary (e.g., updating DNS, load balancer backend pool).

7. Recovery
   The original primary is repaired or replaced and re‑synchronized as a secondary.

8. Failback (optional)
   Once the original primary is healthy, traffic can be switched back.

sequenceDiagram
    participant Client
    participant LB as Load Balancer
    participant Primary
    participant Secondary
    participant Monitor

    Client->>LB: Request
    LB->>Primary: Forward
    Primary->>Client: Response

    Monitor->>Primary: Health Check
    alt Primary fails
        Monitor->>LB: Mark unhealthy
        LB->>Secondary: Promote
        LB->>Client: Redirect
        Secondary->>Client: Response
    else
        Monitor->>Primary: Healthy
    end

5. Practical implementation

Below is a minimal Node.js example using Redis Sentinel for failover.
The app connects to Redis via the sentinel client, which automatically switches to the master when a failover occurs.

// redisClient.ts
import { createClient } from 'redis';

// Sentinel configuration
const sentinelHosts = [
  { host: 'sentinel-1.example.com', port: 26379 },
  { host: 'sentinel-2.example.com', port: 26379 },
  { host: 'sentinel-3.example.com', port: 26379 },
];

// Create a Redis client that talks to Sentinel
const client = createClient({
  // Sentinel mode
  socket: {
    // Sentinel addresses
    host: sentinelHosts[0].host,
    port: sentinelHosts[0].port,
  },
  // Sentinel options
  sentinel: {
    // Name of the master as configured in Sentinel
    name: 'mymaster',
    // Optional: list of all sentinel nodes
    sentinels: sentinelHosts,
  },
});

client.on('error', (err) => console.error('Redis error:', err));

export default client;

// app.ts
import express from 'express';
import redisClient from './redisClient';

const app = express();
const PORT = process.env.PORT || 3000;

// Simple counter endpoint
app.get('/counter', async (req, res) => {
  try {
    // Increment counter atomically
    const count = await redisClient.incr('global:counter');
    res.json({ count });
  } catch (err) {
    console.error('Failed to increment counter', err);
    res.status(500).json({ error: 'Internal server error' });
  }
});

app.listen(PORT, () => {
  console.log(`Server listening on ${PORT}`);
});

Key points

• redisClient automatically follows the master reported by Sentinel.
• If the master fails, Sentinel promotes a replica and updates the client.
• No code changes are required for the application to handle failover.

Running locally

1. Spin up a Redis cluster with Sentinel (Docker compose example).
2. Start the Node.js app.
3. Simulate a failover by shutting down the master container.
4. Observe that the app continues to increment the counter via the new master.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale?

• Architecture decisions

  • Choose between Active‑Standby vs Active‑Active based on consistency requirements.
  • Decide on synchronous vs asynchronous replication.
  • Select a distributed consensus protocol (Raft, Paxos) for leader election.

• Trade‑offs

  • Latency vs consistency: Synchronous replication guarantees no data loss but increases write latency.
  • Cost vs availability: More replicas increase cost but improve fault tolerance.

• Bottlenecks

  • Replication bandwidth: High write throughput can saturate network.
  • Promotion latency: Slow promotion can lead to extended downtime.

• Failure scenarios

  • Network partition: Must detect majority and avoid split‑brain.
  • Simultaneous node failures: Design for multi‑AZ or multi‑region failover.

• Monitoring

  • Track replication lag, promotion events, failover frequency.
  • Set alerts for unusual lag spikes or repeated failovers.

• Cost/performance

  • Use spot instances for standby nodes if acceptable.
  • Employ auto‑scaling to spin up replicas only during peak load.

• When NOT to use

  • For low‑traffic, non‑critical services where manual failover is acceptable.
  • When data consistency is paramount and you cannot tolerate replication lag.
  • In single‑tenant environments where a single node suffices.

9. Interview questions

10. Hands‑on task

Task: Implement a simple HTTP health‑check endpoint for a Node.js service that uses PostgreSQL. The endpoint should return 200 OK only if the database connection is healthy. Then, configure a Docker Compose setup with two PostgreSQL instances (primary + replica) and a Node.js app. Simulate a primary failure and observe the health‑check response.

Steps

1. Add /health route that runs SELECT 1 on the DB.
2. Use pg library with connection pooling.
3. In docker-compose.yml, set up PostgreSQL with streaming replication.
4. Use a script to kill the primary container.
5. Verify the health endpoint still returns 200 after failover.

11. Quick revision

• Failover switches traffic from a failed primary to a healthy secondary.
• Health checks (TCP/HTTP) trigger failover.
• Replication can be synchronous (no loss) or asynchronous (possible lag).
• Sentinel, Patroni, and cloud‑managed services automate failover.
• Promotion updates routing (DNS, load balancer).
• Split‑brain is avoided by quorum or consensus protocols.
• Monitoring replication lag and failover events is critical.
• Cost increases with standby nodes; balance with SLA requirements.
• Idempotent operations help during transient failovers.
• Test failover in staging before production.

12. References

• Redis Sentinel Documentation
• PostgreSQL Streaming Replication
• Patroni – PostgreSQL High Availability
• Kubernetes Liveness and Readiness Probes
• AWS RDS Multi‑AZ Failover
• Prometheus Alerting Rules for Failover
• Raft Consensus Algorithm
• Synchronous vs Asynchronous Replication Trade‑offs