Availability

Backend Fundamentals · System Design · 7 min read

1. What problem does this solve?

• Problem: A single server or component can fail (hardware crash, software bug, network outage). If the system is not designed to tolerate such failures, users experience downtime.
• Why it matters: Modern web services expect continuous operation. Even a few minutes of outage can cost revenue, damage reputation, and violate SLAs.
• Consequences of ignoring availability:
  • Users see error pages or timeouts.
  • Automated monitoring alerts trigger, causing firefighting.
  • Data loss or corruption if writes are lost during a crash.
• Real‑world appearance:
  • E‑commerce: checkout must stay online during traffic spikes.
  • Social media: feeds must keep delivering even if a data center goes down.
  • Financial services: trading platforms require 99.999% uptime.

2. Mental model

Analogy: A city’s power grid

• Single power plant → one point of failure.
• Multiple plants + automatic switching → if one plant fails, the grid keeps running.
• Load balancer is like the grid’s switchboard, routing power (requests) to healthy plants (servers).
• Health checks are the sensors that detect a plant’s status.

Engineering model

• Redundancy: duplicate components (servers, databases).
• Failover: automatic switch to a healthy replica when one fails.
• Health checks: periodic probes that inform the load balancer which instances are alive.
• Graceful degradation: if a non‑critical component fails, the system still serves core functionality.

3. Deep explanation

4. How it works

1. Client sends request → to DNS → resolves to load balancer IP.
2. Load balancer receives request, checks its routing table.
3. Health check: LB periodically queries /health on each backend.
4. Routing: LB forwards request to a healthy instance.
5. Backend processes request, may read/write to replicated DB.
6. Response travels back through LB to client.
7. If a backend fails: LB stops routing to it until health check passes again.

sequenceDiagram
    participant Client
    participant LB as Load Balancer
    participant Service1
    participant Service2
    participant DB1
    participant DB2

    Client->>LB: GET /api/data
    LB->>Service1: Forward request
    Service1->>DB1: SELECT * FROM users
    DB1-->>Service1: Result
    Service1-->>LB: 200 OK
    LB-->>Client: 200 OK

    Note over LB: Health check fails on Service1
    LB->>Service2: Forward request

5. Practical implementation

5.1 Node.js + Express + NGINX load balancer

app.ts

import express from 'express';
const app = express();
const PORT = process.env.PORT || 3000;

// Simple health endpoint
app.get('/health', (_req, res) => {
  // In production, check DB, cache, etc.
  res.status(200).send('OK');
});

// Core API
app.get('/api/users', async (req, res) => {
  // Simulate DB read
  const users = await getUsersFromDB();
  res.json(users);
});

app.listen(PORT, () => {
  console.log(`Server listening on ${PORT}`);
});

nginx.conf

events {}
http {
  upstream backend {
    server 127.0.0.1:3000 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3001 max_fails=3 fail_timeout=30s;
  }

  server {
    listen 80;

    location / {
      proxy_pass http://backend;
    }

    # Health check endpoint
    location /health {
      proxy_pass http://backend;
    }
  }
}

• Run two Node instances on ports 3000 & 3001.
• NGINX will stop sending traffic to an instance that fails 3 times in 30 s.
• Health checks are performed automatically by NGINX.

5.2 Redis Sentinel for HA

import { createClient } from 'redis';

const sentinel = createClient({
  sentinels: [
    { host: 'redis-1.example.com', port: 26379 },
    { host: 'redis-2.example.com', port: 26379 }
  ],
  name: 'mymaster'
});

sentinel.on('error', err => console.error('Redis error:', err));

async function get(key: string) {
  return await sentinel.get(key);
}

• Sentinel monitors the master and promotes a replica if the master fails.
• The client automatically reconnects to the new master.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

• Architecture decisions:
  • Multi‑region vs. single‑region HA.
  • Consistency model: eventual vs. strong.
  • Use of distributed consensus (Raft) for leader election.
• Trade‑offs:
  • Higher availability → higher cost (more nodes, cross‑region traffic).
  • Strong consistency → lower availability under partitions.
• Bottlenecks:
  • Load balancer capacity.
  • Database write throughput.
  • Network latency between regions.
• Failure scenarios:
  • Network partition: use quorum to decide.
  • Data center outage: rely on cross‑region replication.
• Monitoring:
  • Uptime dashboards, replication lag alerts, health‑check metrics.
• Cost/performance:
  • Evaluate cost of extra replicas vs. SLA penalties.
• When NOT to use:
  • For low‑traffic internal tools where 99.9% uptime is acceptable.
  • When consistency is more critical than availability (e.g., banking).

9. Interview questions

10. Hands‑on task

Goal: Build a simple two‑instance Node.js service behind NGINX with health checks, then simulate a failure.

1. Clone the repo and run npm install.
2. Start two Node instances: node app.ts --port=3000 and node app.ts --port=3001.
3. Start NGINX with the provided nginx.conf.
4. Verify that /api/users works from a browser or curl.
5. Stop one Node instance and observe that NGINX stops routing to it after 3 failures.
6. Add a /health endpoint that randomly fails 30 % of the time and observe how NGINX reacts.
7. Document the observed behavior and explain why NGINX stops sending traffic.

11. Quick revision

• Availability = uptime; goal is 99.99%+ for critical services.
• Redundancy + failover = core of HA.
• Load balancer routes traffic; health checks inform routing.
• Replicated databases (PostgreSQL streaming, MongoDB replica set, Redis Sentinel).
• CAP theorem: Availability vs. Consistency under partitions.
• Circuit breakers prevent cascading failures.
• Monitoring: uptime, health‑check metrics, replication lag.
• Common pitfalls: no health checks, single LB, heavy health checks.
• Senior decisions: multi‑region, quorum size, cost vs. SLA.

12. References

• Node.js – HTTP module
• Express – Health check middleware
• NGINX – Upstream health checks
• Redis Sentinel
• CAP Theorem
• AWS ELB – High Availability
• Kubernetes – Pod Disruption Budgets