Health checks

Backend Fundamentals · Reliability Engineering · 9 min read

1. What problem does this solve?

Health checks are lightweight probes that let a monitoring system or orchestrator (Kubernetes, Docker‑Compose, load balancer, etc.) know whether a service is alive and ready to serve traffic.
They solve several real‑world problems:

Without health checks, you risk:

• Silent failures: a process is running but cannot serve requests.
• Unnecessary traffic: load balancers keep sending requests to dead nodes.
• Delayed recovery: operators may not notice a problem until a user reports it.

Health checks appear everywhere:

• Kubernetes (livenessProbe, readinessProbe)
• Docker‑Compose (healthcheck in docker-compose.yml)
• AWS ELB/ALB (target group health checks)
• Prometheus (scrape /health endpoints)
• CI/CD pipelines (post‑deployment checks)

2. Mental model

Analogy

Think of a health check like a doctor’s check‑up for a server.

• Vitals (CPU, memory, DB connectivity) are the physical exam.
• Lab tests (ping a database, run a simple query) are the blood tests.
• The doctor returns a report (200 OK or 503 Service Unavailable).
  If the doctor says “all good”, the patient (service) can go out and work. If not, the patient stays home.

Engineering model

A health check is a self‑contained HTTP endpoint (or other protocol) that:

1. Runs quickly – < 100 ms.
2. Does minimal work – only what is needed to decide “alive” or “ready”.
3. Returns a clear status – 200 for healthy, 5xx for unhealthy.
4. Is idempotent – repeated calls give the same result.

3. Deep explanation

Internals

• Liveness: Does the process exist and is it not stuck?
  Implementation: simple /health that returns 200 if the Node.js event loop is running.
• Readiness: Are all critical dependencies available?
  Implementation: /ready that pings DB, cache, external APIs.
• Probes: In Kubernetes, livenessProbe restarts the pod if it fails; readinessProbe removes it from service endpoints.

Important terminology

• Probe – a check performed by an orchestrator.
• Endpoint – the URL exposed by the service.
• Status code – 200 OK (healthy), 503 Service Unavailable (unhealthy).
• Timeout / interval – how often and how long the probe waits.

Trade‑offs

Failure modes

• False positives: A check passes but the service is still unresponsive (e.g., DB ping succeeds but query times out).
• False negatives: A transient network glitch causes a check to fail.
• Resource exhaustion: Health checks that run heavy queries can degrade performance.

Limitations

• Health checks only detect current state; they cannot predict future failures.
• They cannot replace proper error handling inside your application.
• Over‑aggressive checks can cause unnecessary restarts.

Where it fits

• Service layer: Exposed as /health and /ready.
• Orchestration layer: Kubernetes probes, Docker healthcheck, load balancer health checks.
• Observability layer: Prometheus scrapes health endpoints; logs can be correlated.

Interaction with related concepts

• Circuit breakers: Health checks can be used to reset a circuit breaker.
• Graceful shutdown: A readiness probe can be set to false before shutting down to stop new traffic.
• Rate limiting: Health checks should bypass rate limits to avoid false negatives.

4. How it works

1. Service starts → registers /health and /ready.
2. Orchestrator polls /health at interval.
3. If /health returns 200 → service is alive.
4. Orchestrator polls /ready at interval.
5. If /ready returns 200 → service is ready to receive traffic.
6. If any probe fails → orchestrator takes action (restart pod, remove from load balancer).

sequenceDiagram
    participant Client
    participant LoadBalancer
    participant Service
    participant Orchestrator

    Orchestrator->>Service: GET /health
    Service-->>Orchestrator: 200 OK
    Orchestrator->>Service: GET /ready
    Service-->>Orchestrator: 200 OK
    LoadBalancer->>Client: GET /api
    Service-->>Client: 200 OK

5. Practical implementation

Node.js / Express example

// src/health.ts
import express, { Request, Response } from 'express';
import { Pool } from 'pg';
import { createClient } from 'redis';

const app = express();
const dbPool = new Pool({ connectionString: process.env.DATABASE_URL });
const redis = createClient({ url: process.env.REDIS_URL });

app.get('/health', (_req: Request, res: Response) => {
  // Liveness: just check that the process is running
  res.status(200).send('OK');
});

app.get('/ready', async (_req: Request, res: Response) => {
  try {
    // Quick DB ping
    await dbPool.query('SELECT 1');
    // Quick Redis ping
    await redis.ping();
    res.status(200).send('Ready');
  } catch (err) {
    console.error('Readiness check failed', err);
    res.status(503).send('Not ready');
  }
});

export default app;

Dockerfile

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
EXPOSE 3000
CMD ["node", "dist/index.js"]

docker-compose.yml

services:
  api:
    build: .
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgres://user:pass@db:5432/app
      - REDIS_URL=redis://redis:6379
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 10s
      timeout: 5s
      retries: 3

Kubernetes example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: myrepo/api:latest
          ports:
            - containerPort: 3000
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 5

Key lines explained

• app.get('/health'): minimal check, no DB or network calls.
• app.get('/ready'): performs a lightweight DB and Redis ping.
• Docker healthcheck uses curl -f to treat non‑200 as failure.
• Kubernetes probes use initialDelaySeconds to allow startup time.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale?

• Distributed health: In a micro‑service mesh, each service may need to expose multiple health endpoints (e.g., /health/db, /health/cache).
• Circuit breaker integration: Health checks can be used to reset circuit breakers after a failure.
• Observability stack: Export health metrics to Prometheus; correlate with latency and error rates.
• Dynamic configuration: Health check intervals may vary per environment (dev vs prod).
• Cost: Frequent DB pings can increase load; use lightweight health probes or cache results.

Design decisions

When NOT to use the approach

• Serverless functions: Health checks are unnecessary; the platform manages invocations.
• Stateless micro‑services: If the service has no external dependencies, a simple liveness probe may suffice.
• Legacy monoliths: Adding health endpoints may not integrate with existing monitoring; consider a sidecar instead.

9. Interview questions

10. Hands‑on task

Task: Add a composite readiness endpoint to an existing Express service that checks:

1. PostgreSQL connectivity (SELECT 1).
2. Redis connectivity (PING).
3. External API health (GET https://api.example.com/health).

The endpoint should return 200 only if all checks succeed; otherwise 503. Add unit tests that mock each dependency. Deploy the service in Docker Compose and verify that the docker-compose healthcheck passes only when all dependencies are healthy.

11. Quick revision

• Health checks are lightweight probes that report service health to orchestrators.
• Liveness: is the process alive? → /health.
• Readiness: are all critical dependencies ready? → /ready.
• Keep checks fast (< 100 ms) and stateless.
• Separate liveness and readiness to avoid unnecessary restarts.
• Use timeouts, retries, and proper status codes (200 OK, 503 Service Unavailable).
• Secure health endpoints; expose only internally or with auth.
• In Kubernetes, livenessProbe restarts pods; readinessProbe removes them from service endpoints.
• Monitor health check metrics and logs for early failure detection.
• Graceful shutdown: set readiness to false before SIGTERM.
• Avoid heavy DB queries; use lightweight pings or cached status.
• For distributed systems, expose multiple health endpoints for granular checks.

12. References

• Kubernetes: Liveness and Readiness Probes
• Docker: Healthcheck
• Express.js: Router
• Node.js: pg
• Redis: Node client
• Prometheus: HTTP Exporter
• Istio: Health Checks