Timeouts

Backend Fundamentals · Reliability Engineering · 7 min read

1. What problem does this solve?

• Problem: A backend service can get stuck waiting for an external dependency (DB, cache, third‑party API, another microservice) that never responds or takes too long.
• Why it matters:
  • Resource leakage: The request thread or event loop stays occupied, exhausting CPU or connection pools.
  • User experience: Clients see hanging requests, time‑outs, or degraded latency.
  • Cascade: A single slow call can back‑pressure the entire system, causing a denial‑of‑service.
• If not used:
  • Unbounded waits lead to memory leaks, exhausted connection pools, and eventually crashes.
  • In a distributed system, a single slow node can stall the whole chain.
• Real‑world appearance:
  • HTTP request handlers in Express/Next.js.
  • Database queries in TypeORM/Prisma.
  • Redis commands, Kafka producers/consumers.
  • External AI inference calls (e.g., OpenAI, HuggingFace).

2. Mental model

Analogy:
Imagine a waiter in a busy restaurant who keeps a customer’s order in the kitchen. If the kitchen takes forever, the waiter will eventually give up and ask the customer to leave. The waiter’s “give‑up” time is the timeout.

Engineering model:
A timeout is a deadline attached to an operation. If the operation hasn’t finished by that deadline, the system aborts it, cleans up resources, and returns an error. It’s a safety valve that protects the system from indefinite blocking.

3. Deep explanation

4. How it works

1. Client sends request → API gateway.
2. Gateway forwards to backend service with a request timeout (e.g., 5 s).
3. Backend receives and starts processing.
4. Backend makes external call (DB, Redis, AI) with a dependency timeout (e.g., 2 s).
5. If dependency responds before its timeout → backend continues.
6. If dependency times out → abort signal is fired, backend cleans up, and returns a 504/502 error.
7. If backend’s own processing exceeds its timeout → abort, return 503.
8. Client receives error and can retry or fallback.

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant S as Service
    participant D as Dependency

    C->>G: HTTP GET /data
    G->>S: Forward request (5s timeout)
    S->>D: Query DB (2s timeout)
    D-->>S: Result
    S-->>G: 200 OK
    G-->>C: 200 OK

5. Practical implementation

Node.js / TypeScript example

// src/handlers/getUser.ts
import { Request, Response } from 'express';
import { getUserFromDb } from '../db';
import { AbortController } from 'abort-controller';

export async function getUserHandler(req: Request, res: Response) {
  // 1. Set a 5s timeout for the entire request
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), 5000);

  try {
    // 2. Call DB with a 2s timeout
    const user = await getUserFromDb(req.params.id, controller.signal, 2000);
    res.json(user);
  } catch (err: any) {
    if (err.name === 'AbortError') {
      // Timeout happened
      res.status(504).json({ error: 'Request timed out' });
    } else {
      // Other errors
      res.status(500).json({ error: 'Internal server error' });
    }
  } finally {
    clearTimeout(timeoutId);
  }
}

// src/db/index.ts
import { Pool } from 'pg';
const pool = new Pool();

export async function getUserFromDb(
  id: string,
  signal: AbortSignal,
  timeoutMs: number
) {
  // 3. Pass timeout to the driver
  const client = await pool.connect();
  try {
    const res = await client.query({
      text: 'SELECT * FROM users WHERE id = $1',
      values: [id],
      // pg driver supports a `statement_timeout` per query
      statement_timeout: timeoutMs,
    }, signal);
    return res.rows[0];
  } finally {
    client.release();
  }
}

Key lines explained

• AbortController + setTimeout → global request deadline.
• signal passed to the DB query → driver can abort the query.
• statement_timeout → PostgreSQL enforces the per‑query timeout.

Redis example

import { createClient } from 'redis';

const client = createClient();
await client.connect();

async function getCache(key: string) {
  const controller = new AbortController();
  setTimeout(() => controller.abort(), 3000); // 3s timeout
  try {
    return await client.get(key, { abortSignal: controller.signal });
  } catch (e) {
    if (e.name === 'AbortError') throw new Error('Redis timeout');
    throw e;
  }
}

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

When scaling to thousands of requests per second:

• Architecture decisions

  • Per‑hop timeouts: API gateway, service, DB, external API.
  • Circuit breakers: Open after a threshold of timeouts.
  • Bulkhead isolation: Separate thread pools per dependency.

• Trade‑offs

  • Shorter timeouts reduce resource usage but increase error rates.
  • Longer timeouts improve success but risk cascading failures.

• Bottlenecks

  • Connection pool exhaustion.
  • CPU bound operations that ignore abort signals.

• Failure scenarios

  • Partial failure: DB times out but cache succeeds.
  • Total failure: All dependencies time out → service returns 503.

• Monitoring

  • Latency histograms per dependency.
  • Timeout rate vs. success rate dashboards.

• Cost/performance

  • Timeouts reduce idle connections, lowering cloud costs.
  • Too many retries can increase load on downstream services.

• When NOT to use

  • For idempotent, short‑lived operations where a retry is cheap.
  • In internal micro‑services with guaranteed low latency (e.g., in‑memory cache).

9. Interview questions

10. Hands‑on task

Task: Add a 3‑second timeout to the /search endpoint in the nextjs app that queries a PostgreSQL database. If the query times out, return a 504 error. Then, add a retry mechanism that retries once with exponential back‑off (500 ms) before giving up.

Steps

1. Wrap the DB query in an AbortController.
2. Use setTimeout to trigger abort() after 3000 ms.
3. Catch AbortError and return 504.
4. Implement a retry loop with setTimeout for back‑off.
5. Verify behavior with a slow query (e.g., pg_sleep(5)).

11. Quick revision

• Timeouts are deadlines that abort long‑running operations.
• Use AbortController + setTimeout in Node.js.
• Drivers often expose per‑query timeouts (statement_timeout).
• Combine timeouts with retries, back‑off, and circuit breakers.
• Separate timeouts per hop: client → gateway → service → dependency.
• Monitor timeout rates and latency histograms.
• Avoid global timeouts; tailor per‑dependency.
• Clear timers to prevent memory leaks.
• Return appropriate HTTP status (504, 502, 503).
• In production, tune based on load tests and SLA.

12. References

• Node.js AbortController
• Express Timeout Middleware
• PostgreSQL statement_timeout
• Redis Abort Signal
• OpenTelemetry: Timeout and Cancellation
• AWS SDK v3: Timeout