Rate limiting

Backend Fundamentals · Core Backend Fundamentals · 9 min read

1. What problem does this solve?

Motivation
When a backend service is exposed to the internet, any client can send requests at any speed.
If a client (or a malicious bot) floods the service with thousands of requests per second, the following can happen:

• Resource exhaustion – CPU, memory, database connections, or network bandwidth can be saturated.
• Denial of Service (DoS) – legitimate users experience latency or timeouts.
• Cost blow‑up – cloud resources (e.g., database read/write units, API gateway requests) spike.
• Security risk – brute‑force attacks on authentication endpoints or credential stuffing.

Why it matters
Rate limiting protects the service’s availability, performance, and cost while also providing a first line of defense against abuse.

Consequences of not using it

• Uncontrolled traffic can crash the application or underlying infrastructure.
• Poor user experience due to high latency or timeouts.
• Unexpected billing due to over‑utilization of paid resources.
• Increased attack surface for credential stuffing or credential‑guessing attacks.

Real‑world appearance

• API gateways (AWS API Gateway, Kong, NGINX) enforce per‑IP or per‑API‑key limits.
• Authentication services throttle login attempts.
• Web servers limit concurrent connections per client.
• Cloud functions (AWS Lambda, GCP Cloud Functions) often have built‑in concurrency limits that can be complemented with custom rate limiting.

2. Mental model

Analogy
Imagine a toll booth on a highway.

• Cars are incoming requests.
• The booth has a maximum number of cars it can process per minute (the rate limit).
• If too many cars arrive, the booth queues or rejects them to keep traffic flowing smoothly.

Engineering model
Rate limiting is a policy that says: “Allow at most N requests in a sliding window of T seconds for a given key (IP, API key, user ID, etc.).”
If the policy is violated, the service returns a 429 Too Many Requests response (or similar).
The policy can be enforced in-memory, distributed cache, or via an API gateway.

3. Deep explanation

4. How it works

1. Client sends request → includes identifying key (IP, API key, user ID).
2. Middleware intercepts → extracts key.
3. Retrieve counter from storage (e.g., Redis).
4. Increment counter and check against limit.
5. If over limit → respond with 429 (and optionally Retry-After).
6. If within limit → allow request to proceed.
7. Background job (optional) cleans up expired counters.

sequenceDiagram
    participant Client
    participant API_Gateway
    participant Redis
    participant Service

    Client->>API_Gateway: GET /api/resource (API Key)
    API_Gateway->>Redis: INCRBY key:1
    Redis-->>API_Gateway: 42
    alt over limit
        API_Gateway->>Client: 429 Too Many Requests
    else
        API_Gateway->>Service: Forward request
        Service-->>API_Gateway: 200 OK
        API_Gateway-->>Client: 200 OK
    end

5. Practical implementation

Below is a minimal, production‑ready rate‑limiting middleware for an Express/Node.js app using Redis as a distributed counter store.

// rateLimiter.ts
import { Request, Response, NextFunction } from 'express';
import { createClient } from 'redis';

const redis = createClient(); // default localhost:6379
redis.on('error', console.error);

export interface RateLimitOptions {
  windowMs: number;          // window size in ms
  max: number;               // max requests per window
  keyGenerator: (req: Request) => string; // how to identify client
  message?: string;          // optional custom 429 message
}

export function rateLimiter(opts: RateLimitOptions) {
  const { windowMs, max, keyGenerator, message } = opts;
  const ttl = Math.ceil(windowMs / 1000); // seconds

  return async (req: Request, res: Response, next: NextFunction) => {
    const key = `rl:${keyGenerator(req)}`;
    try {
      // INCR and set TTL atomically
      const [count] = await redis.multi()
        .incr(key)
        .expire(key, ttl)
        .exec();

      if (count > max) {
        res.set('Retry-After', ttl.toString());
        return res.status(429).json({
          error: message ?? 'Too many requests, please try again later.',
        });
      }
      next();
    } catch (err) {
      console.error('Rate limiter error', err);
      // Fail open: allow request if Redis is down
      next();
    }
  };
}

Usage

// app.ts
import express from 'express';
import { rateLimiter } from './rateLimiter';

const app = express();

app.use(
  rateLimiter({
    windowMs: 60 * 1000, // 1 minute
    max: 100,            // 100 requests per minute
    keyGenerator: (req) => req.ip, // per IP
  })
);

app.get('/api/data', (req, res) => {
  res.json({ data: 'Hello, world!' });
});

app.listen(3000, () => console.log('Server listening on 3000'));

Why this works in production

• Redis is a single‑write, multi‑read store that can handle millions of increments per second.
• The INCR command is atomic, so concurrent requests from the same key are correctly counted.
• EXPIRE ensures counters are automatically cleaned up after the window.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale?

• Distributed counters: Use Redis Cluster or a sharded keyspace to avoid single‑point bottlenecks.
• Adaptive limits: Dynamically adjust limits based on traffic patterns or user tier.
• Global vs. per‑service: Decide whether to enforce limits globally (gateway) or per microservice.
• Burst handling: Implement Token Bucket to allow short bursts while maintaining long‑term averages.
• Multi‑tenant isolation: Ensure limits are per tenant, not just per IP.

Design decisions

Bottlenecks & failure scenarios

• Redis saturation: High write traffic can saturate Redis CPU. Mitigate with pipelining or sharding.
• Network partition: Clients may see inconsistent counters. Use quorum reads/writes if necessary.
• Clock drift: Use NTP or atomic timestamps from Redis to avoid window misalignment.

Monitoring & cost

• Metrics: rate_limit_exceeded, rate_limit_hits, redis_latency.
• Alerting: Spike in 429s → possible DoS or misconfiguration.
• Cost: Redis memory cost scales with number of keys; consider key eviction policies.

When NOT to use

• Very low traffic services: A simple in‑memory counter may suffice.
• Stateless micro‑services: If the service is behind a gateway that already enforces limits, duplicate enforcement is wasteful.
• Highly dynamic keys: If the key changes frequently (e.g., per request), the overhead may outweigh benefits.

9. Interview questions

10. Hands‑on task

Goal: Add a per‑user rate limiter to an existing Express API that returns user data.

Steps

1. Create a Redis client in redisClient.ts.
2. Implement a userRateLimiter middleware that limits each authenticated user to 10 requests per minute.
3. Integrate the middleware into the /api/user/:id route.
4. Add unit tests using Jest to verify that the limit is enforced.
5. Deploy locally using Docker Compose (Redis + Node.js).
6. Verify by sending more than 10 requests in a minute and observing 429 responses.

Hints

• Use req.user.id as the key (assume authentication middleware sets req.user).
• Store counters in Redis with key pattern rl:user:{id}.
• Set TTL to 60 s.
• Log each 429 response for debugging.

11. Quick revision

• Rate limiting protects against DoS, abuse, and cost spikes.
• Common algorithms: Fixed Window, Sliding Window, Token Bucket, Leaky Bucket.
• Key concepts: Key, Window, Limit, Burst, Penalty.
• Redis INCR + EXPIRE is a simple, atomic implementation.
• Fail‑open vs. fail‑closed: choose based on availability vs. security.
• Monitor 429 metrics and Redis latency.
• Avoid per‑IP only limits; combine with API keys or user IDs.
• Use Retry-After header to inform clients.
• For distributed systems, use a shared store (Redis Cluster, DynamoDB).
• Token Bucket allows bursts; Sliding Window Log gives precise control at the cost of memory.

12. References

• Express Rate Limit – npm
• Redis INCR command
• AWS API Gateway throttling
• Nginx Rate Limiting
• Token Bucket Algorithm
• Sliding Window Log vs. Counter
• Redis Cluster Overview