Distributed caching

Backend Fundamentals · Caching · 9 min read

1. What problem does this solve?

When a backend serves many users, the same data is often read repeatedly (e.g., user profiles, product catalogs, session tokens).
Problem: Each read hits the primary datastore (PostgreSQL, MongoDB, etc.), causing CPU, I/O, and network load.
Why it matters:

• Latency: Database round‑trips add 10‑50 ms per request; with millions of requests, the cumulative delay hurts user experience.
• Throughput: Databases have a finite number of connections and queries per second; heavy read traffic can saturate them, throttling writes and other services.
• Cost: Cloud DB instances scale with IOPS; more traffic means higher bills.

What happens if we don’t cache:

• Users experience slower pages.
• The database becomes a bottleneck, potentially causing timeouts or throttling.
• Scaling becomes expensive and complex.

Real‑world appearance:

• E‑commerce sites cache product listings.
• Social media feeds cache user timelines.
• SaaS dashboards cache aggregated metrics.
• AI inference services cache model embeddings or pre‑computed results.

2. Mental model

Analogy: The Library & the Librarian’s Desk

Imagine a library where every patron must ask a librarian for a book. The librarian fetches the book from the shelf, reads it, and hands it over. If many patrons ask for the same book, the librarian spends a lot of time retrieving it each time.

Now, place a copy of the book on a desk near the entrance. Patrons can grab it instantly. The librarian only needs to restock the desk when the copy is missing or outdated.

Key takeaways:

• The desk is the cache.
• The books are the data items.
• The librarian is the primary datastore.
• Restocking is cache invalidation or refresh.

Engineering model

• Cache nodes: In-memory data stores (Redis, Memcached, Hazelcast, etc.) distributed across machines.
• Key‑value mapping: Each data item is stored under a unique key.
• Cache strategy: Read‑through, write‑through, write‑back, or cache‑aside.
• Consistency model: Strong, eventual, or stale‑while‑revalidate.
• Partitioning: Data is sharded across nodes (consistent hashing, modulo, etc.).
• Replication: For fault tolerance, each shard may have replicas.

3. Deep explanation

Internals

Important terminology

• Cache miss: Requested key not present; fallback to datastore.
• Cache hit: Key found; return immediately.
• Cache stampede: Many concurrent misses for the same key, overwhelming the backend.
• Cache coherence: Ensuring all replicas see the same data.
• Cache invalidation: Explicitly removing or updating stale entries.
• Cache warming: Pre‑loading hot data at startup or during low traffic.

Trade‑offs

Failure modes

• Network partitions: Some nodes become unreachable; clients may get stale or missing data.
• Node crashes: Data loss if not replicated.
• Cache corruption: Corrupted entries can propagate if not validated.
• Hot spot: One key receives disproportionate traffic, exhausting a node’s memory.

Limitations

• Not a replacement for persistence: Cache is volatile; data must still reside in a durable store.
• Limited query capabilities: Most caches support simple key‑value lookups; complex queries require the database.
• Serialization overhead: Large objects can bloat memory and increase CPU.

Where it fits

Client --> API Gateway --> Service Layer --> Cache (Redis Cluster) --> DB (PostgreSQL)

• Read‑heavy services: Cache sits between service and DB.
• Write‑heavy services: Use write‑through or write‑back to keep cache and DB in sync.
• Microservices: Each service may have its own cache or share a cluster.

Interaction with related concepts

• Rate limiting: Cache can store request counters.
• Session management: Store session data in cache for fast access.
• CQRS: Read models can be cached for quick reads.
• Event sourcing: Cache can hold projections.

4. How it works

1. Client request arrives at the API.
2. Service checks cache for key user:{id}.
3. Cache hit → return data.
4. Cache miss → query database.
5. Store result in cache with TTL.
6. Return data to client.

sequenceDiagram
    participant Client
    participant API
    participant Cache
    participant DB

    Client->>API: GET /users/123
    API->>Cache: GET user:123
    Cache-->>API: MISS
    API->>DB: SELECT * FROM users WHERE id=123
    DB-->>API: {id:123, name:"Alice"}
    API->>Cache: SET user:123 {id:123, name:"Alice"} TTL=300
    API-->>Client: {id:123, name:"Alice"}

5. Practical implementation

TypeScript/Node.js example with ioredis

// cache.ts
import Redis from 'ioredis';

const redis = new Redis({
  host: process.env.REDIS_HOST ?? 'localhost',
  port: Number(process.env.REDIS_PORT ?? 6379),
  // Cluster mode if you have multiple nodes
  // cluster: true,
});

export default redis;

// userService.ts
import redis from './cache';
import { Pool } from 'pg';

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
});

export async function getUser(id: string) {
  const cacheKey = `user:${id}`;

  // 1️⃣ Try cache
  const cached = await redis.get(cacheKey);
  if (cached) {
    console.log('Cache hit');
    return JSON.parse(cached);
  }

  console.log('Cache miss, querying DB');
  const res = await pool.query('SELECT id, name, email FROM users WHERE id = $1', [id]);

  if (res.rows.length === 0) throw new Error('User not found');

  const user = res.rows[0];

  // 2️⃣ Store in cache with TTL (5 minutes)
  await redis.set(cacheKey, JSON.stringify(user), 'EX', 300);

  return user;
}

Notes

• Serialization: JSON.stringify/parse is simple; for binary data use msgpack.
• TTL: Prevents stale data; adjust based on update frequency.
• Cache invalidation: After an update, delete or update the key:

await redis.del(cacheKey); // after UPDATE users SET ... WHERE id=$1

• Cluster: If using Redis Cluster, ioredis automatically routes keys.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale?

• Architecture decisions:

  • Cache tier vs. CDN: For global read traffic, consider edge caches.
  • Cache per service vs. shared cluster: Shared cluster reduces operational overhead but introduces cross‑service contention.
  • Data partitioning: Use consistent hashing to minimize data movement.

• Trade‑offs:

  • Strong consistency (write‑through) vs. eventual consistency (cache‑aside).
  • Memory cost vs. miss rate.
  • Single‑tenant vs. multi‑tenant isolation in the cache.

• Bottlenecks:

  • Network latency between services and cache nodes.
  • Hot keys causing memory pressure on a single node.

• Failure scenarios:

  • Network partition leading to stale data.
  • Cache node crash causing data loss if not replicated.

• Monitoring:

  • Hit/miss ratio, eviction count, replication lag, memory fragmentation.

• Cost/performance:

  • Cloud managed Redis (e.g., AWS ElastiCache) vs. self‑managed.
  • Spot instances for cache nodes to reduce cost but risk eviction.

• When NOT to use:

  • If data changes extremely frequently and you cannot tolerate staleness.
  • If the data set is too large to fit comfortably in memory.
  • If the workload is write‑heavy and you cannot keep cache in sync efficiently.

9. Interview questions

10. Hands‑on task

Task: Implement a rate‑limit middleware using Redis that allows a maximum of 100 requests per minute per user ID.

Steps:

1. Create a new Node.js project (or use the existing repository).
2. Install express, ioredis, and dotenv.
3. Add a .env file with REDIS_URL.
4. Write middleware that:
   • Extracts userId from the request header.
   • Uses a Redis key rl:{userId} with INCR and EXPIRE.
   • Returns 429 Too Many Requests when the count exceeds 100.
5. Add a test route /hello that returns “Hello”.
6. Run the server, hit the route 101 times, and observe the 429 response.

Bonus: Add Prometheus metrics for requests per user and rate‑limit hits.

11. Quick revision

• Cache solves read‑latency and load‑scaling by storing hot data in memory.
• Cache miss → DB query → cache store → response.
• Key‑value stores: Redis, Memcached, Hazelcast.
• Consistent hashing keeps data distribution stable on topology changes.
• Eviction policies: LRU, LFU, TTL.
• Cache stampede: lock or “stale‑while‑revalidate” to avoid DB overload.
• Write‑through guarantees consistency; write‑back reduces latency but risks loss.
• Monitoring: hit/miss ratio, eviction count, memory usage.
• Common pitfalls: stale data, large objects, missing TTL, single‑node failure.
• At scale: consider replication, sharding, CDN integration, and cost‑performance trade‑offs.

12. References

• Redis Documentation – Data Structures
• Redis Cluster Overview
• ioredis GitHub
• PostgreSQL Connection Pooling – pg
• Prometheus – Metrics Exporter for Node.js
• Consistent Hashing – Wikipedia
• Rate Limiting Patterns – Cloudflare