Key rotation

Backend Fundamentals · Backend Security · 10 min read

1. What problem does this solve?

Motivation
Every backend service uses cryptographic keys or secrets to protect data, authenticate users, or secure communication. If a key is compromised, all data encrypted or signed with that key becomes vulnerable. Even if a key is not compromised, over time keys can become weak (e.g., due to advances in cryptanalysis) or simply become stale.

What problem exists

• Key compromise: An attacker who obtains a key can decrypt data, forge tokens, or impersonate services.
• Key leakage: Keys accidentally exposed in logs, source control, or environment variables.
• Key aging: Keys that have been in use for a long time may be more likely to be cracked or misused.
• Compliance: Many regulations (PCI‑DSS, HIPAA, GDPR) require periodic key rotation.

Why the problem matters

• Data confidentiality: If a key is reused indefinitely, a single breach can expose all past data.
• Integrity: Old signing keys can be used to forge tokens if not revoked.
• Availability: Stale keys can cause service outages if not updated across all nodes.
• Auditability: Without rotation, it’s hard to prove that a system follows best‑practice security policies.

What happens if this concept is not used

• A single compromised key can lead to a cascading failure of all services that rely on it.
• Regulatory fines for non‑compliance.
• Loss of customer trust and potential legal liability.

Where this concept appears in real backend systems

• JWT signing keys for authentication.
• TLS certificates and private keys for HTTPS.
• Database credentials (PostgreSQL, MongoDB).
• API keys for third‑party services.
• Encryption keys for data at rest (e.g., S3 object encryption, encrypted PostgreSQL columns).
• Secrets stored in Vault, AWS KMS, or HashiCorp Vault.

2. Mental model

Analogy
Think of a house with a front door lock. If you keep the same key forever, a burglar who finds the key can open the door anytime. By changing the lock (or the key) regularly, you reduce the window of opportunity for the burglar. The lock itself is the key; the lock’s mechanism is the key management system that knows which lock is current.

Engineering model

• Key: The secret value (e.g., a 256‑bit AES key or RSA private key).
• Key store: A secure repository (KMS, Vault, HSM) that holds keys and their metadata.
• Key version: Each time a key is rotated, a new version is created and the old one is retired.
• Key usage: A service fetches the current key version to encrypt/decrypt or sign/verify.
• Key rotation policy: Rules that define when and how often keys are rotated (e.g., every 90 days, after a breach, or on demand).
• Key revocation: The process of marking a key as unusable before it naturally expires.

3. Deep explanation

Internals

• Key generation: Use a cryptographically secure RNG (e.g., crypto.randomBytes or KMS).
• Key storage: Store keys in an HSM or KMS; never hard‑code them in code or config files.
• Key metadata: Keep creation time, expiration, usage policy, and a unique identifier (e.g., kid).
• Key distribution: Services retrieve keys via API calls to the key store; caching is common for performance.

Important terminology

• Key lifecycle: Creation → Activation → Use → Retirement → Deletion.
• Key rotation: The act of replacing an active key with a new one.
• Key revocation: Explicitly marking a key as invalid before its natural expiry.
• Key versioning: Assigning a monotonically increasing number or UUID to each key.
• Key derivation: Generating a key from a master secret (e.g., using HKDF).

Trade‑offs

Failure modes

• Key loss: If a key is deleted before all services have switched, data becomes unreadable.
• Key mismatch: Services using different key versions for encryption/decryption.
• Cache staleness: Cached keys not refreshed after rotation.
• Revocation lag: Old keys still accepted by some services.
• Downtime: Services cannot start if they cannot fetch the current key.

Limitations

• Keys embedded in static binaries or client apps cannot be rotated without a new release.
• Distributed systems with many replicas need a coordination mechanism to propagate new keys.
• Some legacy protocols (e.g., TLS 1.0) may not support dynamic key updates.

Where this concept fits in a backend architecture

• Auth layer: JWT signing keys, OAuth client secrets.
• Data layer: Encryption keys for database columns, file storage.
• Transport layer: TLS certificates.
• Infrastructure layer: API gateway secrets, service mesh mTLS keys.

Interaction with related concepts

• Secrets management: Key rotation is a subset of secrets lifecycle.
• CI/CD pipelines: Deployments must handle key changes (e.g., env var updates).
• Observability: Rotation events should be logged and monitored.
• Compliance: Rotation policies often map to regulatory requirements.

4. How it works

1. Generate a new key (e.g., 256‑bit AES or RSA 2048).
2. Store the key in a secure key store with a unique kid.
3. Publish the new key to all services (via API, webhook, or configuration push).
4. Update service configuration to use the new kid for encryption/signing.
5. Mark the old key as retired; optionally keep it for a grace period.
6. Revoke the old key from the key store (prevent future use).
7. Cleanup: Delete the old key after all data encrypted with it has been re‑encrypted or after a safe period.

sequenceDiagram
    participant Client
    participant Service
    participant KeyStore

    Client->>Service: Request token
    Service->>KeyStore: Get current key (kid=1)
    KeyStore-->>Service: Return key1
    Service->>Client: Sign JWT with key1

    Note over Service,KeyStore: Rotation event
    KeyStore->>KeyStore: Generate key2 (kid=2)
    KeyStore-->>Service: Notify key2 available

    Service->>KeyStore: Get current key (kid=2)
    KeyStore-->>Service: Return key2
    Service->>Client: Sign JWT with key2

    Service->>KeyStore: Revoke key1
    KeyStore-->>Service: Key1 revoked

5. Practical implementation

Below is a minimal Node.js example that demonstrates JWT key rotation using Redis as a key store.
The app keeps a current_kid key in Redis and stores each key under key:{kid}.

// key-rotation.ts
import { createSign, createVerify } from 'crypto';
import { createClient } from 'redis';
import { randomBytes } from 'crypto';
import { promisify } from 'util';

const redis = createClient();
const getAsync = promisify(redis.get).bind(redis);
const setAsync = promisify(redis.set).bind(redis);
const delAsync = promisify(redis.del).bind(redis);

const KEY_PREFIX = 'key:';
const CURRENT_KID = 'current_kid';

// Generate a new 256‑bit key and store it
async function rotateKey() {
  const kid = Date.now().toString(); // simple unique id
  const key = randomBytes(32).toString('hex');

  await setAsync(`${KEY_PREFIX}${kid}`, key);
  await setAsync(CURRENT_KID, kid);

  console.log(`Rotated key. New kid=${kid}`);
  return kid;
}

// Sign a payload with the current key
async function sign(payload: object) {
  const kid = await getAsync(CURRENT_KID);
  const key = await getAsync(`${KEY_PREFIX}${kid}`);
  if (!key) throw new Error('No key found');

  const signer = createSign('SHA256');
  signer.update(JSON.stringify(payload));
  signer.end();
  const signature = signer.sign(key, 'hex');

  return { kid, payload, signature };
}

// Verify a token
async function verify(token: { kid: string; payload: any; signature: string }) {
  const key = await getAsync(`${KEY_PREFIX}${token.kid}`);
  if (!key) throw new Error('Key not found');

  const verifier = createVerify('SHA256');
  verifier.update(JSON.stringify(token.payload));
  verifier.end();
  return verifier.verify(key, token.signature, 'hex');
}

// Example usage
(async () => {
  await redis.connect();

  // Initial rotation
  await rotateKey();

  const token = await sign({ userId: 42 });
  console.log('Token:', token);

  const ok = await verify(token);
  console.log('Verification result:', ok);

  // Rotate again
  await rotateKey();

  // Old token still verifies (if we keep old key)
  const ok2 = await verify(token);
  console.log('Old token after rotation:', ok2);

  await redis.quit();
})();

Key points

• kid is stored in Redis so all services can read the current key.
• Rotating a key does not invalidate existing tokens; they remain verifiable until the key is revoked.
• To enforce revocation, delete the old key from Redis (delAsync) and optionally add a revocation list.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale

• Centralized key management: Use a dedicated KMS or Vault cluster with high availability.
• Policy automation: Define rotation policies in code (e.g., Terraform modules) and enforce via CI/CD.
• Service mesh integration: mTLS keys rotated by the mesh control plane.
• Observability: Emit metrics (keys_rotated_total, key_rotation_latency_ms) and alerts for failed rotations.
• Cost: KMS charges per request; balance between request frequency and cost.

Design decisions

• Key store choice: HSM for high‑value keys vs. cloud KMS for convenience.
• Rotation cadence: 90 days for compliance vs. 24 h for high‑risk keys.
• Grace period: 7 days of dual‑key support to avoid service disruption.
• Revocation strategy: Immediate revocation vs. scheduled revocation.
• Deployment strategy: Canary rotation to validate new keys before full rollout.

Bottlenecks

• Network latency to KMS can become a bottleneck if every request fetches a key.
• Cache invalidation: Too frequent invalidation can cause thrashing.

Failure scenarios

• Key store outage: Services must fallback to cached keys or pause operations.
• Key loss: Data encrypted with lost key becomes irrecoverable.

Monitoring

• Metrics: keys_rotated_total, key_rotation_success_rate, key_rotation_latency.
• Logs: Rotation events, revocation actions, failures.
• Alerts: Rotation failures, key expiry warnings.

When NOT to use

• Low‑risk data: If data is not sensitive, rotation may add unnecessary complexity.
• Static binaries: If keys are embedded in client apps, rotation requires a new release.
• Legacy systems: Systems that cannot handle key versioning may need a phased approach.

9. Interview questions

10. Hands‑on task

Task: Implement a simple key‑rotation service for JWT signing in a local Node.js project.

1. Create a Redis instance (Docker or local).
2. Write a script that:
   • Generates a new 256‑bit key every minute.
   • Stores it in Redis under key:{kid} and updates current_kid.
   • Logs rotation events.
3. Modify an existing Express route that signs a JWT using the current key.
4. Add a health‑check endpoint that verifies the current key can decrypt a sample token.
5. Run the rotation script and the Express app concurrently.
6. Observe that the app continues to issue valid tokens after each rotation.

11. Quick revision

• Key rotation replaces old cryptographic keys with new ones to limit exposure.
• Key store (KMS, Vault, Redis) holds keys and metadata.
• Key version (kid) identifies which key to use.
• Rotation flow: generate → store → publish → update services → retire old key.
• Cache local keys; invalidate on rotation.
• Grace period: accept old and new keys temporarily.
• Revocation: mark key unusable before natural expiry.
• Common pitfalls: hard‑coding, missing service updates, stale caches.
• Production: use centralized KMS, audit logs, metrics, automated policies.
• Scaling: pub/sub for key updates, versioned keys, canary deployments.
• Compliance: rotation policies satisfy PCI‑DSS, HIPAA, GDPR.

12. References

• NIST SP 800‑57 Part 1: Recommendation for Key Management
• RFC 7518 – JSON Web Algorithms
• AWS Key Management Service (KMS) Documentation
• HashiCorp Vault – Secrets Management
• Node.js Crypto Module
• Redis Key‑Value Store
• OpenTelemetry – Observability for Key Rotation