Secrets management

Backend Fundamentals · Backend Security · 8 min read

1. What problem does this solve?

Motivation
When a backend service needs to talk to a database, an external API, or a message broker, it must authenticate itself. The credentials (passwords, API keys, certificates) are called secrets. If these secrets are mishandled, the entire system can be compromised.

What problem exists

• Secrets are often stored in plain text (e.g., in source‑control, in environment variables, or in config files).
• Developers may copy secrets into code or commit them accidentally.
• Secrets may be duplicated across environments (dev, staging, prod) leading to drift.

Why the problem matters

• A leaked secret can give attackers full access to a database or third‑party service.
• Compromised credentials can be used to exfiltrate data, inject malicious code, or launch denial‑of‑service attacks.
• Regulatory compliance (GDPR, PCI‑DSS) requires strict handling of credentials.

What happens if this concept is not used

• Secrets may be exposed in logs, error messages, or version control.
• Manual rotation becomes error‑prone and slow.
• Auditing and compliance become impossible.

Where this concept appears in real backend systems

• A microservice that reads a PostgreSQL password from an environment variable.
• A serverless function that pulls an API key from AWS Secrets Manager.
• A CI/CD pipeline that injects a JWT signing key into a Docker image at build time.

2. Mental model

Analogy
Think of secrets as keys to a vault.

• The vault is your production environment.
• The keys must be kept safe, only given to authorized people, and replaced regularly.

Engineering model

• Secret store: a secure, auditable repository (e.g., HashiCorp Vault, AWS Secrets Manager, Azure Key Vault).
• Access control: IAM policies or role‑based access that limits who can read a secret.
• Injection: secrets are injected into the runtime environment (environment variables, mounted files, or API calls) just before the application starts.
• Rotation: secrets are periodically regenerated and the new value is propagated automatically.

3. Deep explanation

4. How it works

1. Define the secret in the secret store (e.g., db_password).
2. Attach a policy that allows the service’s IAM role to read the secret.
3. Deploy the service with a short‑lived token or role assumption (e.g., AWS IAM role).
4. At startup, the service calls the secret store API to fetch the secret.
5. Inject the secret into the process environment or a configuration object.
6. Use the secret to establish connections (DB, API, etc.).
7. Rotate the secret automatically; the service re‑fetches the new value without downtime.

sequenceDiagram
    participant App
    participant SecretStore
    participant DB

    App->>SecretStore: Request db_password (STS token)
    SecretStore-->>App: Return encrypted password
    App->>App: Decrypt & set env var
    App->>DB: Connect using password

5. Practical implementation

Below is a minimal Node.js/TypeScript example that fetches a PostgreSQL password from AWS Secrets Manager and uses it to connect via pg.

// src/secrets.ts
import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";
import { config } from "dotenv";

config(); // Load .env for AWS credentials

const secretsClient = new SecretsManagerClient({ region: process.env.AWS_REGION });

export async function getSecret(secretName: string): Promise<string> {
  const command = new GetSecretValueCommand({ SecretId: secretName });
  const response = await secretsClient.send(command);

  if (!response.SecretString) {
    throw new Error(`Secret ${secretName} has no string value`);
  }
  return response.SecretString;
}

// src/db.ts
import { Pool } from "pg";
import { getSecret } from "./secrets";

export async function createPool(): Promise<Pool> {
  const password = await getSecret("prod/db_password");
  const pool = new Pool({
    host: process.env.DB_HOST,
    port: Number(process.env.DB_PORT),
    user: process.env.DB_USER,
    password,
    database: process.env.DB_NAME,
  });
  return pool;
}

// src/server.ts
import express from "express";
import { createPool } from "./db";

async function main() {
  const pool = await createPool();
  const app = express();

  app.get("/", async (_req, res) => {
    const { rows } = await pool.query("SELECT NOW() AS time");
    res.json(rows[0]);
  });

  app.listen(3000, () => console.log("Server listening on :3000"));
}

main().catch(console.error);

Key points

• Secrets are never hard‑coded or stored in the repo.
• The service uses an IAM role (AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) to authenticate to Secrets Manager.
• The secret is fetched once at startup; you can cache it or refresh it on a schedule if needed.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale?

• Architecture decisions

  • Centralized vs. decentralized: A single vault per cluster vs. per‑service vaults.
  • Network placement: Vault in a private subnet, accessible via VPC endpoints.
  • Token strategy: Use short‑lived IAM roles or Kubernetes ServiceAccounts with OIDC.

• Trade‑offs

  • Central vault simplifies rotation but can become a bottleneck.
  • Decentralized vaults reduce risk of a single outage but increase operational overhead.

• Bottlenecks

  • Secret store API limits (e.g., 5 000 requests/min).
  • Network latency to the vault.

• Failure scenarios

  • Vault outage → all services fail to start.
  • Mis‑configured ACL → unauthorized access or denial of service.

• Monitoring

  • Track secret read latency, error rates, and access counts.
  • Alert on failed authentication attempts.

• Cost/performance

  • Cloud secret stores charge per request; caching mitigates cost.
  • In‑memory caching reduces latency but increases memory footprint.

• When NOT to use

  • For very small, single‑instance services where the risk is low and the overhead of a vault outweighs benefits.
  • When secrets are truly static and never need rotation (rare in production).

9. Interview questions

10. Hands‑on task

Task:
Add secret management to the existing nextjs-demo repository.

1. Create a new secret in AWS Secrets Manager named demo/db_password with a random password.
2. Update the nextjs-demo API route /api/db to fetch the password at runtime using the AWS SDK (similar to the example above).
3. Store the database connection string in an environment variable that references the secret value.
4. Deploy the updated API to a staging environment and verify that the database connection works.
5. Document the steps you took and any challenges encountered.

11. Quick revision

• Secrets are sensitive credentials that must be protected from exposure.
• Use a dedicated secret store (Vault, Secrets Manager, Key Vault).
• Enforce fine‑grained access control (IAM, policies).
• Fetch secrets at startup or on demand; cache if needed.
• Rotate secrets regularly; automate rotation.
• Never hard‑code secrets; avoid storing them in plain text.
• Log all secret access for auditability.
• Handle secret fetch failures gracefully.
• Consider performance, reliability, and cost when scaling.
• Use short‑lived tokens to limit exposure.
• Monitor latency, error rates, and access patterns.
• When in doubt, prefer a central vault over ad‑hoc env vars.

12. References

• AWS Secrets Manager Documentation
• HashiCorp Vault Documentation
• Node.js AWS SDK v3
• PostgreSQL Node.js Driver (pg)
• Next.js API Routes
• OpenAPI Spec for Secrets