DynamoDB concepts

Backend Fundamentals · Databases · 9 min read

1. What problem does this solve?

In production systems, DynamoDB is often the backbone for session stores, analytics pipelines, leaderboards, and any component that requires fast, scalable key‑value access.

2. Mental model

Analogy: The Library

Engineering model

• Partition key is hashed to a partition (a physical storage unit).
• Sort key (optional) orders items within that partition.
• GSIs create separate hash/sort key pairs that allow alternative query patterns.
• Capacity units (RCU/WCU) quantify how many reads/writes per second you can perform.
• Consistency: reads can be eventual (fast, cheap) or strong (slower, guaranteed).
• TTL automatically deletes items after a timestamp.

3. Deep explanation

Internals

1. Partitioning – DynamoDB distributes data across partitions using the partition key hash.
2. Replication – Each partition is replicated across three Availability Zones (AZs).
3. SSD storage – Low‑latency, high‑throughput.
4. Indexing – GSIs are stored as separate tables; updates to the base table trigger writes to the index.
5. Capacity –
   • Provisioned: you specify read/write units; DynamoDB throttles if exceeded.
   • On‑Demand: DynamoDB scales automatically; you pay per request.

Important terminology

Trade‑offs

Failure modes

• Throttling – Exceeding RCU/WCU → ProvisionedThroughputExceededException.
• Hot partitions – Poor key design → uneven load, throttling.
• Eventual consistency lag – Reads may miss recent writes.
• Index replication lag – GSIs may be out of sync for a few milliseconds.
• Item size > 400 KB – Write fails (ItemSizeTooLargeException).

Limitations

• No joins or subqueries.
• Max item size 400 KB.
• Max 10 GB per partition.
• Limited query patterns (must use key or index).
• No server‑side joins or aggregations (use Athena or Redshift for analytics).

Where it fits

• Microservice data store – per‑service tables.
• Session store – TTL for automatic cleanup.
• Analytics – Append‑only logs, later exported to S3/Redshift.
• Event sourcing – Append events, read by event ID.

Interaction with other backend concepts

4. How it works

Request flow for a PutItem

sequenceDiagram
    participant Client
    participant DynamoDB
    Client->>DynamoDB: PutItem(item)
    DynamoDB->>DynamoDB: Hash partition key -> partition
    DynamoDB->>DynamoDB: Write item to partition
    DynamoDB->>DynamoDB: Replicate to 3 AZs
    DynamoDB->>Client: Success

General flow for a Query with GSI

sequenceDiagram
    participant Client
    participant DynamoDB
    Client->>DynamoDB: Query(GSI, key)
    DynamoDB->>DynamoDB: Locate GSI partition
    DynamoDB->>DynamoDB: Read items
    DynamoDB->>Client: Return results

5. Practical implementation

Below is a minimal TypeScript example using AWS SDK v3.
It demonstrates:

1. Creating a table with a partition key (UserId) and sort key (Timestamp).
2. Adding a Global Secondary Index (EmailIndex).
3. Performing PutItem, Query, and Scan.
4. Using TTL.

// dynamodb-demo.ts
import {
  DynamoDBClient,
  CreateTableCommand,
  DeleteTableCommand,
  PutItemCommand,
  QueryCommand,
  ScanCommand,
  UpdateTimeToLiveCommand,
} from "@aws-sdk/client-dynamodb";
import { marshall, unmarshall } from "@aws-sdk/util-dynamodb";

const REGION = "us-east-1";
const client = new DynamoDBClient({ region: REGION });

const TABLE_NAME = "Users";
const GSI_NAME = "EmailIndex";

// 1️⃣ Create table
async function createTable() {
  const params = {
    TableName: TABLE_NAME,
    AttributeDefinitions: [
      { AttributeName: "UserId", AttributeType: "S" },
      { AttributeName: "Timestamp", AttributeType: "N" },
      { AttributeName: "Email", AttributeType: "S" },
    ],
    KeySchema: [
      { AttributeName: "UserId", KeyType: "HASH" },
      { AttributeName: "Timestamp", KeyType: "RANGE" },
    ],
    GlobalSecondaryIndexes: [
      {
        IndexName: GSI_NAME,
        KeySchema: [
          { AttributeName: "Email", KeyType: "HASH" },
          { AttributeName: "Timestamp", KeyType: "RANGE" },
        ],
        Projection: { ProjectionType: "ALL" },
        ProvisionedThroughput: { ReadCapacityUnits: 5, WriteCapacityUnits: 5 },
      },
    ],
    ProvisionedThroughput: { ReadCapacityUnits: 5, WriteCapacityUnits: 5 },
  };
  await client.send(new CreateTableCommand(params));
  console.log(`Table ${TABLE_NAME} created`);
}

// 2️⃣ Enable TTL on the 'ExpiresAt' attribute
async function enableTTL() {
  await client.send(
    new UpdateTimeToLiveCommand({
      TableName: TABLE_NAME,
      TimeToLiveSpecification: { AttributeName: "ExpiresAt", Enabled: true },
    })
  );
  console.log("TTL enabled");
}

// 3️⃣ Put an item
async function putUser(user: {
  UserId: string;
  Email: string;
  Name: string;
  ExpiresAt: number;
}) {
  const item = {
    UserId: { S: user.UserId },
    Timestamp: { N: Date.now().toString() },
    Email: { S: user.Email },
    Name: { S: user.Name },
    ExpiresAt: { N: user.ExpiresAt.toString() },
  };
  await client.send(new PutItemCommand({ TableName: TABLE_NAME, Item: item }));
  console.log(`User ${user.UserId} stored`);
}

// 4️⃣ Query by Email using GSI
async function queryByEmail(email: string) {
  const params = {
    TableName: TABLE_NAME,
    IndexName: GSI_NAME,
    KeyConditionExpression: "Email = :e",
    ExpressionAttributeValues: {
      ":e": { S: email },
    },
  };
  const result = await client.send(new QueryCommand(params));
  const users = result.Items?.map((i) => unmarshall(i));
  console.log(`Found ${users?.length ?? 0} users with email ${email}`);
  return users;
}

// 5️⃣ Scan the table (not recommended for large tables)
async function scanAll() {
  const result = await client.send(new ScanCommand({ TableName: TABLE_NAME }));
  const items = result.Items?.map((i) => unmarshall(i));
  console.log(`Scanned ${items?.length ?? 0} items`);
  return items;
}

// Example usage
(async () => {
  await createTable();
  await enableTTL();

  await putUser({
    UserId: "u123",
    Email: "alice@example.com",
    Name: "Alice",
    ExpiresAt: Math.floor(Date.now() / 1000) + 60 * 60, // 1 hour
  });

  await queryByEmail("alice@example.com");
  await scanAll();

  // Clean up
  await client.send(new DeleteTableCommand({ TableName: TABLE_NAME }));
  console.log(`Table ${TABLE_NAME} deleted`);
})();

Important lines
AttributeDefinitions – must match key schema.
ProvisionedThroughput – sets RCU/WCU; adjust for load.
UpdateTimeToLiveCommand – TTL attribute name must be numeric epoch.
QueryCommand – uses GSI; otherwise you must query by primary key.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

Architecture decisions

Trade‑offs & bottlenecks

• Hot partitions → throttle, high latency.
• Write amplification → GSIs double write cost.
• Index replication lag → eventual consistency across indexes.
• Large items → high RU consumption, risk of failure.

Failure scenarios

Monitoring

• CloudWatch metrics: ConsumedReadCapacityUnits, ConsumedWriteCapacityUnits, ThrottledRequests.
• CloudTrail logs for table changes.
• X‑Ray traces for request latency.
• Custom dashboards for hot partition detection.

Cost/performance

• Estimate RU usage: ReadCost = (StrongReadUnits * 4KB) + (EventualReadUnits * 4KB); WriteCost = WriteUnits * 1KB.
• Use reserved capacity if traffic is predictable.
• Avoid unnecessary GSIs; each GSI adds write cost.

When NOT to use DynamoDB

• Complex relational queries (joins, multi‑row transactions).
• Need for ACID across many items (>25).
• Heavy analytical workloads (use Redshift or Athena).
• Very small datasets that fit comfortably in memory or a relational DB.

9. Interview questions

10. Hands‑on task

Build a session store

1. Create a table Sessions with SessionId (partition key) and ExpiresAt (TTL).
2. Add a GSI UserIdIndex on UserId.
3. Implement createSession(userId, data) that writes a session with a 30‑minute TTL.
4. Implement getSession(sessionId) and getSessionsByUser(userId).
5. Write a simple Express/Next.js API that uses these functions.
6. Deploy locally with Docker and test with curl or Postman.

Goal: Understand TTL, GSIs, and basic CRUD in a real backend context.

11. Quick revision

• Partition key – distributes data; must be unique and high cardinality.
• Sort key – orders items within a partition; enables range queries.
• GSIs – alternate query patterns; each has its own capacity.
• LSIs – same partition key, different sort key; limited to 5 per table.
• Provisioned vs On‑Demand – capacity planning vs pay‑per‑use.
• RCU/WCU – units of read/write throughput; 1 RCU ≈ 4 KB strongly consistent.
• Strong vs Eventual consistency – trade‑off between latency and freshness.
• TTL – automatic deletion of expired items.
• Hot partitions – uneven load → throttling; mitigate with key design.
• Auto‑scaling – dynamic capacity adjustment.
• PITR – point‑in‑time recovery for accidental deletes.
• IAM & encryption – secure access and data at rest.

12. References

• Amazon DynamoDB Developer Guide
• AWS SDK for JavaScript v3 – DynamoDB Client
• DynamoDB Best Practices
• DynamoDB Capacity Modes
• DynamoDB Global Secondary Indexes
• DynamoDB Time‑to‑Live (TTL)
• DynamoDB Auto Scaling
• AWS CloudWatch Metrics for DynamoDB