SLO

Backend Fundamentals · Reliability Engineering · 8 min read

1. What problem does this solve?

Motivation
In a real‑world backend, multiple teams ship features, fix bugs, and deploy changes continuously. Clients expect a certain level of reliability, but developers often measure success by unit test pass rates or deployment counts. Those metrics don’t capture the user‑experience of the service.

Problem

• No single, shared metric that ties engineering effort to business impact.
• Teams may over‑optimize for latency or throughput while ignoring availability.
• Stakeholders cannot negotiate priorities when they don’t know how much “downtime” is acceptable.

Why it matters

• Customers pay for uptime; a 99.9 % SLA can cost millions in lost revenue.
• Unclear expectations lead to firefighting, scope creep, and mistrust.
• Without a clear target, monitoring becomes ad‑hoc and hard to interpret.

Consequences of not using SLOs

• Over‑engineering for latency at the expense of availability.
• Blindly adding capacity because “the system is slow” without knowing if that matters to users.
• Inconsistent incident response: some teams treat a 1 % error rate as critical, others ignore it.

Where it appears

• Cloud provider SLAs (e.g., AWS, GCP).
• Internal dashboards (e.g., SRE teams publish SLOs for API endpoints).
• Feature‑flag rollouts: “We’ll enable this only if the error rate stays below X% for 30 days.”
• Incident post‑mortems: “We breached the 99.95 % SLO for the payment API.”

2. Mental model

Analogy
Think of a SLO like a traffic light for a highway.

• Green: The road is clear; cars move smoothly.
• Yellow: Minor congestion; drivers should slow down.
• Red: Major blockage; traffic stops.

A SLO is the rule that defines when the light turns from green to yellow or red based on measurable traffic (requests, errors, latency). It tells engineers when to take action.

Engineering model

• Service → Metric (e.g., error rate, latency percentile).
• Target → Threshold (e.g., 99.9 % availability).
• Monitoring → Alerting → Remediation.

The SLO is the bridge between what the service does and how we judge its quality.

3. Deep explanation

Internals

• SLO: A quantitative description of the level of service that a system should provide.
• Service Level Indicator (SLI): The metric that measures the service (e.g., uptime, error rate, latency).
• Service Level Agreement (SLA): The contractual promise to customers, often derived from the SLO.
• Error budget: The amount of failure allowed before the SLO is breached (e.g., 0.1 % downtime in a month).

Important terminology

Trade‑offs

Failure modes

• Metric drift: The SLI metric changes definition (e.g., adding a new error type) without updating the SLO.
• Alert fatigue: Too many alerts for minor deviations.
• Mis‑aligned SLOs: SLO targets that are too aggressive or too lenient for the business.
• Unmonitored components: A critical microservice not included in the SLO leads to blind spots.

Limitations

• SLOs are soft guarantees; they don’t enforce hard limits.
• They require continuous monitoring and tuning.
• They can be gamed if teams focus on the metric rather than real user experience.

Where it fits in backend architecture

• API Gateway: SLOs often defined per endpoint.
• Service Mesh: Observability layer that collects SLIs.
• CI/CD pipeline: Automated tests can check SLO compliance before promotion.
• Incident management: SLO breaches trigger runbooks.

Interaction with related concepts

• Chaos Engineering: Inject failures to test SLO resilience.
• Feature Flags: Use error budgets to decide when to roll back.
• Auto‑Scaling: Scale to keep latency within SLO bounds.
• Rate Limiting: Protect downstream services to maintain availability.

4. How it works

1. Define the SLO: Choose an SLI (e.g., error rate) and target (e.g., 99.9 % uptime).
2. Instrument the service: Emit metrics to a monitoring system (Prometheus, Datadog).
3. Set up alerting: Trigger when the SLI deviates from the target beyond a tolerance.
4. Allocate an error budget: Compute allowed failures for the period.
5. Monitor continuously: Visualize in dashboards; track budget consumption.
6. Respond to breaches: Run runbooks, roll back features, or scale resources.
7. Review and adjust: Post‑mortems refine the SLO or SLI.

sequenceDiagram
    participant Dev as Developer
    participant API as API Gateway
    participant Prom as Prometheus
    participant Alert as Alerting
    participant Ops as Ops Team

    Dev->>API: Deploy new code
    API->>Prom: Emit metrics (latency, errors)
    Prom->>Alert: Evaluate SLI against SLO
    Alert->>Ops: Trigger alert if breached
    Ops->>Ops: Execute runbook
    Ops->>API: Rollback or scale

5. Practical implementation

Example: 99.9 % availability SLO for a /login endpoint

// src/middleware/slo.ts
import { Request, Response, NextFunction } from 'express';
import promClient from 'prom-client';

const loginErrors = new promClient.Counter({
  name: 'login_errors_total',
  help: 'Total number of failed login attempts',
});

const loginRequests = new promClient.Counter({
  name: 'login_requests_total',
  help: 'Total number of login requests',
});

export function loginSLO(req: Request, res: Response, next: NextFunction) {
  loginRequests.inc(); // Count every request

  // Wrap the original handler
  const originalSend = res.send;
  res.send = function (body?: any) {
    if (res.statusCode >= 400) {
      loginErrors.inc(); // Count errors
    }
    return originalSend.call(this, body);
  };

  next();
}

// src/app.ts
import express from 'express';
import promClient from 'prom-client';
import { loginSLO } from './middleware/slo';

const app = express();
app.use(loginSLO);

app.post('/login', async (req, res) => {
  // Simulate auth logic
  const success = Math.random() > 0.1; // 10% failure
  if (!success) return res.status(401).send('Unauthorized');
  res.send('OK');
});

// Expose metrics
app.get('/metrics', async (_req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(await promClient.register.metrics());
});

app.listen(3000, () => console.log('Server running on :3000'));

Observability

• Prometheus scrapes /metrics.
• Grafana dashboard shows login_errors_total / login_requests_total.
• Alert rule (Prometheus) triggers if error rate > 0.1 % over 5 min.

Calculating error budget

// utils/errorBudget.ts
export function errorBudget(available: number, target: number) {
  return (1 - target) * available; // e.g., 1 h * 0.001 = 3.6 s
}

Use this in CI/CD to gate releases: if the error budget is exhausted, block promotion.

6. Production considerations

7. Common mistakes

8. Senior engineer perspective

What changes at scale?

• Granularity: Define SLOs per service, per endpoint, even per user tier.
• Automation: Use policy engines (OPA) to enforce SLO compliance in CI/CD.
• Observability stack: Centralized tracing (Jaeger), metrics (Prometheus), logs (ELK).
• Chaos experiments: Regularly inject failures to validate SLO resilience.
• Cost vs. reliability: Balance extra replicas or caching layers against budget.

Design decisions

When NOT to use the approach

• Very small services with trivial traffic: manual monitoring may suffice.
• Experimental prototypes: SLOs add overhead that outweighs benefits.
• Highly regulated domains: SLAs may be legally binding; SLOs must be carefully aligned.

9. Interview questions

10. Hands‑on task

Task: Add a 99.9 % latency SLO for the /search endpoint in the existing Node.js project.

1. Instrument latency using prom-client histograms.
2. Create a Grafana dashboard to display the 95th percentile latency.
3. Write a Prometheus alert rule that fires if the 95th percentile exceeds 200 ms for 5 min.
4. Update the CI pipeline to block a merge if the alert is active.
5. Simulate a latency spike (e.g., by adding await sleep(300) in the handler) and observe the alert.

11. Quick revision

• SLO: Quantitative target for service quality.
• SLI: Metric that measures the service (error rate, latency, uptime).
• Error budget: Allowed failures = 1 – SLO.
• SLA: Contractual promise derived from SLO.
• Monitoring: Collect SLI, compute SLO compliance, alert on breaches.
• Trade‑offs: Latency vs. availability; strictness vs. innovation.
• Automation: Gate CI/CD with SLO checks; use feature flags.
• Observability stack: Prometheus + Grafana + tracing.
• Common pitfalls: Metric drift, alert fatigue, ignoring downstream services.
• Senior perspective: Granular SLOs, error‑budget allocation, chaos testing.
• Interview nuggets: Distinguish SLO vs. SLA; explain error budget; design for multi‑tenant.

12. References

• Google SRE Book – Service Level Objectives
• Prometheus – Recording Rules
• Datadog – SLOs
• OpenTelemetry – Observability
• GitHub Actions – CI/CD with SLO checks