Project: An Observability Plan

Project: An Observability Plan

Everything in this tutorial has built toward one deliverable: a production-grade observability plan for a real service. At Google, every new service must pass a Production Readiness Review (PRR) before launch. Observability — defined SLIs, SLOs, dashboards, and alert rules — is a mandatory PRR gate. This lesson walks you through producing exactly that artifact, end to end, for a sample e-commerce checkout service.

The output of this lesson is a set of files you can commit to a repository: SLO definitions in YAML, a Prometheus alerting rule file, and a Grafana dashboard JSON model. These are not toy examples — they follow the same structure used by SRE teams at scale.

Step 1: Define the Service and Its User Journeys

Before writing a single metric query, document what the service does and which user actions are critical. For the checkout-service:

• Primary journey: User submits an order — the checkout service validates the cart, calls the payment processor, reserves inventory, and returns a confirmation within 3 seconds.
• Secondary journey: Retrieve past orders for the order history page (latency-sensitive but not revenue-blocking).
• Background job: Send post-purchase email (async, failure acceptable with retry).

Identify the reliability targets your users actually notice. A payment that takes 15 seconds feels broken even if it technically succeeds. An email that arrives 30 minutes late is invisible to the user.

Step 2: Define SLIs and SLOs

For each journey, define a Service Level Indicator (the metric) and a Service Level Objective (the target). Use the request-based availability and latency SLI formulas standardised in the Google SRE Workbook.

SLI definitions for checkout-service:

• Availability SLI: good_requests / total_requests, where a good request is any HTTP 2xx or 3xx response to POST /checkout.
• Latency SLI: Proportion of POST /checkout requests completing in under 2 seconds, measured at the 99th percentile over a 5-minute window.
• Order retrieval latency SLI: Proportion of GET /orders requests completing in under 500 ms at the 95th percentile.

SLO targets (28-day rolling window):

• Availability: 99.9% (allows ~43 minutes of bad requests per month)
• Checkout latency p99 < 2 s: 99.5% of requests
• Order retrieval latency p95 < 500 ms: 99% of requests

Encode these SLOs in a machine-readable format. The OpenSLO spec (CNCF project) provides a vendor-neutral YAML schema; Sloth and Pyrra can generate Prometheus rules from it automatically:

Step 3: Design the Dashboard

A production dashboard answers four questions at a glance: Is the service healthy right now? Is it meeting its SLOs? What changed recently? Where is the bottleneck? Structure every service dashboard around these sections, in this order:

1. SLO Status row — Current error budget burn rate, remaining budget this window, SLO compliance percentage. If this row is red, nothing else matters.
2. Golden Signals row — Latency (p50/p95/p99), traffic (RPS), error rate (%), saturation (CPU %, memory %, queue depth).
3. Dependency Health row — Downstream service latency and error rates (payment processor, inventory service, database).
4. Recent Changes row — Deployment markers from your CI/CD system overlaid on the latency graph. Most incidents are caused by recent changes.

In Grafana, define your dashboard as code using JSON model files committed to git. Never click-to-configure production dashboards — they drift and are unrecoverable after an accidental deletion. Use Grafana's provisioning directory or the grafana-dashboard Terraform resource to deploy them automatically.

Step 4: Write Alerting Rules

Alerts must be actionable. Every alert that fires should have a corresponding runbook entry. If an engineer cannot fix the problem within minutes of reading the alert, the alert is wrong — either it is firing too early, the threshold is wrong, or it lacks a runbook. The multi-window multi-burn-rate alerting strategy (from the Google SRE Workbook) gives you fast detection of severe issues and slow detection of gradual degradation without excessive alert noise:

Step 5: The Observability Plan Document

Combine everything into a single structured document — a living artifact checked into the service repository at docs/observability-plan.md. At big-tech companies this is the document a new on-call engineer reads on their first rotation. It must cover:

• Service overview: What it does, its dependencies, and its traffic patterns (peak RPS, typical latency, batch job schedule).
• SLI catalogue: Every defined SLI with its PromQL query and the reasoning for that specific threshold.
• SLO table: Target, window, error budget per month, and what "SLO breach" means for the business.
• Dashboard index: Links to Grafana dashboards with a one-line description of what each answers.
• Alert catalogue: Every alert, its severity, its burn rate multiplier, and a link to its runbook.
• Data retention policy: How long raw metrics, traces, and logs are kept, and at what resolution (e.g., raw metrics 15 days, downsampled 5-minute aggregates 13 months).
• On-call escalation path: Who gets paged for critical vs. warning alerts, and the escalation chain if they do not acknowledge within N minutes.

Putting It All Together: File Structure

A production-grade observability plan lives in version control alongside the service code. Here is the recommended layout:

This is the standard you hold yourself to as a professional DevOps engineer. An observable service is not one that has Prometheus installed — it is one where, when an incident happens at 2 AM, the on-call engineer opens a dashboard and knows exactly what is broken and why within five minutes. Building that capability intentionally, before incidents happen, is what separates operational excellence from fire-fighting.