Metric Types & Exposition

Metric Types & Exposition

Prometheus defines four fundamental metric types, each modeling a different aspect of system behavior. Choosing the wrong type is one of the most common mistakes new users make — a mislabeled metric will produce subtly wrong query results in ways that are hard to debug under production pressure. This lesson covers what each type is, when to use it, how its wire format works, and what breaks when you get it wrong.

Counter

A counter is a monotonically increasing integer or float that only goes up (or resets to zero on process restart). It is the right choice for any total-since-start measurement: requests served, bytes sent, errors encountered, tasks completed.

The critical rule: never use a counter for something that can decrease. Current queue depth is not a counter. CPU temperature is not a counter. If you find yourself needing to subtract two counter readings, you are using the type correctly — rate() and increase() are designed for exactly that.

Naming convention: counters always end in _total. For example: http_requests_total, process_cpu_seconds_total, rpc_errors_total.

Gauge

A gauge is a value that can go up and down arbitrarily. It represents a snapshot of a current state. Examples: memory in use, number of goroutines, active connections, current temperature, queue depth, cache hit ratio, disk usage percentage.

Gauges are used with functions like avg_over_time(), max_over_time(), and plain arithmetic — no rate needed. They are also the right type for configuration values, version numbers (via label tricks), and boolean state encoded as 0/1.

Histogram

A histogram samples observations and counts them into configurable buckets. It also tracks the running total count and sum of all observed values, enabling accurate percentile estimation and average computation across time series aggregations.

A single histogram named http_request_duration_seconds automatically creates three exposed series:

• http_request_duration_seconds_bucket{le="0.1"} — count of requests that finished in ≤ 100ms
• http_request_duration_seconds_count — total number of observations
• http_request_duration_seconds_sum — sum of all observed durations

The bucket label le (less-than-or-equal) is mandatory. Prometheus always appends an le="+Inf" bucket that equals _count. Percentile estimation uses histogram_quantile(), which performs linear interpolation within a bucket — accuracy depends entirely on bucket placement relative to where your data actually falls.

Default bucket recommendations for HTTP latency (.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10 seconds) suit typical web services. For internal RPC or database calls, shift the buckets an order of magnitude lower. Define buckets in your instrumentation before first deployment — changing them retroactively breaks existing dashboards and recording rules.

Summary

A summary also samples observations but computes configurable quantiles on the client side over a sliding time window. It exposes pre-computed quantile labels like {quantile="0.99"} alongside _count and _sum.

The key tradeoff versus histograms:

• Summaries are accurate at the quantile value but cannot be aggregated across instances. sum(rate(...)) works on _count and _sum, but you cannot meaningfully average the {quantile="0.99"} series from 50 pods — the result is statistically nonsensical.
• Histograms are approximate (bucket-interpolated) but fully aggregatable. You can call histogram_quantile() across the merged buckets from all instances to get a fleet-wide percentile.

The Exposition Format

Prometheus scrapes metrics over HTTP from the /metrics endpoint. The wire format is a plain-text line protocol defined in the OpenMetrics specification. Each metric family is prefixed with a HELP line (human description) and a TYPE declaration, followed by one data line per time series.

Key format rules: metric names must match [a-zA-Z_:][a-zA-Z0-9_:]*. Label values are UTF-8 strings quoted with double quotes; internal backslashes and double-quotes are escaped. A timestamp (Unix milliseconds) is optionally appended as a third field but is rarely used — Prometheus tracks its own scrape timestamp. Comment lines start with #; any other line starting with # is invalid.

To inspect any target's raw exposition output, curl it directly. This is your first debugging step whenever a metric is missing from a dashboard:

Instrumentation Best Practices at Scale

At Google, Lyft, and similar companies, the instrumentation contract is part of the service's API. A few hard-won rules:

• Use base units always: seconds, bytes, ratios (0–1). Never milliseconds or megabytes — PromQL and Grafana assume SI base units, and mixing them creates dashboard scaling errors that waste on-call hours.
• Label cardinality kills TSDB: Never use a user ID, IP address, request URL, or any unbounded value as a label. Each unique label set creates a new time series in Prometheus's TSDB. A service with 100k users exposing {user_id=...} will OOM a Prometheus server within hours. Cap label cardinality at a few hundred distinct values per label.
• Prefix metric names with the subsystem: myservice_http_requests_total not requests_total. The prefix survives federation and remote-write pipelines where metrics from many services merge.
• Expose a build-info gauge: A myservice_build_info{version="1.4.2", commit="a3f91c"} 1 gauge (always value 1) enables version-aware alerting and rollback correlation in dashboards without requiring log scraping.