PromQL¶

Instant vectors vs. range vectors¶

http_requests_total alone is an instant vector — the current value of every matching time series, one number each, at the moment the query runs. http_requests_total[5m] is a range vector — every sample recorded for each matching series over the last 5 minutes, not just one value. Functions like rate() consume a range vector and produce an instant vector back out (one rate value per series, computed from that series' samples over the window).

rate(): turning a counter into a rate¶

A raw counter is nearly useless graphed directly — it only climbs, and it resets to 0 whenever the process restarts, producing a misleading sawtooth rather than a meaningful trend. rate(metric[5m]) computes the per-second average rate of increase over the window, and — this is the part that matters — it specifically detects and correctly handles counter resets.

Tracing rate() across this sequence: from t=0 to t=60s, the counter rose from 100 to 160 — a normal increase, rate ≈ 1/sec. From t=60s to t=120s, the counter appears to decrease (160 → 15) — rate() recognizes a decrease as a restart-induced reset (a counter can only go up during normal operation) and treats the actual increase since the reset as 15 - 0 (not 15 - 160, which would be nonsensical negative), correctly reporting a small positive rate for that interval instead of a meaningless negative one. This reset-handling is the entire reason rate() exists as a dedicated function rather than a simple (end - start) / duration calculation, which would silently produce wrong (negative) numbers across every restart.

irate() is the same idea but uses only the last two data points in the range instead of averaging across the whole window — more responsive to sudden spikes, but noisier; rate() is the standard default for alerting and dashboards specifically because smoothing out noise matters more than millisecond responsiveness for those use cases.

Aggregation: sum, avg, by, without¶

sum(rate(http_requests_total[5m])) by (status)

Computes the per-second request rate, then sums it, grouped by the status label — collapsing every other label (like handler, method) into one number per status code, across all instances. by (status) keeps only that label in the output; without (status) is the inverse — keep every other label, drop just this one.

Error rate, a canonical pattern combining this with a label filter:

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

The numerator sums the rate of 5xx responses (=~"5.." is a regex label matcher); the denominator sums the rate of all responses; their ratio is the error rate as a fraction, ready to multiply by 100 for a percentage or feed directly into an alerting rule.

histogram_quantile(): percentiles from bucket data¶

histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Given cumulative bucket counts (le="0.1" → 950 requests, le="0.5" → 990, le="1" → 998, le="+Inf" → 1000, out of 1000 total), this estimates the value below which 99% of observations fall — by finding which bucket boundary the 99th percentile lands between and linearly interpolating within that bucket (it's an estimate, not an exact value, because the raw individual observations aren't stored, only which bucket each fell into). Summing bucket counts across instances before computing the quantile (as shown, via sum(...) by (le)) is exactly the aggregation Summary metrics can't support — this is the concrete PromQL expression of why histograms were the right metric type choice for anything that needs a fleet-wide percentile.

Common pitfall¶

Using a rate() window shorter than roughly 4x the metric's scrape interval produces noisy or gap-riddled results — if Prometheus scrapes every 15 seconds and a query uses rate(metric[15s]), a single missed or delayed scrape leaves that window with only one data point, which isn't enough to compute a meaningful rate (rate needs at least two samples in the window to measure a change). A common convention is a range at least 4x the scrape interval ([1m] for a 15s scrape interval, [5m] as a common safe default) to stay resilient to an occasional missed scrape without the query going blank.