Prometheus Fundamentals

Pull, not push

graph LR prom["Prometheus server"] -->|"scrapes /metrics every N seconds"| t1["Target 1"] prom -->|"scrapes"| t2["Target 2"] prom -->|"scrapes"| t3["Target 3"]

Prometheus pulls metrics — the server scrapes an HTTP /metrics endpoint on each target at a configured interval, rather than targets pushing metrics to a central collector (the StatsD/Graphite model). This has a specific, deliberate consequence: a scrape that fails is itself a signal — Prometheus knows immediately that a target is down or unreachable, because the absence of a response is directly observable, unlike a push model where a silent target and a target that simply has nothing new to report look identical from the collector's side. It also means an application only needs to expose its current metric values on demand — it never needs to know where to send them, handle push failures, or buffer metrics if a collector is temporarily unreachable.

The data model: metric name + labels

Every time series is uniquely identified by a metric name plus a set of key-value labels:

http_requests_total{method="GET", status="200", handler="/api/users"} 8021
http_requests_total{method="POST", status="500", handler="/api/users"} 12

These are two different time series (different label sets), even though they share a metric name — every unique combination of label values is tracked as its own independent series. This is what makes labels powerful (slice and filter by any dimension in a query) and also what makes them dangerous — see the cardinality pitfall below.

The four metric types

graph TD counter["Counter<br/>only increases (or resets to 0 on restart)<br/>e.g. total requests served"] gauge["Gauge<br/>goes up AND down<br/>e.g. current memory usage, queue depth"] histogram["Histogram<br/>observations sorted into buckets<br/>e.g. request latency distribution"] summary["Summary<br/>like histogram, but quantiles<br/>computed client-side, not aggregatable"]
  • Counter — monotonically increasing, used for anything counted cumulatively (requests served, errors, bytes sent). Never graph a raw counter directly — see PromQL for why, and how rate() fixes it.
  • Gauge — a value that can move in either direction (current memory usage, number of items in a queue, temperature). Graphed directly, no transformation needed.
  • Histogram — buckets observations into configurable ranges (le="0.1", le="0.5", le="1", le="+Inf" — cumulative counts: the le="0.5" bucket includes everything the le="0.1" bucket does, plus more). Used to compute percentiles/quantiles at query time via histogram_quantile(), and — critically — histogram buckets from multiple instances can be summed together before computing a quantile, because the raw bucket counts are simple counters.
  • Summary — computes quantiles inside the application at observation time, before Prometheus ever sees them. The real limitation: a summary's precomputed quantile from one instance cannot be meaningfully averaged or combined with another instance's — averaging two different p99s doesn't produce the true p99 across both instances combined. Histograms avoid this because the aggregation (summing buckets) happens before the quantile calculation, not after.

Common pitfall

Using a high-cardinality value as a label — a raw user ID, a full URL path with query parameters, a UUID — causes a cardinality explosion: each unique label value combination is a distinct time series that Prometheus has to store and index in memory. A million distinct user IDs used as a label value means a million time series for that one metric, potentially overwhelming Prometheus's memory and query performance — this is the single most common way a Prometheus deployment degrades in production, and it's easy to introduce accidentally by labeling something that felt reasonably low-cardinality at first ("just the handler path") until URLs with embedded IDs (/users/12345) turn out to generate a new label value per user instead of one per route.