Prometheus Alerting

Alerting rules and the for duration

groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for 10 minutes"

An alerting rule is a PromQL expression (reusing exactly the error-rate query from the PromQL page) evaluated on a schedule; when it's true, the alert enters Pending, not immediately Firing.

stateDiagram-v2 Inactive --> Pending: expression becomes true Pending --> Firing: stays true for the full 'for' duration Pending --> Inactive: expression becomes false before 'for' elapses Firing --> Inactive: expression becomes false

The for: 10m duration exists specifically to avoid firing on a transient blip — a 30-second error spike that resolves on its own never reaches Firing, staying in Pending and dropping back to Inactive once the condition clears. Only a condition that holds true for the entire for duration actually fires and notifies anyone — this is the direct mechanism against alert flapping, and setting for too short (or omitting it) reintroduces exactly the noisy, low-signal alerting this mechanism is meant to prevent.

Alertmanager: dedup, grouping, routing, silencing

graph LR p1["Prometheus 1"] --> am["Alertmanager"] p2["Prometheus 2"] --> am am --> group["Group related alerts<br/>(e.g. by cluster + alertname)"] group --> route{"Route by label match"} route -->|"severity=critical"| pd["PagerDuty (pages on-call)"] route -->|"severity=warning"| slack["Slack channel"]

Prometheus itself only evaluates rules and sends firing alerts onward — Alertmanager is the separate component that decides what actually happens with them:

  • Deduplication — the same alert firing from multiple Prometheus replicas (a common HA setup) is collapsed into one notification, not duplicated per replica.
  • Grouping — many related alerts (every instance in a cluster hitting the same threshold at once, during a real outage) are bundled into a single notification instead of flooding the on-call channel with one message per instance — critical for avoiding the same alert-fatigue problem a notification system has to solve for user-facing notifications, just applied to operational alerts instead.
  • Routing — different alerts go to different receivers based on label matching (severity: critical pages on-call via PagerDuty; severity: warning just posts to a Slack channel), so a genuinely urgent issue and an FYI-level one don't compete for the same attention channel.
  • Silencing — temporarily mute alerts matching a label selector (during planned maintenance, or while a known issue is already being worked), without disabling the underlying alerting rule itself.

Common pitfall

Writing alerting rules for every metric that could indicate a problem, rather than for symptoms that actually require a human to act right now, produces alert fatigue — a stream of low-value pages trains on-call engineers to skim and dismiss notifications, which is exactly the failure mode that causes a genuinely critical alert to get missed in the noise. A useful discipline: every alerting rule should map to an answerable "what does the on-call engineer actually do when this fires" — if the honest answer is "nothing, it's just informational," that belongs on a dashboard, not as a page-worthy alert.