Kubernetes Observability

Overview

Adds metrics, dashboards, alerts, and logs while keeping the footprint reasonable for one VPS.

  • kube-prometheus-stack: Prometheus, Alertmanager, Grafana, node-exporter, and kube-state-metrics
  • Loki in single-binary mode: log storage with short retention
  • Grafana Alloy: pod log collection; prefer it over the deprecated Promtail path for a new setup

Start with metrics only. Add Loki after measuring available memory and disk. Traces with Tempo and an OpenTelemetry Collector are optional and should be enabled only for an application that emits traces.

Use Elasticsearch and Kibana instead of Loki when full-text search or Elastic-specific analytics are required. Do not run two log backends by default; see Elasticsearch and the Elastic Stack.

GitOps Settings

Pin chart versions and set explicit retention and resource limits. A sensible starting policy is 7 days of Prometheus metrics, 3–7 days of logs, persistent volumes, and alerts for node disk, memory pressure, pod crash loops, certificate expiry, database availability, and backup failure.

Do not expose Prometheus or Alertmanager publicly. Access Grafana through port-forward, VPN, or authenticated TLS ingress:

kubectl port-forward -n observability svc/kube-prometheus-stack-grafana 3000:80

Open http://localhost:3000. Add PostgreSQL, Redis, and broker exporters or ServiceMonitor resources only for services actually deployed.

Verification

kubectl get pod,pvc,servicemonitor,prometheusrule -n observability
kubectl top node
kubectl top pod -A

In Grafana, confirm Kubernetes dashboards contain node and pod data and Loki Explore returns logs. Test alerts with a temporary safe rule rather than waiting for a real outage.