Kubernetes Health Checks and Observability¶

The problem: "the process is running" isn't the same as "it's working"¶

kubelet, by default, only knows whether a container's main process has crashed — the same limitation a bare Docker container has without a HEALTHCHECK. A process can be alive, consuming CPU, and completely unable to serve a single valid request — deadlocked, stuck waiting on a dependency that will never respond, or still finishing a slow startup sequence — and kubelet has no way to tell the difference without being explicitly told how to check. Kubernetes has three distinct probe types, precisely because "is it alive" and "is it ready for traffic" and "has it finished starting" turn out to need genuinely different consequences when they fail.

The three probes, and what happens when each fails¶

containers:
  - name: api
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 10
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 5
      failureThreshold: 2
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      failureThreshold: 30
      periodSeconds: 2

• livenessProbe — "should this container be restarted?" A failure here is a destructive action: the container is killed and recreated. Reserve this for genuinely unrecoverable states — a deadlock, a corrupted internal state that only a fresh restart fixes — not for anything that might recover on its own.
• readinessProbe — "should traffic be routed here right now?" A failure here is non-destructive: the Pod is pulled out of the Service's routing rotation (the same endpoint mechanism from Services and Networking) but the container keeps running untouched, and rejoins rotation the moment the check passes again.
• startupProbe — runs before liveness/readiness are considered at all, with its own (typically longer) failure budget (failureThreshold: 30 at periodSeconds: 2 here gives a full 60 seconds to finish starting). Once it succeeds once, it's done for the Pod's lifetime — liveness/readiness take over from there.

The trap: using the same check for liveness and readiness¶

This is the single most common real-world Kubernetes probe misconfiguration, and it's easy to fall into because it looks economical — one /health endpoint, wired to both probes.

Consider an API whose /health endpoint checks database connectivity. The database has a brief, transient blip — a few seconds of connection issues, nothing that indicates the API process itself is broken.

• Correct behavior wanted: readiness fails, traffic stops routing here for those few seconds, the Pod rejoins rotation once the DB recovers. Graceful, temporary, no restart needed — the process itself was never the problem.
• What actually happens with a shared check wired to liveness too: livenessProbe also starts failing (same endpoint, same DB dependency) — and after failureThreshold consecutive failures, kubelet kills and restarts the container. Restarting the API process does nothing to fix a database problem; the new container boots, immediately hits the same still-broken DB check, and gets killed again — a crash-loop caused entirely by a dependency issue that the application itself had no part in, and that restarting never could have fixed.

The fix is intentionally asymmetric: liveness should check only "is this process internally broken" (ideally with no external dependency calls at all — an in-memory check, or nothing more than "can this handler respond"), while readiness can and should check external dependencies, because pulling traffic away during a dependency blip is exactly the correct, safe response — restarting is not.

Observability: logs¶

kubectl logs my-pod -f
kubectl logs my-pod -c specific-container   # multi-container Pod

Exactly like Docker's stdout/stderr convention, kubectl logs only has something to show if the application writes to stdout/stderr rather than to an internal log file. At cluster scale, manually running kubectl logs per Pod doesn't scale past a handful of Pods — the standard pattern is a log-collection agent running as a DaemonSet (Fluent Bit, Fluentd, or similar), one per node, reading every container's stdout/stderr from the node's local log files and shipping it to a central store — the exact "one Pod per node, always" shape a DaemonSet exists for.

Observability: metrics¶

Two separate metrics paths, often both present: resource metrics (CPU/memory actually used per Pod, surfaced through kubelet's built-in cAdvisor, aggregated by metrics-server) power kubectl top pods and the HorizontalPodAutoscaler — which is itself just another reconciliation loop: compare observed CPU utilization against the target percentage of requested CPU, and adjust replica count to close the gap. Application metrics (request counts, latency histograms, custom business metrics) are a separate concern entirely — an app exposing its own /metrics endpoint (commonly in Prometheus's text format), scraped directly by a Prometheus server, unrelated to what kubelet/metrics-server track.

Common pitfall¶

Setting initialDelaySeconds too low on a liveness probe for an application with real startup work to do (loading a large cache, running migrations) causes kubelet to start checking — and potentially kill and restart — a container that's still legitimately initializing, before it ever gets a chance to finish. A startupProbe with a generous failure budget is the correct fix, rather than just inflating initialDelaySeconds on the liveness probe itself, which either still isn't generous enough for a slow cold start or becomes unnecessarily slow to detect a real post-startup hang once the container is actually up and running.