Kubernetes Scheduling, Resource Management, and RBAC¶

Requests vs. limits: two different jobs, often confused as one¶

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Requests are what the scheduler uses to decide which node a Pod can even go on — it sums up the requests of everything already running on a node and only places a new Pod there if the node has enough unreserved capacity left. Requests are a reservation, not a measurement of actual usage; a Pod requesting 256Mi that only ever uses 50Mi still reserves the full 256Mi against that node's capacity as far as the scheduler is concerned.

Limits are what kubelet enforces at runtime via the same cgroup mechanism Docker uses directly — but CPU and memory limits fail very differently when hit:

CPU is a compressible resource (the kernel can just give a process fewer CPU cycles per second); memory is not (an allocation either succeeds or the process crashes) — this asymmetry is why a memory limit set too low causes hard crashes and restarts, while a CPU limit set too low "just" causes silent slowness that's easy to miss without monitoring specifically for CPU throttling.

QoS classes: which Pods get killed first under memory pressure¶

Kubernetes derives a Quality of Service class from a Pod's requests/limits, entirely automatically:

• Guaranteed — every container's requests equal its limits, for both CPU and memory. Highest priority; last to be killed under node memory pressure.
• Burstable — requests are set, but lower than limits (or limits aren't set at all on some resource). The middle tier — killed before Guaranteed Pods, in order of how far over their request they've grown.
• BestEffort — no requests or limits set at all. Lowest priority; first to be killed the moment the node comes under memory pressure, regardless of how much or little memory it was actually using.

A Pod with no resource specification at all isn't "using the defaults safely" — it's BestEffort, the first thing sacrificed when a node runs low on memory, even if it's the most important workload on that node. Setting requests/limits isn't just about the scheduler's bin-packing; it directly determines eviction priority under pressure.

Steering placement: node affinity and Pod affinity/anti-affinity¶

Requests/limits answer "does this Pod fit here" — affinity rules answer "should this Pod go here even if it fits elsewhere."

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: web
        topologyKey: "kubernetes.io/hostname"

This tells the scheduler: never place two Pods labeled app: web on the same node — spreading replicas across nodes so a single node failure can't take down every replica at once (the actual point of running multiple replicas in the first place, undermined if they all happen to land on the same physical machine anyway). topologyKey can also be topology.kubernetes.io/zone to spread across availability zones instead of individual nodes, for a coarser-grained fault domain.

Node affinity is the analogous rule for a Pod's relationship to node labels rather than to other Pods — pinning a Pod to nodes labeled disktype: ssd, for instance, when only specific nodes have the hardware a workload actually needs.

Taints and tolerations: nodes repelling Pods by default¶

Affinity rules are opt-in attraction; taints are the opposite — opt-out repulsion, applied to the node, not the Pod.

kubectl taint nodes gpu-node-1 gpu=true:NoSchedule

tolerations:
  - key: "gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

Without the matching tolerations entry, no Pod schedules onto gpu-node-1 — the taint repels everything by default. Only Pods that explicitly declare they tolerate this specific taint are considered. This is the standard way to reserve expensive or specialized nodes (GPU nodes, nodes on spot/preemptible instances) for only the workloads that specifically need them, rather than having the scheduler treat them as interchangeable general-purpose capacity.

RBAC: who can do what, to which resources¶

The Secret-reading concern from the previous page is enforced here — RBAC (Role-Based Access Control) is what actually restricts who can call the API server to read a Secret, delete a Deployment, or exec into a Pod.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-secrets-binding
  namespace: production
subjects:
  - kind: ServiceAccount
    name: api-service-account
    namespace: production
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

A Role is a namespaced set of permissions (verbs like get, list, create, delete on specific resource types); a RoleBinding grants that Role to a specific user or ServiceAccount, within that same namespace. ClusterRole/ClusterRoleBinding are the cluster-scoped equivalents — needed for cluster-scoped resources (like Node), or to grant the same permission across every namespace at once. The least-privilege habit: grant exactly the verbs and resource types actually needed, in the narrowest scope (namespaced Role, not ClusterRole, whenever the access doesn't genuinely need to span every namespace).

Common pitfall¶

Omitting resource requests/limits entirely — leaving a Pod at BestEffort QoS — is easy to do (nothing enforces setting them) and easy to miss as the cause when that Pod is mysteriously the first thing killed during a node memory spike caused by something else entirely. If a Pod keeps getting OOM-killed or evicted seemingly at random, check its QoS class (kubectl get pod <name> -o jsonpath='{.status.qosClass}') before assuming the workload itself has a memory leak — it may simply be the node's first eviction target by design.