Runbook for a node under Memory/Disk/PID pressure, triggered by the NodePressure Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). This mirrors the Datadog [kubernetes] Kubelet is evicting pods on host alert but uses the stable node-condition signal rather than the transient eviction event.

Meaning

The kubelet watches node-level resource pressure and sets NodeConditions to signal it. When a resource is pressured, the kubelet evicts pods (gracefully terminating lower-priority pods) to relieve the pressure:

MemoryPressure — node memory is below the kubelet's evictionHard.memoryAvailable threshold. The kubelet evicts pods by OOM-score (best-effort pods first, then burstable) to reclaim memory.
DiskPressure — node filesystem is below the evictionHard.nodefs.available / imagefs.available threshold. The kubelet garbage-collects unused images and logs, then evicts pods to reclaim disk.
PIDPressure — pid exhaustion (too many processes on the node). The kubelet evicts pods to reclaim pids.

kube_node_status_condition is a gauge from kube-state-metrics (deployed by the synapse-client chart, scraped on the cluster and remote-written to synapse-server Mimir). It is 1 when the condition is true, 0 otherwise, with node, condition, and status labels (one series per condition×status state). The alert matches condition=~"MemoryPressure|DiskPressure|PIDPressure",status="true" — i.e. any of the three pressure conditions is true. The expr is pre-aggregated with sum by (k8s_cluster_name, environment, node, condition) to strip the scrape-integration labels (job, instance, source), so the alert carries only k8s_cluster_name, environment, node, and condition and reads as a node problem — not as a problem with kube-state-metrics itself. Each pressured node+condition becomes its own alert instance (the condition label distinguishes them).

Why node-condition, not the eviction event

The Datadog alert fired on the eviction event (kubelet evicted pod). Evictions are transient — the kubelet evicts a pod, the pressure drops, the condition clears, and the pod reschedules. Alerting on the eviction event flaps (the pod is gone in seconds) and doesn't tell you the node is still under pressure. The node condition is stable: MemoryPressure=true stays true until the kubelet has reclaimed enough, so the alert catches a node that is persistently pressured, which is the actionable signal (free the resource or add capacity). The eviction is a symptom; the condition is the cause.

Impact

The kubelet is evicting pods from this node. Evicted pods reschedule to other nodes (if the workload tolerates it), so the impact is usually pod churn + rescheduling, not a permanent outage.
MemoryPressure — pods are OOMKilled (the node's cgroup OOM killer, or the kubelet's eviction OOM-kill). Stateful pods (CNPG, Kafka) evicted from their node lose their in-memory state and must restart; their PVC survives but the pod restarts. See 07-kubernetes/06-statefulset-no-running-replicas.md if a stateful pod is evicted and cannot reschedule.
DiskPressure — the node filesystem is full. The kubelet cannot pull new images (image pulls fail → ErrImagePull on the node), cannot write logs, and may fail pod starts. If the node's disk is the Ceph/rook-ceph OSD disk, this can cascade to storage issues (see 04-storage/).
PIDPressure — pid exhaustion; new processes cannot spawn (pods fail to start, exec fails).
If the pressure is on a stateful-workload node, evictions can cascade: CNPG replicas evicted → replication alert; Kafka brokers evicted → partition reassignment.
If only one node is pressured, the cluster absorbs it (pods reschedule). If many nodes are pressured (e.g. a node pool is undersized for a load spike), the cluster cannot reschedule and workloads go pending.

Diagnosis

The alert labels carry k8s_cluster_name, node, condition (which pressure), and environment. From the alert in Grafana, note all of them.

1. Confirm the condition and identify the node

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NODE=<node>
kubectl get node $NODE -o wide
kubectl describe node $NODE | sed -n '/^Conditions:/,/Taints:/p'
kubectl describe node $NODE | sed -n '/^Allocated resources:/,$p'
kubectl describe node $NODE | sed -n '/^Non-terminated Pods:/,$p' | head -40

The Conditions section shows which pressure conditions are True and since when. Allocated resources shows CPU/memory requests vs capacity. Non-terminated Pods lists the pods on the node (and their requests) — the biggest consumers are the eviction candidates.

2. MemoryPressure — what is consuming node memory

kubectl describe node $NODE | sed -n '/^Allocated resources:/,$p'   # total requests vs capacity
kubectl get pods -A --field-selector spec.nodeName=$NODE -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,MEM:.spec.containers[*].resources.requests.memory'

Common causes:

A single large pod — a pod with a high memory request (or no limit, consuming all spare) dominates the node. Check kubectl top pods -A --field-selector (if available) or the cAdvisor memory metrics (container_memory_working_set_bytes per pod on this node).
Too many pods — the node is over-scheduled (sum of requests near capacity). A bad rollout or a missing limit let many pods land on the node.
Memory leak — a pod's memory grows without bound (a leak) and consumes the node. The pod's container_memory_working_set_bytes trends up; the OOMKill eventually fires (see 07-kubernetes/02-pod-crashloop.md).
Node too small — the node pool's memory is genuinely insufficient for the workloads scheduled on it. Add capacity or move workloads to a larger pool.

3. DiskPressure — what is consuming node disk

# node-level disk (if you can reach the node or via node-problem-detector):
# df -h  on the node
# image vs filesystem:
kubectl describe node $NODE | grep -A5 "Images:"
# kubelet garbage collection status:
kubectl -n kube-system logs kubelet-server-$NODE 2>/dev/null | grep -iE "garbage|gc|image" | tail -10

Common causes:

Image buildup — too many image layers (frequent deploys, no image GC). The kubelet's image GC should reclaim this; if DiskPressure persists, GC is not keeping up (threshold misconfigured, or images are pinned by running pods).
Container logs — pods writing to the container log directory (not stdout) fill the node disk. Fix the pod to log to stdout, or add log rotation.
EmptyDir / local storage — a pod's emptyDir or hostPath is filling the node disk. Find it: kubectl get pods -A --field-selector spec.nodeName=$NODE -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.spec.volumes[*].emptyDir}{"\n"}{end}'. An unbounded emptyDir can fill the node.
Ceph/rook-ceph OSD disk — if the node is a storage node, the OSD disk filling can cause DiskPressure and cascade to the Ceph cluster. See 04-storage/01-ceph-osd-full-bluefs-wedge.md.

4. PIDPressure — what is consuming pids

# on the node: ps -e | wc -l   (or via node-problem-detector)
kubectl get pods -A --field-selector spec.nodeName=$NODE -o custom-columns='NS:.metadata.namespace,POD:.metadata.name'

PIDPressure is rare; usually a single pod spawning many processes (a fork bomb, a misconfigured worker pool, a leak). Find the pod with the highest process count and fix its config, or add node pid capacity.

5. Check for evicted pods (the symptom)

kubectl get pods -A --field-selector spec.nodeName=$NODE,status.phase=Failed -o wide | grep -i evicted
kubectl get pods -A --field-selector spec.nodeName=$NODE -o wide | grep -iE "Evicted|OOMKilled"

Evicted pods are the symptom (not the cause). They reschedule to other nodes if the cluster has capacity; if they keep getting evicted back to this node, the node is the problem (free the resource or cordon it).

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Free the resource on the node

Based on Diagnosis:

MemoryPressure — a runaway pod — identify the high-memory pod and restart/resize it. If it's a leak, the pod must be restarted (and the app fixed). A chart-values PR to add/raise a memory limit prevents recurrence. See 07-kubernetes/02-pod-crashloop.md for OOMKill handling.
MemoryPressure — over-scheduled node — move some pods off (cordon the node so new pods avoid it; existing pods reschedule on the next evict/restart). Or add node capacity (scale the node pool).
DiskPressure — image buildup — the kubelet's image GC should reclaim; if not, restart the kubelet GC or manually remove unused images on the node (crictl rmi --prune if you have node access). A chart-values PR to tune evictionHard.imagefs.available if GC thresholds are wrong.
DiskPressure — emptyDir/local logs — fix the pod (log to stdout, bound the emptyDir, add an emptyDir sizeLimit). Chart-values PR.
DiskPressure — Ceph OSD — see 04-storage/ runbooks; the node disk issue may cascade to storage.
PIDPressure — fix the pod spawning too many processes; chart-values PR.

Step 3 — Add node capacity (if the node is genuinely undersized)

If the node is persistently pressured because it is too small for its workloads, add capacity:

Scale the node pool up (if autoscaled) — more nodes spread the load.
Move to a larger node pool — adjust the workload's node selector / tolerations in a chart-values PR so it lands on a bigger pool.

Step 4 — Cordon the node (emergency hold)

If the node is in a bad pressure loop (evicting pods that immediately reschedule back), cordon it so no new pods land on it while you fix the resource:

kubectl cordon $NODE
# pods already on the node keep running (or get evicted by the pressure);
# no new pods schedule here until uncordoned.

Drain it only if the workloads can reschedule elsewhere and you are sure the node needs to be emptied:

kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data   # force evict
# fix the node, then:
kubectl uncordon $NODE

For an ArgoCD/Kommodity-managed node pool, cordon/drain is a manual hold — the cluster autoscaler / Kommodity will reconcile node counts based on the chart config. Make the capacity change in a chart-values PR so the pool is durably right-sized.

Verify

The alert clears once kube_node_status_condition{condition=~"MemoryPressure|DiskPressure|PIDPressure",status="true"} == 0 for the node, sustained for the for: 5m window. Practically it clears within a scrape interval after the condition flips to false.
kubectl get node $NODE shows the pressure condition False and Ready (the Ready condition should be True once pressure clears).
kubectl describe node $NODE shows no recent Status changes to pressure conditions.
Evicted pods have rescheduled to other nodes and are Running (kubectl get pods -A --field-selector spec.nodeName=$NODE shows the survivors; the rescheduled pods are on other nodes).
For a capacity fix (more/larger nodes), the node pool has the new capacity and pods are no longer over-scheduled (kubectl describe node <new-node> shows balanced Allocated resources).