DaemonSet Has No Running Pods

Runbook for a DaemonSet that has zero available pods, triggered by the DaemonSetNoRunningPods Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). There is no matching Mimir-ruler alert.

This is the DaemonSet sibling of 03-deployment-no-running-replicas.md (which covers Deployments). StatefulSets are not covered by either — they need a separate runbook (CNPG/Kafka PVC-bound pods have different failure characteristics).

Meaning

A DaemonSet runs one pod per node that matches its nodeSelector / tolerations — it is the workload kind for node-level components: Cilium, rook-ceph CSI nodeplugins, node-feature-discovery, the alloy-logs collector, GPU operator / DCGM exporter, spegel. kube_daemonset_status_desired_number_scheduled is the number of nodes the DaemonSet wants to be on; kube_daemonset_status_number_available is the number of pods that are available (Ready + minReadySeconds elapsed).

The alert fires when desired > 0 but available == 0 for 15m — i.e. not a single matching node has a serving pod. The node-level component is not running anywhere it should be.

Both metrics are gauges from kube-state-metrics (deployed by the synapse-client chart, scraped on the cluster and remote-written to synapse-server Mimir). The and on (k8s_cluster_name, environment, namespace, daemonset) matches the series per-DaemonSet. Each leg is pre-aggregated with sum by (...) to strip the scrape-integration labels (job, instance, source), so the alert carries only k8s_cluster_name, environment, namespace, and daemonset and reads as a workload problem — not as a problem with kube-state-metrics itself. Each DaemonSet with zero available pods becomes its own alert instance.

Impact

• The node-level component is not running on any matching node. Because DaemonSets run node-level infrastructure, the blast radius is often wider than a Deployment:
  • Cilium (kube-system/cilium) down → networking broken on every node (no pod-to-pod, no egress). Cluster-wide outage.
  • rook-ceph CSI nodeplugins (rook-ceph.*.csi.ceph.com-nodeplugin) down → PVC mounts cannot be created/expanded; new pods mounting a Ceph PVC stay Pending. Existing mounts keep working until remount.
  • alloy-logs (monitoring-system/synapse-client-alloy-logs) down → pod logs stop shipping to Loki for the whole cluster.
  • node-feature-discovery down → node labels go stale; GPU/feature-based scheduling stops working.
  • GPU operator / DCGM exporter down → GPU workloads cannot schedule or have no metrics.
• 15m of zero-available means the DaemonSet's pods are failing (crash, pull, unschedulable) on every node, or the node pool was drained/cordoned while the DaemonSet has no toleration for the new state.

Diagnosis

The alert labels carry k8s_cluster_name, namespace, daemonset, and environment. From the alert in Grafana, note all of them.

1. Confirm the DaemonSet has zero available pods

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. dev-par01.yaml
NS=<namespace>
DS=<daemonset>
kubectl -n $NS get ds $DS
kubectl -n $NS get pods -l app=<daemonset> -o wide   # or the DaemonSet's pod-template label

kubectl get ds shows DESIRED / CURRENT / READY / UP-TO-DATE / AVAILABLE. Confirm AVAILABLE 0 and DESIRED > 0. The pods will be in Pending, CrashLoopBackOff, ImagePullBackOff, Running 0/1 (not Ready), or Node unavailable states.

2. Distinguish "no matching node" from "pods failing"

Check whether the DaemonSet's nodeSelector / tolerations match any node:

kubectl -n $NS get ds $DS -o jsonpath='nodeSelector={.spec.template.spec.nodeSelector}{"\n"}tolerations={.spec.template.spec.tolerations}{"\n"}'
# How many nodes match the selector?
kubectl get nodes --selector='<selector-key>=<selector-value>' -o wide

• No node matches the selector → DESIRED 0 is correct (this alert does not fire — see NOTE above). If DESIRED > 0 but no node matches, the selector is wrong or the node labels changed. Fix the selector or restore the labels in a chart-values PR.
• Nodes match but pods are Pending → unschedulable / taints. Go to step 3.
• Pods are failing (CrashLoop/ImagePull) → Go to step 4.
• Pods are Running 0/1 → readinessProbe failing. Go to step 5.

3. Pods are Pending / unschedulable

A DaemonSet pod is Pending when the node cannot accept it — usually a taint the pod does not tolerate, or a resource the node does not have:

kubectl -n $NS describe pod <pod> | sed -n '/^Events:/,$p'
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key,CORDONED:.spec.unschedulable'

Common causes:

• Taint mismatch — a node was tainted (e.g. node.kubernetes.io/disk-pressure, nvidia.com/gpu) and the DaemonSet's tolerations do not cover it. A node-problem-DaemonSet (Cilium, logs) should tolerate most taints; if it does not, add the toleration in a chart-values PR.
• Resource short — the DaemonSet pod requests more CPU/memory than any node has spare. DaemonSet pods are usually small; a sudden request bump in the chart can break this.
• All nodes cordoned — kubectl get nodes shows SchedulingDisabled. Uncordon or scale the node pool.

4. Pods are CrashLoopBackOff / ImagePullBackOff

If pods are in a crash or pull-failure state, the root cause is in the pod, not the DaemonSet scheduling. Switch to the sibling runbook for the specific reason:

kubectl -n $NS get pods -l app=<daemonset> -o jsonpath='{range .items[*]}{.metadata.name}: {range .status.containerStatuses[*]}{.state.waiting.reason}{" "}{end}{"\n"}{end}'

• ErrImagePull / ImagePullBackOff → 07-kubernetes/01-pod-imagepull.md (a bad image tag in a recent chart bump is the most common cause — the DaemonSet rolled out and every node's pod failed to pull).
• CrashLoopBackOff → 07-kubernetes/02-pod-crashloop.md (a config / env-var / secret change, or a version that crashes on startup).

The DaemonSet will recover automatically once the underlying pod issue is fixed (the pods reach Ready on every node).

5. Pods are Running but not Ready (failing readinessProbe)

If pods are Running 0/1 (not Ready):

kubectl -n $NS describe pod <pod> | sed -n '/^Events:/,$p'   # look for "Readiness probe failed"
kubectl -n $NS logs <pod> -c <container> --tail=30

The readinessProbe is failing on every node — the component is up but not healthy (e.g. Cilium cannot reach the apiserver, the CSI plugin cannot reach the Ceph monitors, the logs collector cannot reach the Loki gateway). A bad rollout (config change, upstream dependency down) can make every pod fail its probe. Fix the probe target or the dependency, then the pods go Ready.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the root cause

This alert is always a symptom — the root cause is in the pods or the node matching. Follow Diagnosis steps 2–5:

• Selector / node labels wrong — fix the DaemonSet's nodeSelector or restore the node labels in a chart-values PR.
• Taint mismatch — add the toleration to the DaemonSet in a chart-values PR.
• Image pull / crash — see 07-kubernetes/01-pod-imagepull.md and 02-pod-crashloop.md. For a bad image-tag rollout, rolling back the chart tag (or pushing the missing tag) is the fast path; the DaemonSet controller re-rolls-out to every node once the tag pulls.
• Readiness probe failing — fix the probe or the component's dependency (chart-values PR for initialDelaySeconds, or fix the upstream dependency the probe checks).

Step 3 — Roll back to the last good DaemonSet revision (emergency)

If the new DaemonSet revision is bad and the fix will take too long, roll back to the previous revision:

kubectl -n $NS rollout undo ds $DS
kubectl -n $NS rollout status ds $DS

rollout undo flips the DaemonSet to the previous template; the controller re-rolls-out the old (working) pods to every node. This is a temporary hold — ArgoCD self-heal will reconcile on the next sync, so make the chart-values change match (roll back the tag) in a PR so GitOps and the live state agree.

Step 4 — Verify the fix

After the root cause is fixed, pods reach Ready on every matching node and kube_daemonset_status_number_available rises to match desired. The alert clears once available > 0 sustained for the for: 15m window (practically, within a few scrape intervals after the first node's pod is Ready, then as the rollout progresses).

Verify

• The alert clears once kube_daemonset_status_number_available > 0 for the DaemonSet, sustained for the for: 15m window. (Full convergence to available == desired may take longer as the rollout hits every node; the alert clears as soon as the first pod is available, which is the right semantics — the component is no longer "zero running".)
• kubectl -n $NS get ds $DS shows AVAILABLE > 0 and rising toward DESIRED.
• kubectl -n $NS get pods -l app=<daemonset> shows pods Running and Ready on matching nodes.
• For a readinessProbe fix, kubectl describe pod shows no recent Readiness probe failed events.