Runbook for a Deployment that has zero Ready replicas, triggered by the DeploymentNoRunningReplicas Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). This mirrors the Datadog [platform/availability] Deployment has no running replicas alert. There is no matching Mimir-ruler alert.

Meaning

A Kubernetes Deployment owns a ReplicaSet, which owns pods. kube_deployment_spec_replicas is the desired replica count (from spec.replicas). kube_deployment_status_replicas_ready is the number of pods that have passed their readinessProbe and are Ready. The alert fires when desired > 0 but ready == 0 — i.e. not a single pod is serving — for 15m.

Both metrics are gauges from kube-state-metrics (deployed by the synapse-client chart, scraped on the cluster and remote-written to synapse-server Mimir). The and on (k8s_cluster_name, environment, namespace, deployment) matches the two series per Deployment so the condition is per-Deployment. Each leg is pre-aggregated with sum by (...) to strip the scrape-integration labels (job, instance, source), so the alert carries only k8s_cluster_name, environment, namespace, and deployment and reads as a workload problem — not as a problem with kube-state-metrics itself. Each Deployment with zero ready replicas becomes its own alert instance.

This is distinct from a stuck rollout (spec - updated > 0), where some old replicas are still up but the new ReplicaSet has not finished rolling out — the service is still serving, just on the old version. That condition is milder and not what this alert catches. See Diagnosis step 2 to distinguish the two.

Impact

The Deployment has no serving pods. If it fronts user traffic (via Service + Ingress/HTTPRoute), requests fail — the service is down from the client's perspective.
If the Deployment is a worker (no traffic), its work is not being done — queues back up, scheduled jobs do not run.
15m of zero-ready means a fresh deploy failed to come up, or all running pods were lost (node loss, eviction, crash loop) and the new pods never reached Ready.
This is the availability condition Datadog alerts on; treat as a service-down for the owning team.

Diagnosis

The alert labels carry k8s_cluster_name, namespace, deployment, and environment. From the alert in Grafana, note all of them.

1. Confirm the Deployment has zero ready pods

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=<namespace>
DEPLOY=<deployment>
kubectl -n $NS get deploy $DEPLOY
kubectl -n $NS get rs -l app=<deployment>   # ReplicaSets owned by the Deployment
kubectl -n $NS get pods -l app=<deployment> -o wide

kubectl get deploy shows READY as 0/N — confirm. The pods will be in one of: Pending, CrashLoopBackOff, ImagePullBackOff, ErrImagePull, Running but not Ready (failing readinessProbe), or Terminating.

2. Distinguish from a stuck rollout

Check whether the Deployment is mid-rollout (some old pods up, new ReplicaSet at 0 ready):

kubectl -n $NS get deploy $DEPLOY -o jsonpath='desired={.spec.replicas} updated={.status.updatedReplicas} ready={.status.readyReplicas} available={.status.availableReplicas}{"\n"}'

ready == 0 AND updated == 0 → all pods are gone (this alert's condition; fresh deploy failed, or all pods lost). Go to step 3.
ready == 0 AND updated == desired → the new ReplicaSet's pods are all not-Ready (failing readinessProbe). Go to step 4.
ready > 0 but updated < desired → a stuck rollout (old pods still serving, new ReplicaSet not progressing). Not this alert — the service is still up. Check the new ReplicaSet's pods for the rollout blocker (image pull, crash, etc.).

3. Pods are Pending / unschedulable

If pods are Pending:

kubectl -n $NS describe pod <pod> | sed -n '/^Events:/,$p'

Common causes:

Unschedulable (0/N nodes are available) — not enough CPU/memory/GPU, taints no node can satisfy, or all nodes cordoned. Check kubectl get nodes for cordoned nodes and kubectl describe pod for the scheduler's reason. Free capacity (scale the node pool, or lower the pod request) or fix the taints.
PVC Pending — a volumeClaimTemplate cannot bind (storage class missing, no capacity). See 04-storage/04-pvc-near-full.md if storage is full; otherwise check the PVC event.
ConfigMap / Secret not found — a volume mount references a missing resource. Confirm it exists in the namespace.

4. Pods are CrashLoopBackOff / ImagePullBackOff

If pods are in a crash or pull-failure state, the root cause is in the pod, not the Deployment. Switch to the sibling runbook for the specific reason:

kubectl -n $NS get pods -l app=<deployment> -o jsonpath='{range .items[*]}{.metadata.name}: {range .status.containerStatuses[*]}{.state.waiting.reason}{" "}{end}{"\n"}{end}'

ErrImagePull / ImagePullBackOff → see 07-kubernetes/01-pod-imagepull.md.
CrashLoopBackOff → see 07-kubernetes/02-pod-crashloop.md.

The Deployment will recover automatically once the underlying pod issue is fixed (the pods reach Ready).

5. Pods are Running but not Ready (failing readinessProbe)

If pods are Running 0/1 (not Ready):

kubectl -n $NS describe pod <pod> | sed -n '/^Events:/,$p'   # look for "Readiness probe failed"
kubectl -n $NS logs <pod> -c <container> --tail=30            # the app may be up but slow/failing the probe

The readinessProbe is failing — the app is either not ready yet (slow startup; raise initialDelaySeconds / periodSeconds) or is in a bad state the probe catches. A bad rollout (config change, bad config) can make every new pod fail its probe. Fix the probe target or the app's readiness, then the pods go Ready.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the root cause

This alert is always a symptom — the root cause is in the pods. Follow Diagnosis steps 3–5 to the specific cause:

Unschedulable — free capacity or adjust the pod requests/tolerations in a chart-values PR.
PVC / missing resource — see 04-storage/ or create the missing Secret/ConfigMap.
Image pull / crash — see 07-kubernetes/01-pod-imagepull.md and 02-pod-crashloop.md.
Readiness probe failing — fix the probe or the app's readiness (chart-values PR for initialDelaySeconds, or fix the app).

Step 3 — Roll back to the last good ReplicaSet (emergency)

If the service is down and the root cause fix will take too long, scale the last good ReplicaSet back up so the service resumes on the previous version while the fix merges:

kubectl -n $NS get rs -l app=<deployment> -o custom-columns='NAME:.metadata.name,READY:.status.readyReplicas,DESIRED:.spec.replicas'
kubectl -n $NS scale rs/<last-good-rs> --replicas=<desired>

The last good ReplicaSet is the one with READY > 0 and DESIRED 0 (scaled down by the last rollout). Scaling it up brings the old pods back. This is a temporary hold — ArgoCD self-heal will reconcile once the chart-values fix merges. Only use this to keep the service alive; it does not fix the root cause.

Step 4 — Verify the fix

After the root cause is fixed, the pods reach Ready and kube_deployment_status_replicas_ready rises to match spec_replicas. The alert clears once ready > 0 sustained for the for: 15m window (practically, within a few scrape intervals after the first pod is Ready).

Verify

The alert clears once kube_deployment_status_replicas_ready > 0 for the Deployment, sustained for the for: 15m window.
kubectl -n $NS get deploy $DEPLOY shows READY as N/N (matching desired).
kubectl -n $NS get pods -l app=<deployment> shows all pods Running and Ready.
For a readinessProbe fix, kubectl describe pod shows no recent Readiness probe failed events.