Runbook for a StatefulSet that has zero Ready replicas, triggered by the StatefulSetNoRunningReplicas Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). There is no matching Mimir-ruler alert.

This is the StatefulSet sibling of 03-deployment-no-running-replicas.md (Deployments) and 05-daemonset-no-running-pods.md (DaemonSets). StatefulSets are distinct: pods are ordered (pod-0, pod-1, …) and PVC-bound (each pod has a named persistent volume from a volumeClaimTemplate), so the failure mode and diagnosis differ.

Meaning

A StatefulSet runs ordered, stable-named pods (<sts>-0, <sts>-1, …), each bound to its own PVC (volumeClaimTemplate). Pods start in order: pod-1 is only created once pod-0 is Ready. kube_statefulset_replicas is the desired replica count (spec.replicas); kube_statefulset_status_replicas_ready is the number of pods that are Ready.

The alert fires when desired > 0 but ready == 0 for 15m — no pod is Ready. Because pods start in order, this almost always means pod-0 is down: without pod-0 Ready, the controller never creates pod-1, pod-2, etc. So "zero ready" reduces to "why is pod-0 not Ready."

Both metrics are gauges from kube-state-metrics (deployed by the synapse-client chart, scraped on the cluster and remote-written to synapse-server Mimir). The and on (k8s_cluster_name, environment, namespace, statefulset) matches the series per-StatefulSet. Each leg is pre-aggregated with sum by (...) to strip the scrape-integration labels (job, instance, source), so the alert carries only k8s_cluster_name, environment, namespace, and statefulset and reads as a workload problem — not as a problem with kube-state-metrics itself. Each StatefulSet with zero ready becomes its own alert instance.

StatefulSets in this platform are the stateful workloads: CNPG (cnpg-*-cluster), Kafka, Mimir (mimir-ingester, mimir-store-gateway, mimir-compactor), Loki (loki-backend, loki-write), Keycloak, redis-node. Losing all ready replicas = the stateful service is down, and (for CNPG/Kafka) at risk of data unavailability.

Impact

The stateful service is down. For CNPG this is the database (every tenant reading/writing through it fails); for Kafka this is the broker (producers/consumers fail); for Mimir/Loki this is the observability backend (metrics/logs ingestion stops); for Keycloak this is auth (every login fails).
PVC-bound risk: if the cause is a PVC issue (rook-ceph crash destroyed the volume — the 2026-07 incident), the data may be unrecoverable without a backup. See 06-databases/01-cnpg-backup-failure.md for CNPG recovery.
Ordered startup: even after the root cause is fixed, a StatefulSet with N replicas takes N × (startup time) to come back, because pods start one at a time. A 3-pod CNPG cluster takes 3× the single-pod startup.
15m of zero-ready means pod-0 has been failing for 15m+ — a fresh deploy failed, pod-0 was lost (node loss, PVC gone), or pod-0 is crash-looping.

Diagnosis

The alert labels carry k8s_cluster_name, namespace, statefulset, and environment. From the alert in Grafana, note all of them.

1. Confirm the StatefulSet has zero ready pods (and find pod-0)

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=<namespace>
STS=<statefulset>
kubectl -n $NS get sts $STS
kubectl -n $NS get pods -l app=<statefulset> -o wide

kubectl get sts shows READY as 0/N. List the pods: they are named <sts>-0, <sts>-1, … in order. Pod-0 is the one to investigate first — if it is not Ready, no higher ordinal exists or is Running.

2. Why is pod-0 not Ready?

The pod state tells you the branch:

kubectl -n $NS get pod <sts>-0 -o wide
kubectl -n $NS describe pod <sts>-0 | sed -n '/^Events:/,$p'
kubectl -n $NS get pod <sts>-0 -o jsonpath='{range .status.containerStatuses[*]}{.name}: {.state.waiting.reason} — {.state.waiting.message}{"\n"}{end}'

Pending → unschedulable or PVC issue. Go to step 3.
CrashLoopBackOff → the container is starting then exiting. Go to step 4.
ErrImagePull / ImagePullBackOff → 07-kubernetes/01-pod-imagepull.md (bad image tag).
Running 0/1 (not Ready) → readinessProbe failing. Go to step 5.

3. Pod-0 is Pending (PVC / unschedulable)

StatefulSet pods are PVC-bound, so a Pending pod is usually a PVC problem:

kubectl -n $NS get pvc <sts>-<sts-name>-0   # or the PVC named in the volumeClaimTemplate
kubectl -n $NS describe pod <sts>-0 | sed -n '/^Events:/,$p'
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key,CORDONED:.spec.unschedulable'

Common causes:

PVC Pending — the volumeClaimTemplate cannot bind (storage class missing, no capacity, the storage backend is down). For rook-ceph/Ceph-backed PVCs, confirm the Ceph cluster is healthy: kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status (see 04-storage/ runbooks). The 2026-07 rook-ceph crash deleted prd-par01 PVCs — a Pending PVC with PersistentVolumeClaim ... is being deleted or a missing PV is that scenario.
PVC missing / destroyed — the PV was deleted (rook-ceph pool recreation). The pod cannot mount. For CNPG this is the database volume — recovery is via object-store backup (see 06-databases/01-cnpg-backup-failure.md); if no backup, a fresh initdb (last resort).
Unschedulable — taints/tolerations or resource short. Less common for StatefulSets (they usually tolerate their node pool). Check kubectl describe pod for the scheduler reason.

4. Pod-0 is CrashLoopBackOff

The container starts then exits. For a StatefulSet this is often a data-level issue (the PVC has stale/corrupt data, or a migration that fails on restart):

kubectl -n $NS logs <sts>-0 -c <container> --previous --tail=50   # the crashed instance's logs
kubectl -n $NS logs <sts>-0 -c <container> --tail=50

Common StatefulSet crash causes:

CNPG: replica trying to rejoin from a corrupt/stale PVC, or a primary that cannot start because its data dir is from an incompatible WAL position. For CNPG, deleting the replica's PVC and pod forces a rejoin from the primary (kubectl -n $NS delete pvc <replica-pvc> && kubectl -n $NS delete pod <replica-pod>). Never delete the primary's PVC. See 06-databases/02-cnpg-replication-unhealthy.md.
Kafka / Mimir / Loki: a corrupt on-disk state (bad segment, bad block). The fix is often deleting the PVC and letting the pod rejoin/rewrite — but that can lose in-flight data; check the component's runbook before deleting.
Bad config / migration: an init container or a startup migration fails (e.g. init-db for CNPG, a schema migration). See 07-kubernetes/02-pod-crashloop.md.

5. Pod-0 is Running 0/1 (readinessProbe failing)

kubectl -n $NS describe pod <sts>-0 | sed -n '/^Events:/,$p'   # look for "Readiness probe failed"
kubectl -n $NS logs <sts>-0 -c <container> --tail=30

The component is up but not healthy. For a stateful service this is often a dependency: CNPG cannot reach its own storage / a peer, Kafka cannot reach Zookeeper/KRaft peers, Mimir cannot reach the ring store. A bad rollout (config change) can make pod-0 fail its probe. Fix the probe target or the dependency.

6. Check the workload-specific status (CNPG / Kafka)

For a CNPG StatefulSet, the Cluster CR gives richer status than the STS:

kubectl -n $NS get cluster <cnpg-cluster-name> -o wide   # status.currentPrimary, readyReplicas, phase

A CNPG cluster with Ready: false and phase: ... tells you the CNPG-level state (e.g. Waiting for the instances to become active, Scaling down, Online resize). See 06-databases/01-cnpg-backup-failure.md and 06-databases/02-cnpg-replication-unhealthy.md for CNPG-specific recovery.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix pod-0

This alert is always a symptom — the root cause is in pod-0. Follow Diagnosis steps 2–6 to the specific cause:

PVC Pending / missing — fix the storage backend (see 04-storage/ runbooks) or, for CNPG, recover from the object-store backup (06-databases/01-cnpg-backup-failure.md). For a destroyed PVC with no backup, a fresh initdb is the last resort.
CrashLoopBackOff — see 07-kubernetes/02-pod-crashloop.md. For a CNPG replica, delete the replica's PVC + pod to force a rejoin. Never delete the primary's PVC.
Image pull — see 07-kubernetes/01-pod-imagepull.md (roll back the chart tag or push the missing image).
Readiness probe failing — fix the probe or the component's dependency (chart-values PR for initialDelaySeconds, or fix the upstream dependency).

Step 3 — Do not roll back a StatefulSet blindly

Unlike Deployments, kubectl rollout undo sts is risky for a StatefulSet: it can trigger a rolling restart that recreates pods in order, and if pod-0's PVC is the problem, the undo does not fix the PVC and can make things worse (the rolling restart may not progress past pod-0). Only roll back via a chart-values PR (GitOps) after the root cause is fixed, so ArgoCD reconciles the template cleanly. For CNPG, recovery is via the Cluster CR's recovery mode, not rollout undo — see 06-databases/01-cnpg-backup-failure.md.

Step 4 — Verify the fix

After the root cause is fixed, pod-0 reaches Ready, then the controller creates pod-1, pod-2, … in order. kube_statefulset_status_replicas_ready rises toward desired. The alert clears once ready > 0 sustained for the for: 15m window (practically, within a few scrape intervals after pod-0 is Ready; full convergence to ready == desired may take longer as higher ordinals start one at a time).

Verify

The alert clears once kube_statefulset_status_replicas_ready > 0 for the StatefulSet, sustained for the for: 15m window. (Full convergence to ready == desired may take longer due to ordered startup; the alert clears as soon as pod-0 is Ready, which is the right semantics — the service is no longer "zero running".)
kubectl -n $NS get sts $STS shows READY > 0 and rising toward the desired count.
kubectl -n $NS get pods -l app=<statefulset> shows pod-0 Running and Ready; higher ordinals follow.
For a CNPG StatefulSet, kubectl -n $NS get cluster <cnpg-cluster-name> shows Ready: true and Current Primary set.
For a PVC fix, kubectl -n $NS get pvc shows the PVC Bound.