Runbook for a pod that has been stuck in phase Pending for more than 30 minutes, triggered by the PodPendingOver30Minutes Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). There is no matching Mimir-ruler alert.

GPU-requesting pods (pods with nvidia.com/gpu resource requests) are excluded from this alert — they have a distinct Pending pattern (GPU capacity exhaustion, autoscaler provisioning delays, KEDA self-correction) and are covered by a separate alert with a longer for window in 24-pod-pending-gpu.md. The unless clause in the expr uses kube_pod_container_resource_requests{resource="nvidia_com_gpu"} to filter them out — this metric is always available from kube-state-metrics (no label allowlist needed, it's a core resource metric).

Pending is the API phase a pod holds after it is admitted but before any container starts — not yet scheduled to a node (FailedScheduling), waiting for its init containers to finish, or blocked on a referenced resource (PVC not bound, ConfigMap/Secret missing). Note the overlap: a pod whose init container is crash looping stays in phase Pending forever (Init:CrashLoopBackOff — image pull failures behave the same way as Init:ImagePullBackOff), and a pod with regular containers crash looping also reports phase Pending until the container ever runs. So this alert overlaps the container-level siblings by design: follow 01-pod-imagepull.md / 02-pod-crashloop.md first when kubectl describe pod shows those waiting reasons, and use this runbook as the catch-all when no container-level reason is shown (or when the only reason is FailedScheduling). A Pending rollout that leaves the Deployment with zero Ready replicas is also caught separately by DeploymentNoRunningReplicas (see 03-deployment-no-running-replicas.md).

The alert is scoped to environment="production" by the expr filter. In a 6h window (2026-10-02), 263 pods across all clusters hit Pending at least once and 62 stayed Pending ≥5h — but the sustained offenders are dev/staging steady states (KEDA-scaled kafka-detect-* / transcript-service pools on cortistaging / engagestaging / try, the dev ml-service-medical-coding-* fleet on dev-par01, sm-sha-check* on dev-weu, sandbox interaction-service-*): parked capacity, a backlog to review in 08-capacity/, not an oncall page. The transient production Pending (e.g. daemonset pods on prd-ric02 / hamiltonctyoh during node cycles) appears as single grid samples — under the 30m window by construction.

Live examples (2026-10-02)

Real Pending pods on production clusters, from kube_pod_status_phase{phase="Pending"} == 1 queried on synapse-server Mimir. No container-level cause was visible from the metric labels alone (the phase gauge carries none — pod-level signal by design); all causes below are read from the series' own labels:

prd-par01 / mlservices, 8 pods Pending for 2h40m+ (11:45–14:25 UTC and still Pending). All ml-service-medical-coding-*, e.g. ml-service-medical-coding-kvaa-665d9b6448-xwtsg (Pending continuously across every 10m sample), plus cim10fr-inpatient, icd10cm-outpatient, icd10pcs, meddra-en, meddra-se, snomedctes, snomedctfr — exactly one new pod per medical-coding Deployment (the old-ReplicaSet pods report phase gauge 0 = still serving): an in-flight rollout whose new pods never left Pending in production.
prd-par02 / mlservices, GPU predictors Pending ~2h (from 12:24 UTC). corti-s1-mini-20260423-predictor-fbc67bcdc-7mvjk, corti-s1-mini-20260831-predictor-7c7d96c7c7-5szpw, corti-s1-tiny-20260707-predictor-65c7d5d65d-zvk29, corti-llm-gen1-predictor-84b5b484cd-5tkp8 — model-serving pods on the production GPU inference cluster, the classic scheduling/waiting-for-GPU-capacity shape.

Meaning

kube_pod_status_phase is a gauge from kube-state-metrics (deployed by the synapse-client chart, scraped on the cluster and remote-written to synapse-server Mimir), one series per pod per phase: it is 1 for the phase the pod is currently in and 0 for the others. So kube_pod_status_phase{phase="Pending"} == 1 filters to currently-Pending pods — the series goes 1 → 0 the moment the pod starts running (or is deleted, when the series goes stale and drops out). for: 30m is what implements "pending over 30 minutes": the gauge must stay at 1 continuously for the pod to page — no offset window in the expr, and it automatically stops firing (and does not re-fire) if the pod reaches a container state.

sum by (k8s_cluster_name, environment, namespace, pod) pre-aggregates the raw gauge so each alert instance carries only workload identity. The raw series carries kube-state-metrics scrape labels — job="integrations/kubernetes/kube-state-metrics", instance, source, and a duplicate cluster label — which would make the alert READ as a kube-state-metrics problem instead of a workload problem. Pre-aggregating in the expr (rather than at the alert) is what strips those labels; alert-level label transforms would keep them on the series (same convention as 01-pod-imagepull.md / 02-pod-crashloop.md).

Unlike those container-level alerts, this one has no container or reason dimension — the phase gauge describes the pod as a whole. The cause (FailedScheduling vs init container vs crashed regular container vs unbound PVC) must be read from kubectl describe pod. If a cause series exists with finer granularity (kube_pod_container_status_waiting_reason, kube_pod_init_container_status_waiting_reason), the sibling alerts (01/02) fire on it first at 5m — this alert at 30m is the no-cause-described catch-all.

The environment="production" filter keeps dev/staging capacity backlog out of incident.io (see the scope note in the intro).

Impact

The pod is doing no work. For a single-replica workload mid-rollout this means running on fewer pods than desired (degraded redundancy), or — if the old pods are already gone — no serving pods at all (which the zero-ready DeploymentNoRunningReplicas alert catches at P1 well before this one's 30m).
Scheduled work does not start: Jobs whose pods stay Pending do not run (e.g. opik-minio-post-job-29jmc on prd-par01 spotted Pending in the same window); StatefulSet scaling stalls at the missing ordinal.
Model-serving predictors stuck Pending on GPU clusters mean the deployment serves cold/fallback or steadily under capacity.
The alert tells you a scheduling/resource problem exists but not its cause — expect a short triage, not a one-line readout.

Diagnosis

The alert labels carry k8s_cluster_name, namespace, pod, and environment — no container or reason. From the alert in Grafana, note all of them.

1. Confirm the phase and how the pod sees it

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=<namespace>
POD=<pod>
kubectl -n $NS get pod $POD -o wide
kubectl -n $NS get pod $POD -o jsonpath='phase={.status.phase} PodScheduled={.status.conditions[?(@.type=="PodScheduled")].status}{"\n"}'

The STATUS column of get pod already points at the branch: plain Pending (scheduling/PVC problem), Init:0/N / Init:Error / Init:CrashLoopBackOff / Init:ImagePullBackOff (init container stuck), or ContainerCreating (pull/setup in flight).

2. Read the scheduler's reason

kubectl -n $NS describe pod $POD | sed -n '/^Status:/,$p'    # Conditions: PodScheduled=False, Reason: Unschedulable
kubectl -n $NS describe pod $POD | sed -n '/^Events:/,$p'

The Events section carries the cause verbatim. The four scenarios below cover the causes seen in production; match the event message to the relevant section.

2a. Missing Secret or ConfigMap (`FailedMount`, `CreateContainerConfigError`)

Most common cause for new tenant onboarding and new service deployments. The pod references a Secret or ConfigMap that doesn't exist or is missing a key. The pod shows CreateContainerConfigError or Init:CreateContainerConfigError in get pod, and the events carry FailedMount: ... Secret/ConfigMap "x" not found or couldn't find key "x" in Secret "y".

# Check which secret keys the pod expects
kubectl -n $NS describe pod $POD | grep -A 2 'secretKeyRef\|configMapKeyRef\|secretName\|configMapName'
# List the secret's actual keys
kubectl -n $NS get secret <secret-name> -o jsonpath='{.data}' | python3 -c "import sys,json; d=json.load(sys.stdin); print('\n'.join(sorted(d.keys())))"
# For ExternalSecret-managed secrets, check if it has synced
kubectl -n $NS get externalsecret
kubectl -n $NS get externalsecret <name> -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'

Live example: prd-weu / capioswe, agent-api-7fb44dd447-9vnmb Pending 53m (2026-10-21). ArgoCD deployed the agent-api Deployment before the Azure Key Vault secrets were provisioned. The tenant-unique Secret existed but was missing the cnpg-agent-api-pg-owner-password key — the ExternalSecret's tag-based discovery (dataFrom.find.tags: {tenant-unique: external-secrets}) hadn't found the KV entries yet. Fixed by provisioning the keys in Azure Key Vault; the ExternalSecret synced them within 5m and the pod started.

2b. Affinity / node selector mismatch (`FailedScheduling: didn't match affinity/selector`, `had untolerated taint`)

The pod's nodeSelector, nodeAffinity, or tolerations don't match any node in the cluster — either the node pool is full, cordoned, doesn't exist, or the toleration is missing for a tainted pool.

# What the pod requires
kubectl -n $NS get pod $POD -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity}{"\n"}{.spec.tolerations}{"\n"}'
# What nodes match
kubectl get nodes --show-labels | grep <label-value>
# Are any nodes cordoned?
kubectl get nodes | grep SchedulingDisabled

Common patterns:

Wrong pool: service pinned to workload: gpu-h100 on a cluster without that pool, or a CPU service accidentally pinned to an inference pool.
Missing toleration: node pool has a NoSchedule taint (e.g. workload=gpu-h100-mig) but the pod has no matching toleration.
Cordoned nodes: all nodes in the target pool are cordoned (maintenance, upgrade).

Live example: prd-par01 / mlservices, 8 ml-service-medical-coding-* pods Pending 2h40m+ (2026-10-02). A rollout created one new pod per medical-coding Deployment, but the new pods' CPU/memory requests exceeded the available capacity on the pool. The old ReplicaSet pods kept serving, so the services were degraded-but-up.

2c. Unbound PVC (`Unbound immediate PersistentVolumeClaims`)

StatefulSet ordinals or standalone pods stuck because the referenced PVC cannot bind — storage class missing, no matching PV, storage capacity exhausted, or zone mismatch.

# Check PVC status
kubectl -n $NS get pvc
kubectl -n $NS describe pvc <pvc-name> | sed -n '/^Events:/,$p'
# Check the storage class
kubectl get storageclass
# Check available PVs
kubectl get pv | grep <storage-class-name>

See 04-storage/ for storage-specific runbooks (e.g. 04-storage/04-pvc-near-full.md). A StatefulSet ordinal that never binds looks like redis-replicas-2 style Pending — the pod is created for the next ordinal but the PVC stays Pending.

2d. Init container stuck (`Init:ImagePullBackOff`, `Init:CrashLoopBackOff`)

The pod phase stays Pending while init containers haven't completed. This is caught earlier by the 5m container-level alerts — if you arrive here, the 30m catch-all fired because the init container has been stuck long enough.

kubectl -n $NS get pod $POD | grep Init

Init:ImagePullBackOff → 01-pod-imagepull.md
Init:CrashLoopBackOff → 02-pod-crashloop.md

While the pod is Pending, its regular containers have not started — there are no logs to read yet; kubectl logs $POD returns container <name> in pod is waiting until the init sequence completes.

No events / no FailedScheduling

Pending with PodScheduled=False for a long time and no events — the scheduler itself is not assigning the pod (rare: scheduler crash, heavy API pressure). This is a scheduler/kubelet problem — escalate to the platform team; do not chase pod config.

5. Watch for the pod to schedule

kubectl -n $NS wait --for=condition=Ready pod/$POD --timeout=35m
kubectl -n $NS get events --field-selector involvedObject.name=$POD --watch

A freshly actionable check after a capacity/taint change: if the pod schedules, you will see Successfully assigned <ns>/<pod> to <node> followed by Pulling events.

6. GPU-requesting pods

If the pod requests nvidia.com/gpu it will not fire this alert — it fires the separate PodPendingGPU alert instead. See 24-pod-pending-gpu.md for the GPU exhaustion diagnosis flow (H100 node saturation, cluster-autoscaler ScaleUpTimedOut, KEDA HPA self-correction). If you arrived here because the pod name looks like a predictor (*-predictor-*) but the alert did not fire, that is why — check the GPU alert instead.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the cause the events named

Unschedulable (GPU capacity on inference clusters) — GPU-requesting pods are handled by the separate PodPendingGPU alert — see 24-pod-pending-gpu.md for the full diagnosis flow. If a GPU pod shows up on this alert (it should not), the unless clause in the expr may have missed it — check whether kube_pod_container_resource_requests{resource="nvidia_com_gpu"} is actually scraped for that pod.
Unschedulable (taints/affinity) — chart-values PR adding the matching tolerations / fixing nodeSelector/affinity (wrong pool for the workload class, e.g. a service pinned to the inference pool).
PVC unbound — see 04-storage/; fix the storage class, capacity, or binding, then the pod schedules on the next kubelet retry.
Missing ConfigMap/Secret — create/fix the referenced object (chart-values PR; for secrets the ExternalSecret must actually sync — kubectl get externalsecret -n $NS).
Init container waiting — fix via 01-pod-imagepull.md / 02-pod-crashloop.md.

Patches via kubectl are reverted by ArgoCD self-heal in ~15s — durability means a PR (chart values or the referenced resource's manifest).

Step 3 — Unstick a stuck rollout (emergency)

If the Pending pods belong to a rollout you must back out before the capacity fix lands:

kubectl -n $NS rollout undo deploy/<deployment>            # or revert the PR — GitOps win
# last resort, temporary: scale the deployment down so the rollout pressure eases
kubectl -n $NS scale deploy/<deployment> --replicas=<n-1>

Old pods keep serving during the whole time (the Pending new pods are the only ones stuck), so this is rarely needed — only when the scheduling problem is taking down other workloads. This is a temporary hold; ArgoCD self-heal reconciles once the fix merges.

Verify

The alert clears once the pod leaves Pending: the phase="Pending" gauge drops to 0 (the pod is Running, or deleted) and the series stops matching == 1, sustained past the for: 30m window. Practically it clears within a few scrape intervals after the pod schedules.
kubectl -n $NS get pod $POD shows Running (and Ready once probes pass); for init cases the pod progresses past Init: to Running.
kubectl -n $NS describe pod $POD shows Scheduled (successfully assigned to a node) and no new FailedScheduling / FailedMount events.
If the cause was a rollout, kubectl -n $NS rollout status deploy/<deployment> reports completed, with the new ReplicaSet at desired.

Pod Pending Over 30 Minutes