Runbook for a container that cannot pull its image, triggered by the PodImagePull Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). This covers both ErrImagePull (first pull attempt failed) and ImagePullBackOff (retries with exponential backoff), for both regular containers and init containers (the or in the expr unions kube_pod_container_status_waiting_reason and kube_pod_init_container_status_waiting_reason). There is no matching Mimir-ruler alert.

For a container that pulls fine but crashes on startup (CrashLoopBackOff), see the sibling runbook 02-pod-crashloop.md — that is an app-level failure, not a pull failure.

Live examples (2026-08-05)

Two real firing instances, used below to illustrate the branches:

ImagePullBackOff — oidc-discovery-proxy on sbxtestgmr (sandbox, regular container). Image corti.azurecr.io/kommodity-io/oidc-discovery-proxy:v1.3.0 does not exist in ACR. The kubelet message reads failed to resolve image: ...: not found, ... 401 Unauthorized — note both appear together (see step 2 for why).
ImagePullBackOff — transcript-service-init on cortistaging / engagestaging / try (dev+staging, init container). The init container image cannot be pulled; the pod stays in Init:ImagePullBackOff and the main containers never start.

Meaning

kube_pod_container_status_waiting_reason (regular containers) and kube_pod_init_container_status_waiting_reason (init containers) are gauges from kube-state-metrics (deployed by the synapse-client chart, scraped on the cluster and remote-written to synapse-server Mimir). For each container, the metric reports the current waiting.reason from the pod status, or is absent when the container is running. The metric is 1 while the container is in that waiting state — so == 1 filters to currently-stuck containers, and the or unions regular + init containers into one alert stream. Each stuck container becomes its own alert instance with exactly the labels the expr preserves — k8s_cluster_name, environment, namespace, pod, container, and reason. sum by (...) intentionally drops the scrape-integration labels (job from the kube-state-metrics integration, instance, source), so the alert reads as a workload problem for the stuck container — not as a problem with kube-state-metrics itself. Pre-aggregating in the expr (rather than at the alert) is what strips those labels; alert-level label transforms would keep them on the series.

Images live in Azure Container Registry (corti.azurecr.io), organised as corti.azurecr.io/<team>/<app>:<tag>. ACR is reachable from all clusters and pull works without per-workload imagePullSecrets (anonymous pull is enabled for the corti registry), so a pull failure is almost never an auth problem for corti images — it is a wrong/missing tag or an egress block. A handful of system SAs (keycloak, traefik) have explicit AcrPull via workload-identity federated credentials (see clusters/<cluster>/infra.yaml), and ArgoCD pulls via the cortiacr secret in the argocd namespace — those are only relevant if anonymous pull is ever disabled.

An init container pull failure is operationally distinct: the pod never reaches Running (it stays Init:ImagePullBackOff), so the main containers never start. The fix is the same as for a regular container — the image must be pullable — but the blast radius is the whole pod, not one container.

Impact

The workload cannot start. A new Deployment/rollout stalls: the new ReplicaSet stays at 0 ready. If the old pods are gone (rollback, scale-down) the service is down.
StatefulSets (CNPG, Kafka) cannot rejoin after a node loss if the image is unpullable → data-plane impact.
An init container pull failure blocks the pod entirely — no main container starts, even if their images are fine.
A misconfigured rollout (bad tag) can affect the whole fleet if it is the in-production Deployment. The 15m for rides out a normal rolling update; a pod still stuck after 15m is a real problem.

Diagnosis

The alert labels carry k8s_cluster_name, namespace, pod, container, reason, and environment. From the alert in Grafana, note all of them. The container label tells you which container; check kubectl describe pod to see whether it is an init container or a regular container.

1. Confirm the reason and get the kubelet message

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01-admin.yaml
NS=<namespace>
POD=<pod>
kubectl -n $NS describe pod $POD | sed -n '/^Events:/,$p'

The Events section shows the kubelet's pull message verbatim. The message tells you the branch:

... <image>: not found (often paired with ... 401 Unauthorized — see step 2) → the image/tag does not exist in the registry. Go to step 2.
failed to authorize: ... unauthorized: authentication required without not found → registry auth failed for a repo that does exist (anonymous disabled, or pulling from a non-corti registry that needs a secret). Go to step 3.
failed to do request: ... timeout / connection refused / no such host → egress to the registry is blocked (NetworkPolicy, DNS, node egress). Go to step 4.

Also confirm the current state directly (shows both regular and init containers):

kubectl -n $NS get pod $POD -o jsonpath='{range .status.containerStatuses[*]}{.name}: {.state.waiting.reason} — {.state.waiting.message}{"\n"}{end}'
kubectl -n $NS get pod $POD -o jsonpath='{range .status.initContainerStatuses[*]}{.name}: {.state.waiting.reason} — {.state.waiting.message}{"\n"}{end}'

2. Image not found (`ErrImagePull: not found`)

The image reference is wrong. Check what the pod is actually requesting vs what exists:

# What the pod asked for (regular + init containers):
kubectl -n $NS get pod $POD -o jsonpath='{range .spec.containers[*]}{.name}: {.image}{"\n"}{end}'
kubectl -n $NS get pod $POD -o jsonpath='{range .spec.initContainers[*]}{.name}: {.image}{"\n"}{end}'

# What exists in ACR (list tags for the repo):
az acr repository show-tags --name corti --repository <team>/<app> --top 20 -o table

Important

ACR returns 401 Unauthorized on the anonymous-token endpoint for a repo that does not exist (it will not grant an anonymous pull token for a missing repo, to avoid leaking which repos exist). So a non-existent image shows both not found and 401 Unauthorized in the same kubelet message — the not found text is the signal, not the 401. This is the live oidc-discovery-proxy:v1.3.0 case above. A true auth failure (step 3) is unauthorized without not found, against a repo that demonstrably exists.

Common causes:

Tag typo / wrong tag — the chart config.json or values bump pointed at a tag that was never pushed (CI built but didn't push, or pushed to a different repo). Compare the live image tag against the latest ACR tags.
Image pushed to the wrong repo — <team>/<app> path mismatch between the chart and ACR. The orca flow auto-bumps config.json chart tags; an app image tag is bumped separately and can drift.
Tag deleted — a :latest or mutable tag was re-pushed/overwritten, or an old tag was garbage-collected. Pin immutable tags (v1.2.3), never :latest, for production.

Confirm the tag exists before fixing. If it is missing, the owning team must push the image (or you roll back config.json / chart values to the last known-good tag).

3. Registry auth failed (`ErrImagePull: unauthorized` without `not found`)

This is rare for corti images (anonymous pull is enabled) but can happen for:

A non-corti registry (Docker Hub, ghcr) referenced by an image without a matching imagePullSecret. Docker Hub rate-limits anonymous pulls → 429 Too Many Requests shows as pull failure.
ACR anonymous pull was disabled (check the corti ACR --anonymous-pull-enabled setting: az acr show --name corti --query anonymousPullEnabled).

# Does the pod's SA have an imagePullSecret?
SA=$(kubectl -n $NS get pod $POD -o jsonpath='{.spec.serviceAccountName}')
kubectl -n $NS get sa $SA -o jsonpath='{.imagePullSecrets}{"\n"}'

# Does the namespace have a registry secret the chart should have mounted?
kubectl -n $NS get secret -o jsonpath='{range .items[*]}{.metadata.name}:{.type}{"\n"}{end}' | grep -iE "docker|registry|acr"

For a corti image showing unauthorized, first re-check step 2 — a not found is sometimes misread. True unauthorized on corti.azurecr.io means anonymous pull was disabled; raise it with the platform team that owns the ACR rather than patching a pull secret ad hoc.

4. Egress blocked (`ErrImagePull: timeout / connection refused`)

The node/kubelet cannot reach corti.azurecr.io:443. This is the NetworkPolicy or node-egress path:

# Does the pod's namespace have a restrictive default-deny NetworkPolicy?
kubectl -n $NS get networkpolicy -o wide

# Is DNS resolving the registry?
kubectl -n $NS exec $POD -- nslookup corti.azurecr.io 2>/dev/null || echo "(pod not running — check from an alive pod in the namespace)"
kubectl -n $NS run dns-test-$$ --image=busybox --rm -it --restart=Never -- nslookup corti.azurecr.io

# Node-level egress (Cilium on Kommodity/Scaleway clusters):
kubectl -n kube-system get ciliumnetworkpolicy -o wide 2>/dev/null | head

The common-chart renders an egress NetworkPolicy per service; a service without an explicit registry/443 allow rule and a default-deny namespace will be blocked. The durable fix is a chart-values PR adding the egress rule (ArgoCD self-heal reverts a manual kubectl patch in ~15s). On Kommodity clusters the CNPG chart's missing IPv6 egress (::/0) was a similar class of bug (chart PR #2868) — registry egress can have the same gap.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the image

Wrong tag — roll the chart config.json / image tag back to the last known-good tag in a deployments PR. For an ArgoCD-managed rollout, the fix is GitOps: edit the tag, merge, ArgoCD syncs and the new ReplicaSet pulls the correct image. Do not kubectl set image directly (ArgoCD self-heal reverts it).
Missing image — the owning team must push the tag to ACR (az acr build / CI pipeline). Roll back to a known-good tag in the meantime so the service is not down while the image is built.
Egress blocked — raise a chart-values PR adding the registry/443 (and IPv6 ::/0 if the cluster resolves ACR to v6) egress rule to the service's NetworkPolicy.

Step 3 — Unstick a stuck rollout (emergency)

If a bad rollout is taking the service down and the GitOps fix will take too long, scale the good ReplicaSet back up while the fix merges:

# Find the ReplicaSets; the bad one is at 0 desired-after-fix, the last good one is the previous:
kubectl -n $NS get rs -l app=<app>
kubectl -n $NS scale rs/<last-good-rs> --replicas=<desired>

This is a temporary hold — ArgoCD self-heal will reconcile once the PR merges. Only use this to keep the service alive; it does not fix the root cause.

Verify

The alert clears once the container pulls the image and reaches Running (the waiting.reason series disappears, so == 1 is no longer true) sustained for the for: 15m window. Practically it clears within a few scrape intervals after the pull succeeds.
kubectl -n $NS get pod $POD shows Running with the container Ready (or, for an init container, the pod progresses past Init: to Running).
kubectl -n $NS describe pod $POD shows no recent Failed/Back-off pull events, only Pulled/Created/Started.
For an image-tag fix, the live image matches the chart: kubectl -n $NS get pod $POD -o jsonpath='{.spec.containers[*].image}' equals the tag in config.json/values.
For an egress fix, a fresh pod in the namespace can reach the registry: kubectl -n $NS run egress-test-$$ --image=busybox --rm -it --restart=Never -- wget -qO- https://corti.azurecr.io/v2/ returns {} (200, empty manifest list) rather than timing out.

Pod ImagePull / ImagePullBackOff