Runbook for a container that pulls its image fine but crashes on startup, triggered by the PodCrashLoopBackOff Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). This covers both regular containers and init containers (the or in the expr unions kube_pod_container_status_waiting_reason and kube_pod_init_container_status_waiting_reason). There is no matching Mimir-ruler alert.

For a container that cannot pull its image (ErrImagePull / ImagePullBackOff), see the sibling runbook 01-pod-imagepull.md — that is a pull failure, not a crash.

Live examples (2026-08-05)

Two real firing instances, used below to illustrate the branches:

CrashLoopBackOff — agent-api on prd-par01 (production, regular container). Image corti.azurecr.io/ai-agents/agent-api:v2.32.0 pulls fine (already present on machine in events); the container exits on startup with panic: failed to parse storage config: env: required environment variable "AZURE_STORAGE_CONNECTION_STRING" is not set — a missing env var.
CrashLoopBackOff — agent-api-init-db on dev-neu / prd-par01 (init container). The init container crashes before the main container starts; the pod stays in Init:CrashLoopBackOff and never reaches Running.

Meaning

CrashLoopBackOff is not a pull failure — the image is on the node, the container starts, and then the process inside exits repeatedly. The kubelet restarts it with exponential backoff, hence the Back-off restarting failed container event. The kubelet's waiting.message reads back-off ... restarting failed container=<container> pod=<pod>; there are no Failed to pull image events (unlike ImagePullBackOff).

kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} (regular containers) and kube_pod_init_container_status_waiting_reason{reason="CrashLoopBackOff"} (init containers) are gauges from kube-state-metrics (deployed by the synapse-client chart, scraped on the cluster and remote-written to synapse-server Mimir). The metric is 1 while the container is in that waiting state and absent when the container is running — so == 1 filters to currently-stuck containers, and the or unions regular + init containers into one alert stream. Each stuck container becomes its own alert instance with exactly the labels the expr preserves — k8s_cluster_name, environment, namespace, pod, container, and reason. sum by (...) intentionally drops the scrape-integration labels (job from the kube-state-metrics integration, instance, source), so the alert reads as a workload problem for the stuck container — not as a problem with kube-state-metrics itself. Pre-aggregating in the expr (rather than at the alert) is what strips those labels; alert-level label transforms would keep them on the series.

An init-container crash loop is operationally distinct: the pod never reaches Running (it stays Init:CrashLoopBackOff), so the main containers never start. The diagnosis is the same — read the init container's logs — but the blast radius is the whole pod.

Impact

The container is restarting repeatedly. If it is a main serving container, the service is degraded/down.
An init container crash loop blocks the pod entirely — no main container starts, even if their images and config are fine.
A misconfigured rollout (bad env var, bad config) can affect the whole fleet if it is the in-production Deployment. The 15m for rides out a normal rolling update; a pod still stuck after 15m is a real problem.

Diagnosis

The alert labels carry k8s_cluster_name, namespace, pod, container, and environment. From the alert in Grafana, note all of them. The container label tells you which container; check kubectl describe pod to see whether it is an init container or a regular container.

1. Confirm it is a crash, not a pull failure

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01-admin.yaml
NS=<namespace>
POD=<pod>
kubectl -n $NS describe pod $POD | sed -n '/^Events:/,$p'

Confirm the events show Back-off restarting failed container and Pulled (image already present on machine), not Failed to pull image. If there are pull errors, switch to the sibling runbook 01-pod-imagepull.md.

2. Get the container logs

The crashed process's exit reason is in the logs. For a regular container:

kubectl -n $NS logs $POD -c <container> --previous --tail=50   # the crashed instance's logs (—previous is key)
kubectl -n $NS logs $POD -c <container> --tail=50              # current attempt

For an init container crash loop (the pod shows Init:Error / Init:CrashLoopBackOff), target the init container by name:

kubectl -n $NS logs $POD -c <container> --previous --tail=50   # <container> is the init container name from the alert
# or all init containers at once:
kubectl -n $NS logs $POD --all-containers --init-containers --previous --tail=50

--previous is essential — it reads the logs of the last exited instance, which contains the crash message. Without it you get the current (restarting) attempt, which is usually empty.

3. Identify the app-level cause

The log's last lines tell you why the process exited. Common causes:

Bad config / missing env var / missing secret — the app exits on startup validation. This is the live agent-api case: panic: failed to parse storage config: env: required environment variable "AZURE_STORAGE_CONNECTION_STRING" is not set — the chart/ExternalSecret did not inject the var. Confirm with:
```
kubectl -n $NS get pod $POD -o jsonpath='{range .spec.containers[?(@.name=="<container>")].env[*]}{.name}={.value}{"\n"}{end}'
kubectl -n $NS get pod $POD -o jsonpath='{range .spec.containers[?(@.name=="<container>")].envFrom[*].secretRef.name}{"\n"}{end}'
kubectl -n $NS get externalsecret -o wide   # is the source Secret synced?
```
For a missing ExternalSecret key, confirm the key exists in the source Key Vault / external secret store and that ESO has synced it (kubectl get externalsecret -n $NS).
DB migration failure — an init-db container fails to connect to the DB or a migration is non-reversible. Check the DB is reachable from the namespace (NetworkPolicy on 5432 — see 06-databases/).
Liveness probe too aggressive — the probe fails before the app is ready and the kubelet kills it. Check kubectl describe pod for Liveness probe failed and the probe's initialDelaySeconds vs the app's startup time.
OOM during init — the container is OOMKilled on startup. Check kubectl describe pod for OOMKilled and the memory limit vs the app's actual startup peak (kubectl top pod).
Wrong entrypoint / args — a chart override changed command/args to something that exits. Check kubectl get pod $POD -o jsonpath='{.spec.containers[?(@.name=="<container>")].command}'.

This branch is usually the owning app team's domain, not platform. Route by the team label once per-team routing is wired; until then, platform triages and re-routes.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the crash

Bad config / missing secret — fix the chart values or ExternalSecret in a PR. For a missing ExternalSecret key, confirm the key exists in the source Key Vault / external secret store and that ESO has synced it.
DB migration failure — see 06-databases/01-cnpg-backup-failure.md if the DB itself is down. If the migration is stuck, the owning team must fix/revert the migration; rolling back the image tag to pre-migration is the fast unblock.
Probe / OOM — adjust the chart values (liveness initialDelaySeconds, memory limit) in a PR.

Step 3 — Unstick a stuck rollout (emergency)

If a bad rollout is taking the service down and the GitOps fix will take too long, scale the good ReplicaSet back up while the fix merges:

# Find the ReplicaSets; the bad one is at 0 desired-after-fix, the last good one is the previous:
kubectl -n $NS get rs -l app=<app>
kubectl -n $NS scale rs/<last-good-rs> --replicas=<desired>

This is a temporary hold — ArgoCD self-heal will reconcile once the PR merges. Only use this to keep the service alive; it does not fix the root cause.

Verify

The alert clears once the container reaches Running (the waiting.reason series disappears, so == 1 is no longer true) sustained for the for: 15m window. Practically it clears within a few scrape intervals after the pod is healthy.
kubectl -n $NS get pod $POD shows Running with the container Ready (or, for an init container, the pod progresses past Init: to Running).
kubectl -n $NS describe pod $POD shows no recent Back-off restarting failed container events.
For a config/secret fix, the pod env includes the previously-missing var: kubectl -n $NS get pod $POD -o jsonpath='{range .spec.containers[?(@.name=="<container>")].env[*]}{.name}{"\n"}{end}' lists it.