Runbook for a container that hit its cgroup memory limit and keeps restarting, triggered by the ContainerOOMKilled Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). There is no matching Mimir-ruler alert.

For a container that cannot pull its image (ImagePullBackOff) see sibling 01-pod-imagepull.md; for one that runs but crashes on startup (CrashLoopBackOff) see 02-pod-crashloop.md. OOMKilled is memory-cgroup specific — exit code 137 with reason: OOMKilled on the last termination.

Live examples (2026-08-11)

Real firing/fired instances motivating this runbook:

synapse-server-mimir-store-gateway-zone-{a,b,c}-0 (production, monitoring-system) — all three zones OOMKilled simultaneously at 13:13 at the 2Gi limit while answering a 60m Series enumeration on the ~8.4M-series tenant. Fixed by raising memory to 4Gi + GOMEMLIMIT=3600MiB.
synapse-server-alloy-gateway-* (production) — 4 OOMKills across the 18-pod pool in ~30h, each pod sitting at ~7Gi of the 8Gi limit. Fixed by raising memory to 12Gi + GOMEMLIMIT=10800MiB.
synapse-server-alloy-loki-consumer-* (production) — 16–20 OOM restarts per pod over Aug 5–7 at the 1Gi limit (log batch buffers per cluster spike past the envelope).

Meaning

kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1 is a kube-state-metrics gauge that stays pinned at 1 after an OOM until the container terminates again for another reason — it alone is stale: a pod that OOMKilled last week and has been healthy since still reports 1. The alert therefore ands it with recent restart churn (increase(kube_pod_container_status_restarts_total[30m]) > 0), so it fires only when the container OOMKilled and has restarted again inside the last 30 minutes — i.e. the workload is actively bouncing, not historically crashed. A container that OOMs once and recovers drops out of the expr within ~30 min and never pages. Both legs are pre-aggregated with sum by (...) to strip the scrape-integration labels (job, instance, source), so the alert carries only k8s_cluster_name, environment, namespace, pod, container, and reason and reads as a workload problem — not as a problem with kube-state-metrics itself. The and on join keys include k8s_cluster_name and environment so the join does not cross clusters or environments.

OOMKill mechanics: the container's cgroup memory.max (= resources.limits.memory) is exceeded and the kernel picks the biggest process to kill. Go runtimes will happily run right up to the limit under memory pressure because they only GC on allocation growth or the GOMEMLIMIT soft bound — without GOMEMLIMIT a Go service OOMs at 100% of its limit instead of GC-ing earlier. Fixing a re-firing Go service with only a memory bump just moves the ceiling; the process will grow into it again.

Impact

Every OOMKill loses in-flight work for that container: query results, remote_write WAL batches, log batches — callers see 503s or gaps until a healthy replica takes over.
A wedged singleton (compactor, store-gateway zones) OOM-looping removes the whole function; a multi-replica deployment degrades silently while its last pod covers everything.
Repeated restarts also amplify the original pressure: each restart replays state (WAL, caches) into memory faster, tightening the next OOM.

Diagnosis

The alert carries k8s_cluster_name, namespace, pod, container, environment.

1. Confirm OOM + get the kill context

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
kubectl -n <namespace> get pod <pod> -o jsonpath='{range .status.containerStatuses[*]}{.name} restarts={.restartCount} lastTerminated={.lastState.terminated.reason} exit={.lastState.terminated.exitCode} at={.lastState.terminated.finishedAt}{"\n"}{end}'
kubectl -n <namespace> describe pod <pod> | sed -n '/^Events:/,$p'

reason: OOMKilled exit: 137 confirms the cgroup kill; a second OOM inside 30 min is what paged you.

2. How close to the limit is it?

kubectl top pod -n <namespace> <pod> --containers
curl -sG "https://synapse.corti.app/api/datasources/proxy/uid/mimir/api/v1/query" \
  -H "Authorization: Bearer <grafana-token>" \
  --data-urlencode 'query=container_memory_working_set_bytes{pod="<pod-prefix>.*", container="<container>", k8s_cluster_name="<cluster>"}'

Compare against resources.limits.memory (kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources}'). ~90%+ sustained = sizing; wild swings = burst workload.

3. Is it a Go workload without GOMEMLIMIT?

Most Corti platform services (Alloy, Mimir, Loki, Envoy proxy, controllers) are Go. Check the container env directly:

kubectl get pod -n <namespace> <pod> -o jsonpath='{range .spec.containers[*]}{.name}: {range .env[*]}{.name}={.value} {end}{"\n"}{end}' | tr ' ' '\n' | grep GOMEMLIMIT || echo "GOMEMLIMIT NOT SET"

4. Leak vs sizing

Memory graph rising steadily across hours with no plateau → leak or unbounded cache. Take a pprof heap snapshot if available (kubectl exec ... curl localhost:<port>/debug/pprof/heap) and hand to the owning team.
Memory plateaus just under the limit and only OOMs under burst → sizing. Only raise the limit, then confirm the new util plateaus at ~60-70% of the new envelope.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Set GOMEMLIMIT for Go services (the durable fix)

If the container is Go and GOMEMLIMIT is absent (or > limit), add it at ~85-90% of the memory limit so the runtime GCs before the kernel kills the process:

resources:
  requests:
    memory: 4Gi
  limits:
    memory: 4Gi
env:
  - name: GOMEMLIMIT
    value: "3600MiB"   # ~88% of 4Gi

Existing repo conventions to copy: synapse-server ingester 10800MiB at 12Gi, compactor 3600MiB at 4Gi, alloy-gateway 10800MiB at 12Gi (deployments environments/production/prd-par01/monitoring-system/apps/synapse-server/values.yaml), loki GOMEMLIMIT blocks in the same file. For the Grafana Alloy subchart the env var goes under alloy.extraEnv, not alloy.env.

Step 3 — Raise the memory limit (GitOps)

Patches via kubectl are reverted by ArgoCD self-heal in ~15s. Open a deployments/chart PR that raises resources.limits.memory (and requests.memory to keep Guaranteed QoS) for the workload, then let ArgoCD sync. If the workload is a StatefulSet or has PVCs, the rollout is one-at-a-time — plan around it.

Step 4 — Real leak

If memory climbs linearly and never plateaus even after the bump, it is a leak. Capture the heap profile (step 4 in Diagnosis), bounce the pod to stop the restart loop noise, and hand the profile to the owning team — do not keep growing the limit to cover a leak.

Verify

The alert clears when the container stops OOMKilling: kube_pod_container_status_restarts_total stops increasing and last_terminated_reason no longer shows OOMKilled on the latest termination, sustained past the for: 10m window.
kubectl top pod (or the Grafana workload dashboard) shows the container sitting comfortably below the new limit (under ~70%) even under burst load.
For a GOMEMLIMIT fix: the memory graph shows a soft sawtooth (GC cycles) below the limit instead of a hard climb to 100%.