  sum by (k8s_cluster_name, environment, namespace, pod, container) (
    container_memory_working_set_bytes{container!="", container!="POD", container!="kafka"} offset 30m
  )
)
/
on (k8s_cluster_name, environment, namespace, pod, container)
sum by (k8s_cluster_name, environment, namespace, pod, container) (
  kube_pod_container_resource_limits{resource="memory", container!="kafka"}
)
> 0.05

for: 10m labels: team: platform severity: warning annotations: summary: "Container {{ $labels.container }} in {{ $labels.pod }} ({{ $labels.namespace }} on {{ $labels.k8s_cluster_name }}, {{ $labels.environment }}) is above 94% of its memory limit and still climbing — suspected memory leak" description: 'Container {{ $labels.container }} in pod {{ $labels.pod }} (namespace {{ $labels.namespace }} on cluster {{ $labels.k8s_cluster_name }}, environment {{ $labels.environment }}) is above 94% of its memory limit AND has grown by more than 5% of the limit over the last 30 minutes, sustained for 10m. At that rate the container will cross its limit (and be OOMKilled) within roughly 20-40 minutes. This is leak behaviour - memory that climbs near the ceiling instead of plateauing. GC-managed runtimes (JVM, Go, Node) and caches legitimately plateau at 94-99% of the limit, which is exactly why high usage alone does not fire this alert. Recommended: capture a heap profile and hand to the owning team; do NOT raise the memory limit to cover a leak. If the container has already been OOMKilled, see the sibling runbook 07-kubernetes/11-container-oomkilled.md - that alert is the safety net, this one is the early warning.'

Container Memory Leak (Rising Near Limit)

Runbook for a container whose memory is still climbing while already near its cgroup limit, triggered by the ContainerMemoryLeakRisingNearLimit alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts → PrometheusRule CRD synced to the Mimir ruler). Previously this runbook carried a static working_set / limit > 0.94 threshold alert (Container Memory Usage High (> 94%)); that rule was rewritten on 2026-10-16 because it paged on workloads that plateau near their limit by design while missing the containers that actually got OOMKilled (see What changed below for the forensic data).

For a container that has already OOMKilled and is restarting, see sibling 07-kubernetes/11-container-oomkilled.md — that is the post-OOM runbook and the structural safety net (it catches every kill, including the leak archetypes no metric-based rule can see). This runbook is the pre-OOM early warning: the container is still running, but its memory trend says it will hit the ceiling soon. The sibling 08-capacity/01-cpu-throttling.md is the CPU-side analogue.

Meaning

Two conditions must hold simultaneously, sustained for for: 10m:

Near the ceiling: container_memory_working_set_bytes / kube_pod_container_resource_limits{resource="memory"} > 0.94 — the working set is within ~6% of the cgroup limit, the same memory the OOM killer evaluates.
Still climbing: the working set grew by more than 5% of the limit over the last 30 minutes ((now - offset 30m) / limit > 0.05).

The key design decision is anchoring the growth to the limit, not to current consumption. "Grew 5%" is meaningless at 20% of the limit (post-deploy warmup, cache fill — a pod doubling from 2 Gi to 4 Gi of a 64 Gi limit is fine; a pod adding 5% of the limit per 30m at 94% is OOM in ~20 minutes). At 5%-of-limit-per-30m, the math to OOM is bounded no matter where the climb started.

Why the static threshold alone was wrong: memory utilization is state, and for GC-managed runtimes and caches high state is healthy. A JVM with -Xmx equal to the limit, a Go service without GOMEMLIMIT, cockroachdb's block cache, Redis, and lmcache all ramp to 94-99% of the limit and sit there — that is their designed operating point. What separates a leak from a healthy plateau is behavior: a plateau is flat, a leak keeps climbing. The and clause is the behavior test; a container that merely sits at 96% for a week never fires this alert.

The for: 10m clause additionally requires the climb to persist (the 30m window is long enough to ride out GC sawteeth; for rides out single-window bursts like a trivy scan or a large request payload).

The expr is generic across clusters and environments — no cluster or environment filter — so it fires for every limited container that exhibits leak-like behavior. Containers without a memory limit have no kube_pod_container_resource_limits{resource="memory"} series, so the vector-matching division drops them: the alert only covers containers that actually declare a limit.

What changed from the old static (94%) threshold

The old rule (working_set / limit > 0.94 for 15m) was evaluated against live data on 2026-10-16:

It fired on 14 containers at once, of which 0 were page-worthy: 9× cockroachdb db (on-prem customer clusters), 2× lmcache, argocd-redis, and a kserve-container — all plateau workloads by design. The alert had become background noise.
It missed the containers that actually OOMKilled. A 24h replay of the 15 real OOM events in the preceding day showed the misses fall into three archetypes:
1. True leaks (climbs until it crosses) — the target of this rule. The leak condition would have pre-warned on these with usable lead time. Verified lead times against the real events: ai-gateway-operator (dev-weu): 22 and 34 minutes on two of its three kills (the third: 4 min); ai-gateway-operator (prd-par01): 28 minutes; keycloak-1 (dev-weu): 24 minutes; argocd-server (prod-weu): 4 minutes.
2. Saturation crashloops — the container lives above 94%, gets OOMKilled, refills, repeats, all day. Before each kill the 30m growth is ~zero (it is already plateaued), so no growth-based rule can catch these; the 07-kubernetes/11-container-oomkilled.md sibling does, with perfect precision.
3. Spike kills — the container OOMKilled with the working set sampled at only ~84% of the limit (allocation burst faster than the 60s scrape, and the kernel counts file cache working_set underreports). No working-set-based rule, static or growth, can see these. (container_memory_max_usage_bytes is not in the cAdvisor allow-list — adding it would close this gap.)
The growth condition alone is quiet in steady state: replayed at multiple points across two days, the leak condition matched 0-1 containers cluster-wide — versus the old rule's constant 14.
Threshold choice — 5% per 30m, not 10%: at 10%-of-limit/30m the rule caught only 4 of the 15 real OOM events; at 5% it caught 8 (three more real events, for ~zero extra steady-state noise). Don't tighten this without re-running the OOM-replay analysis.

Scope changes

trivy-system, spegel, and monitoring-system namespaces re-included (previously the old rule's static noise filter). They were excluded because plateau/hot-by-design infra lives there — but the leak condition is precisely what makes plateau workloads safe to watch: a burst (trivy scan) does not survive for: 10m, a plateau is flat and never matches the growth clause. Verified quiet across sampled replays, and re-inclusion immediately surfaced a real candidate the old exclusion hid: policy-reporter on dev-neu at 97% of limit and climbing. If a specific workload here proves noisy in practice, exclude that workload, not the namespace.
container="kafka" stays excluded — on customer on-prem clusters (e.g. gmr) the kafka brokers ramp 0.67 → 0.98 of limit in a repeating sawtooth (JVM heap + page cache): each ramp phase looks like a 5%-of-limit climb and the rule would fire on ~3 pods several times a day. Kafka manages its own memory; real failures are caught by the OOMKilled sibling.
Routing moved from on-call to platform — matching the sibling capacity/OOMKilled runbooks; a leak warning with 20+ minutes of headroom does not need generic triage.

Impact

Expected lead time from first page to OOMKill is ~20-40 minutes at 5%-of-limit per 30m — enough time to capture diagnostics before the kill, not enough to ignore it.
If the climb is not interrupted, the container will be OOMKilled: in-flight work is lost (request results, remote_write WAL batches, active sessions), callers see 503s until a healthy replica takes over.
A firing leak alert often precedes a crashloop (saturation archetype): after the first OOMKill the refill is faster than the leak, and the 07-kubernetes/11-container-oomkilled.md alert takes over.

Diagnosis

The alert labels carry k8s_cluster_name, environment, namespace, pod, container.

1. Confirm the climb — live ratio and 30m growth

export KUBECONFIG=~/.kube/<cluster>.yaml
NS=<namespace>; POD=<pod>; CT=<container>

Live ratio and growth via Mimir (Grafana datasource proxy, or port-forward synapse-server-mimir-gateway in monitoring-system with header X-Scope-OrgID: corti):

RATIO='sum (container_memory_working_set_bytes{k8s_cluster_name="<cluster>",namespace="<ns>",pod="<pod>",container="<ct>"}) / sum (kube_pod_container_resource_limits{resource="memory",k8s_cluster_name="<cluster>",namespace="<ns>",pod="<pod>",container="<ct>"})'
GROWTH='(sum (container_memory_working_set_bytes{k8s_cluster_name="<cluster>",namespace="<ns>",pod="<pod>",container="<ct>"}) - sum (container_memory_working_set_bytes{k8s_cluster_name="<cluster>",namespace="<ns>",pod="<pod>",container="<ct>"} offset 30m)) / sum (kube_pod_container_resource_limits{resource="memory",k8s_cluster_name="<cluster>",namespace="<ns>",pod="<pod>",container="<ct>"})'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$RATIO" | jq -r '.data.result[0].value[1]'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$GROWTH" | jq -r '.data.result[0].value[1]'

Ratio > 0.94 and growth > 0.05 confirms the alert. Also graph the working set over the last few hours (Grafana workload dashboard): a leak is a climb with no plateau across multiple windows; a single sawtooth ramp then flat is warmup/traffic.

Cross-check whether it has already been OOMKilled recently (kube_pod_container_status_restarts_total increasing) — if so, the OOMKilled runbook takes precedence.

2. Capture a heap profile before the kill

The value of this alert's lead time is capturing forensics while the container still lives:

kubectl -n $NS exec $POD -c $CT -- curl -s localhost:<pprof-port>/debug/pprof/heap > heap-$(date +%s).pb.gz   # Go, if exposed
kubectl -n $NS exec $POD -c $CT -- jmap -dump:live,format=b,file=/tmp/heap.hprof 1                            # JVM

For Kubernetes-level platforms without pprof/jmap endpoints, use kubectl inspect / an kubectl debug ephemeral container with gops/gdb.

3. Identify the owning workload and team

kubectl -n $NS get pod $POD -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}'

The owner is the Deployment/StatefulSet/DaemonSet whose values rendered the limit. The owning team is the service's corti.team label in its chart values (config/services/<service>/base/values.yaml) or the namespace's bootstrap. Hand the heap profile to that team — they own the fix.

4. Check what changed

Leaks are usually introduced by a deploy or a traffic pattern change. kubectl rollout history on the owner + recent image tag, and correlate the climb's start time with deploys (kube_deployment_status_updated timestamps). A climb that started at the exact rollout time points at a release regression, not a long-lived leak.

Mitigation

Step 1 — Relieve pressure (the page's job)

If the container is minutes from crossing (growth > 10% of limit per 30m), bounce the pod to restore service and let a replica take over:

kubectl -n $NS delete pod $POD

This resets the symptom, not the leak — if the refill climb repeats on the new pod, the leak is confirmed and the owning team needs the profile from Diagnosis step 2.

Step 2 — Fix the leak (owning team, with the profile)

The durable fix is in the service code or its unbounded cache/config sizing, driven by the heap profile. Common causes: unbounded in-memory cache lacking eviction; accumulating goroutines/handlers per request; retained request/response buffers; JVM heap sized above the container limit.

Step 3 — Do NOT raise the limit to cover a climb

Raising resources.limits.memory is the correct fix for a plateau above 94% (legitimate usage, e.g. after a legitimate scale-up in working set) — but a genuine leak will consume any new limit on the same ~30-minute-clock, and now each OOMKill is more expensive. Only bump limits in a deployments PR (not kubectl set resources — ArgoCD self-heal reverts it in ~15s) after the owning team confirms the usage is legitimate.

Step 4 — GOMEMLIMIT for Go services (the plateau preventive)

If the container is a Go service and GOMEMLIMIT is absent, add it at ~85-90% of the memory limit so the Go runtime GCs before the kernel OOM-kills the process. This prevents the plateau-at-100% pattern from ever reaching this alert's ceiling:

env:
  - name: GOMEMLIMIT
    value: "3600MiB"   # ~88% of 4Gi

Existing repo conventions: synapse-server ingester 10800MiB at 12Gi, alloy-gateway 10800MiB at 12Gi — environments/production/prd-par01/monitoring-system/apps/synapse-server/values.yaml. For the common-chart, the env var goes under envs.fromLiterals.

Verify

The alert clears when the growth clause stops matching for 10m — a true plateau, or a GC that arrests the climb at < 0.94. Note the alert can re-fire immediately after a pod bounce replaces the pod name — that re-fire is confirmation the leak is real, not noise.
kubectl -n $NS top pod $POD --containers shows the container flat below its (repaired) plateau.
Re-run the Mimir queries from Diagnosis step 1: ratio should plateau with growth ≤ ~0.05 over 30m windows.
After a durable fix, watch 24h: the sibling OOMKilled alert staying quiet for that container is the final confirmation.

Notes

Leak vs saturation vs spike — this rule only detects the true-leak archetype (memory climbs until it crosses). Saturation crashloops (lives above 94%, killed, refills, repeats) have ~zero pre-kill growth and are invisible to any growth rule; spike kills (allocation burst beyond what working_set samples) are invisible to any utilization rule. The 07-kubernetes/11-container-oomkilled.md alert is the only rule covering all three archetypes — treat it as the safety net and this alert as the early-warning channel.
Lead-time validation (2026-10-16 replay of 15 real OOM events, all clusters): 8/15 pre-warned with a 4-34 minute lead; the near-misses were saturation-cycle and spike archetypes by construction. A 10%-of-limit/30m growth threshold caught only 4/15 and is deliberately not used.
Metrics already collected — container_memory_working_set_bytes and kube_pod_container_resource_limits are both in the synapse-client allow-list; the offset 30m clause needs no extra metrics. container_memory_max_usage_bytes (true cgroup peak) is NOT in the allow-list; adding it would make spike kills visible retroactively.
container="kafka" excluded on purpose — brokers on customer on-prem clusters ramp in a repeating 0.67 → 0.98 sawtooth that would fire this rule several times a day. The kafka namespace's other containers (kminion, entity operators, zookeeper) are covered.
trivy-system, spegel, monitoring-system re-included — plateau/hot-by-design infra proves the growth clause's point: plateaus are flat and quiet. First real catch after re-inclusion was policy-reporter (dev-neu). If a workload here grows noisy, exclude the workload, not the namespace.
for: 10m and evaluation — the Mimir ruler evaluates the expr with the 30m offset against the same allow-listed series on every tick; no recording rules are involved.
Filename unchanged — the runbook file keeps 04-container-memory-usage-high.md so the Orca runbook URL (runbooks/capacity/container-memory-usage-high, auto-derived from the path by the runbook_alerts module) and existing wiki links stay stable, even though the alert title is now Container Memory Leak (Rising Near Limit).