Runbook for any Envoy Gateway data-plane proxy pod — the external gateway, the internal gateway, the AI gateway and the public LLM gateway — whose heap is approaching its fixed-heap ceiling; covering the EnvoyProxyHeapNearCeiling alert.

Three related alerts route here — one runbook file per alert, each carrying its own frontmatter block (frontmatter is what the tooling syncs to Mimir):

EnvoyProxyHeapNearCeiling (warning, priority 2, 10m) — this file. Heap > 90% of the fixed ceiling. No user impact yet; hours of lead time.
EnvoyProxyHeapShedding (warning, priority 1, 2m) — overload manager shrink_heap active. Own file: 14-envoy-heap-shedding.md. Requests still served but the proxy trades latency for memory.
EnvoyProxyOverloadRejectingRequests (critical, priority 0) — the 503 generator. Own file: 15-envoy-overload-rejecting-requests.md. expr: sum by (pod, namespace, k8s_cluster_name, environment) (rate(envoy_http_rq_overload_local_reply[5m])) > 0 (deduped per pod — the metric carries envoy_http_conn_manager_prefix, 281 series vs 63 heap series; labels kept for the alert templates). This is what clients observe as intermittent 503/500 from all models regardless of upstream (PLA-7173, dev-weu).

Each tier escalates when the previous one is crossed: ceiling pressure → shrink_heap → local replies → clients failing.

Meaning

Envoy Gateway derives a fixed-heap ceiling of 80% of the container memory limit (calculateMaxHeapSizeBytes in envoyproxy/gateway, hardcoded — no env var or CRD field). The Envoy overload manager activates shrink_heap at 95% and stop_accepting_requests at 98% of that ceiling.

The fixed-heap monitor measures tcmalloc's mapped heap: heap_size − pageheap_unmapped − pageheap_free (admin /memory). That is live allocations plus tcmalloc's retained memory (thread caches, fragmentation). It exports the result as envoy_overload_envoy_resource_monitors_fixed_heap_pressure, in percent of the ceiling. The overload actions fire on exactly this value, so the alert uses it directly.

envoy_server_memory_heap_size is not what the monitor measures. It also counts pages already returned to the OS (pageheap_unmapped), and it only ever grows for the life of the pod. On long-lived pods it ratchets towards the ceiling while real pressure stays low. Example, prod-weu public LLM gateway on 2026-10-09: heap_size 428 MiB (104% of the 410 MiB ceiling), but unmapped 299 MiB and free 32 MiB, so pressure 23%.

envoy_server_memory_allocated is live bytes only. A large gap between it and envoy_server_memory_physical_size (194Mi allocated vs 259Mi physical on the dev-weu incident) means the pressure is retention, not load.

The metrics are scraped from the proxy pods by synapse-client and remote-written to the central Mimir — no chart PR is needed.

Impact

The HPA does not react. The overload manager sheds load as heap approaches the ceiling, so container working set plateaus below the HPA's memory-utilization target — the HPA never fires while the proxy sheds. In dev-weu (PLA-7173) shrink_heap was active for 7.5 hours overnight while the HPA read 20% memory. Kubernetes sees Ready pods; the outage is invisible to standard workload signal.
Clients on the affected gateway see intermittent 503/500 for every model once shedding escalates.
Scale-out does not reclaim anything by itself: tcmalloc holds freed memory (RSS stays high) so replicas added later each carry the same retained footprint. With a real leak, HPA-on-memory is a treadmill to maxReplicas — the underlying fix is upstream: fixed_heap_use_allocated (envoyproxy/envoy#41442, adopted in Istio in istio/istio#58058) changes the monitor to measure allocated bytes instead of the mapped heap.

Diagnosis

The alert labels carry k8s_cluster_name, environment, namespace, and pod. From the alert, note all of them.

1. Confirm which tier is active and since when

Run against Mimir (datasource prometheus), replacing the placeholders:

# Which pods are shedding or rejecting, since when
envoy_overload_envoy_overload_actions_shrink_heap_active > 0
sum by (pod, namespace, k8s_cluster_name, environment) (rate(envoy_http_rq_overload_local_reply[5m])) > 0

# Pressure vs ceiling, percent (primary alert expr; shrink_heap at 95, stop_accepting_requests at 98)
max by (k8s_cluster_name, namespace, environment, pod) (envoy_overload_envoy_resource_monitors_fixed_heap_pressure)

# Live allocation vs reserved physical memory (retention vs active load)
envoy_server_memory_allocated
envoy_server_memory_physical_size

Look for:

Is the primary tier also firing (heap actually near ceiling), or is only the rejection alert active?
envoy_server_memory_allocated vs envoy_server_memory_physical_size — a large gap means tcmalloc retention, not active load.
The envoy_http_rq_overload_local_reply multi-fires per envoy_http_conn_manager_prefix; the sum by (pod, namespace, k8s_cluster_name, environment) in the alert handles it (keeps the labels the templates need). Plain queries for eyeballing should too.

2. Check HPA state — the defining symptom

export KUBECONFIG=~/.kube/<cluster>.yaml
kubectl -n shared get hpa | grep envoy
kubectl -n shared top pods -l 'app.kubernetes.io/component=proxy' | grep envoy

REPLICAS far below MAXPODS while alerts fire = the documented failure mode; HPA is blind here, this alert chain is the signal.
kubectl -n shared top pods — working set plateaued under target; compare against kubectl get deploy <gateway-deploy> -o jsonpath='{.spec.template.spec.containers[?(@.name=="envoy")].resources.limits.memory}'.

3. Correlate with the Scaling (HPA) row

The Envoy Gateway dashboard (team platform / Envoy Gateway / Overview) has a Scaling (HPA) row with the replica overlay and the heap/ceiling pair — connect an alert instance there to see both sides of the mismatch on one screen.

Mitigation

Step 1 — Bump the memory limit (the knob that actually exists)

The ceiling has no slider — it is 80% of the container memory limit, hardcoded. Raising the limit is the only handle:

# chart overlay via environments/<env>/shared/apps/envoy-gateway/values.yaml
internalAIGateway:
  deployment:
    # was 300Mi on dev-weu; bumped to 500Mi 2026-10-02 (PLA-7173 incident day)
    resources:
      limits:
        memory: "500Mi"

Confirm the requesting pod(s) restart cleanly and the ratio metric resets below the 0.9 threshold. Every extra limit is also honest new ceiling, not just new headroom, so the alert ratio scales with it.

Step 2 — Spread retained memory manually (interim)

Raising minReplicas or manually scaling out does not reduce retention per pod — it adds more pods with the same behavior. It still helps spread new load; do this if the drop is urgent, but with the expectation that the residual heap persists.

Step 3 — Backport plan for the real fix

Track upstream adoption of fixed_heap_use_allocated (envoy#41442, Istio-verified in istio#58058). When available via Envoy Gateway's bootstrap, this redefines "ceiling" from the mapped heap (with retention) to allocated bytes, restoring all of the ceiling as usable headroom and making the HPA-memory interaction meaningful again. Not available as a flag in Envoy Gateway v1.8.0 — see envoyproxy/gateway#3604 for the (open) request to make overload settings configurable.

Verify

The alert clears once envoy_overload_envoy_resource_monitors_fixed_heap_pressure drops below 90.
shrink_heap_active and envoy_http_rq_overload_local_reply rates return to 0 on the affected pods.
Clients stop seeing intermittent 503/500 on all models (envoy_http_downstream_rq_xx{envoy_response_code_class="5"} back to baseline on the public LLM gateway).

Notes

Why the pressure gauge, not a heap/limit ratio — it is the exact value the overload manager acts on, so it needs no join with kube_pod_container_resource_limits. A join like that also fails with "duplicate series" while two kube-state-metrics instances overlap, and then the rule does not evaluate at all.
max by — drops scrape labels (job, instance, endpoint) and dedupes, keeping the four labels the annotations use.
k8s_cluster_name not cluster — Mimir-series labels follow the remote-written convention; {{ $labels.k8s_cluster_name }} in annotations matches the existing capacity/observability runbooks.
Related — 08-capacity/02-hpa-at-max-replicas.md (HPA headroom, paired scaling signal), 08-capacity/04-container-memory-usage-high.md (container-level working set vs limit — does not fire while shedding holds it low), Envoy Gateway dashboard Scaling (HPA) row.