Runbook for the final tier of the envoy heap chain: the overload manager is answering requests with local error replies — clients are failing. Tiers 1–2 and the shared diagnostic flow live in 05-observability/13-envoy-heap-saturation.md; this file covers what changes once clients are already affected.

Meaning

When the heap ceiling stops being negotiable (stop_accepting_requests / shrink_heap exhausted), the overload manager answers new requests with locally-generated 503s. Kubernetes sees Ready pods; the HPA reads low memory because shedding capped the working set — the outage is invisible to the standard workload signals, which is why this alert is the one guaranteed to exist when clients hurt.

The sum by (pod, namespace, k8s_cluster_name, environment) de-duplicates a per-envoy_http_conn_manager_prefix metric (281 series = 63 pods — one series per connection-manager prefix) while keeping the labels the alert templates need — without them the annotations would render empty namespace/cluster/environment.

The metrics are scraped from the proxy pods by synapse-client and remote-written to the central Mimir — no chart PR is needed.

Impact

Clients see intermittent 503/500 for every model regardless of upstream (PLA-7173, dev-weu) — the gateway, not the models, is failing.
Rate of failure depends on which overload action is active; it can oscillate between serving and rejecting as heap hovers at the ceiling.
Prod exposure: this is the public entry-point alert class for all envoy-gateway-* data planes in shared on each cluster.

Diagnosis

Tier-specific checks (shared flow in 13-envoy-heap-saturation.md):

Confirm the blast radius — which gateway deployment is affected:

sum by (pod) (rate(envoy_http_rq_overload_local_reply{container="envoy"}[5m]))

Which tier signatures are on the pod simultaneously — shrink_heap_active and the tier-1 heap ratio expr from 13-envoy-heap-saturation.md tell you whether this pod is oscillating at the ceiling or fully stuck.
Client-side confirmation — limit/extent visible in the 5xx signal:
```
envoy_http_downstream_rq_xx{envoy_response_code_class="5", container="envoy"}
```
The Team Platform "LLM Proxy / Envoy AI Gateway — 5xx & Upstream Health" dashboard (uid llm-proxy-5xx) shows the client-visible surface.

Mitigation

Step 1 — Page and confirm scope, then apply the limit bump now

The mitigation is the same memory-limit bump as in tier 1/2 — but here it is an outage intervention, not a capacity task. Bumping the limit restarts the pods, which resets retained heap and stops the local replies. Follow the exact values change in 13-envoy-heap-saturation.md (Step 1 of its Mitigation; the dev-weu incident precedent is 300Mi → 500Mi).

Step 2 — If the bump cannot merge (e.g. out of hours)

Manual scale-out / higher minReplicas spreads new requests across more pods with the same behavior; it mitigates but does not stop the rejecting pods. Combined with a rolling restart of the offending deployment (kubectl -n shared rollout restart deploy/<gateway-deploy>) it is the interim path — restarts drop retained heap immediately.

Recovery check before closing: clients off the gateway confirm 2xx baseline on envoy_http_downstream_rq_xx{envoy_response_code_class="5"} falls back and envoy_http_rq_overload_local_reply rate returns to 0.

Verify

After the bump/restart: envoy_http_rq_overload_local_reply rate is 0 on the affected pods for the for: 2m window — the alert clears.
Client-side metric: no fresh local-replies and 5xx code class baseline recorded on the public LLM gateway — clients no longer see failure bursts.
Tier-1 heap ratio stays below the 0.9 threshold on the restarted pods, so tier 1 does not re-trigger.

Notes

Three-in-one — tiers route per the state machine: near-ceiling → shedding → rejecting. Files: 13-envoy-heap-saturation.md (tier 1), 14-envoy-heap-shedding.md (tier 2), this file.
k8s_cluster_name not cluster — Mimir-series labels follow the remote-written convention; annotations use {{ $labels.k8s_cluster_name }}.