Runbook for an Envoy Gateway data-plane proxy pod whose overload manager has activated shrink_heap — tier 2 of the envoy heap chain. Tier 1 (near ceiling) and its full diagnostic flow live in 05-observability/13-envoy-heap-saturation.md; this file covers what changes once shedding is already happening.

Meaning

The overload manager keeps the heap under Envoy Gateway's fixed ceiling (80% of the container memory limit, from calculateMaxHeapSizeBytes). At 95% of that ceiling it activates shrink_heap, which pauses the tcmalloc background freelist — the proxy trades allocation throughput (latency) for memory. The next hard step is stop_accepting_requests, then the overload local replies that clients see as 503s (EnvoyProxyOverloadRejectingRequests).

The metrics are scraped from the proxy pods by synapse-client and remote-written to the central Mimir — no chart PR is needed.

Impact

Requests still served, but allocation pressure degrades proxy latency.
HPA does not react: shedding caps the working set below the HPA's memory target, so scale-out never fires on its own (the defining symptom of this alert chain — see 08-capacity/02-hpa-at-max-replicas.md).
Hours of lead time before clients see errors, but only if acted on. Ignored, it escalates to rejecting requests on the affected gateway.

Diagnosis

Start from the shared flow in 13-envoy-heap-saturation.md — confirm which tier is active, then answer:

Is tier 1 firing too? If yes, work through its memory-limit mitigation; tier 2 confirms the ceiling is actively binding.
Retained or live? envoy_server_memory_allocated vs envoy_server_memory_physical_size — a large gap (the dev-weu incident sat at ~194Mi allocated vs 259Mi physical) means tcmalloc retention: the pressure is not current load and pure scale-out will not reclaim it.
How long has shedding run? Sustained shrink_heap overnight (the PLA-7173 dev-weu incident ran 7.5h) is the same failure as a fraction — it will not self-resolve without the limit bump or scale-out.

Mitigation

Step 1 — Raise the container memory limit (the real knob)

The ceiling is a fixed fraction of the limit; bumping the limit bumps the ceiling. Same change as documented in the tier-1 runbook's mitigation — this is what resolved the dev-weu incident (300Mi → 500Mi, 2026-10-02). Restarting the pods for the new limit also resets retained heap.

Step 2 — Spread (only if the bump cannot land immediately)

Bumping minReplicas or manually scaling out spreads new load, not the already-retained bytes; existing pods keep shedding until restarted or trimmed. Use it for urgent spread while the limit PR merges. Options and the demand-side sizing signal live in 08-capacity/ and the Scaling (HPA) row of the Envoy Gateway dashboard (team platform / Envoy Gateway / Overview).

Verify

shrink_heap_active returns to 0 on the affected pods and stays 0 for the for: 2m window — the alert clears.
Heap ratio (tier-1 expr in 13-envoy-heap-saturation.md) drops below the 0.9 threshold — otherwise tier 1 re-triggers on the next growth cycle.
No escalation: stop_accepting_requests_active stays absent and no overload local replies (envoy_http_rq_overload_local_reply) appear.

Notes

Three-in-one — tiers route per the state machine: near-ceiling → shedding → rejecting. Files: 13-envoy-heap-saturation.md (tier 1), this file, 15-envoy-overload-rejecting-requests.md (tier 3).
k8s_cluster_name not cluster — Mimir-series labels follow the remote-written convention; annotations use {{ $labels.k8s_cluster_name }}.