Runbook for when the rate of 4xx/5xx responses returned by the envoy-gateway data planes rises well above its rolling 7d baseline, per cluster, environment, and response code class (4/5).

Meaning

The expr is a slope test against history, same shape as the sibling traefik alert (16-traefik-elevated-error-spike.md):

Numerator — increase(envoy_http_downstream_rq_xx{envoy_response_code_class=~"4|5"}[1h]) summed per k8s_cluster_name, environment, envoy_response_code_class.
Denominator — increase(...[7d] offset 1h) / (7 * 24) — the rolling 7d mean hourly count, excluding the current hour. The 7d window spans a full week so weekends and holidays — which naturally depress traffic — are only 2/7 of the baseline instead of dominating a 24h window, preventing false spikes on Monday mornings / post-holiday mornings.
Fires when the 1h count exceeds 6x the baseline and the >10/h floor holds. The 6x threshold (up from 3x in the prior 24h-based version) accounts for the wider, smoother baseline — see the sibling traefik runbook for the full rationale and backtest numbers.

The sum by (...) exists because downstream counters are emitted once per envoy_http_conn_manager_prefix (one series per connection-manager prefix; see 15-envoy-overload-rejecting-requests.md): without it the alert fires one instance per prefix.

Class-level caveat — envoy only exports response classes (envoy_response_code_class 1–5), not exact codes: this alert cannot distinguish a 401 from a 403 or a 404. Traefik metrics carry exact codes and front most user-facing auth traffic (assistant, keycloak), so the traefik sibling is the fine-grained auth surface; this alert covers the envoy data planes (LLM/AI gateways etc.) at class level. Exact-code counters (..._upstream_rq_400-style) do not exist in Mimir today — enabling them needs an envoy-gateway stats change, not just a scrape addition.

Downstream vs upstream — this alert reads envoy_http_downstream_rq_xx, what envoy returned to clients. Backends can be failing much harder upstream while retries mask it from clients — compare the upstream view (Diagnosis step 2) against the downstream rate when the two disagree.

Impact

Client-visible error ramps on the LLM/AI gateway surfaces — model requests failing (5xx) or being rejected upstream (4xx).
Baselines on these gateways can be large but persistent — the ratio test is what turns persistent background into event detection.

Diagnosis

Labels give k8s_cluster_name, environment, envoy_response_code_class.

1. Confirm which gateway and how the rate evolved

QUERY='sum by (namespace, pod) (rate(envoy_http_downstream_rq_xx{envoy_response_code_class="<class>", k8s_cluster_name="<cluster>"}[15m]))'

Check whether one pod or the whole data plane is throwing the class — a single noisy pod points at the heap/overload chain (13-envoy-heap-saturation.md), a uniform ramp points at upstreams.

2. Which upstream cluster is failing — attribution via `label_replace`

rate()/increase() drop the __name__ label, and for envoy cluster stats the cluster name is embedded in the metric name (envoy_cluster_<name>_upstream_rq_xx). Extract it before the rate:

topk(8, sum by (envoy_cluster)
  (rate(label_replace(
    {__name__=~"envoy_cluster_.*_upstream_rq_xx", envoy_response_code_class="<class>", k8s_cluster_name="<cluster>"},
    "envoy_cluster", "$1", "__name__", "envoy_cluster_(.*)_upstream_rq_xx")[15m:1m])))

Counter semantics: upstream_rq_xx counts what the backends returned — with retries configured it will exceed what clients observed; treat it as the upstream-failure view, not the client view.

3. Dashboard and logs

Grafana dashboard Team Platform / LLM Proxy / Envoy AI Gateway — 5xx & Upstream Health (uid llm-proxy-5xx) shows the client-visible surface.
Loki streams for the gateway pods: service names in the envoy-gateway, envoy-shared-envoy-* family — filter {service_name=~"envoy.*", k8s_cluster_name="<cluster>"}.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production, as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix per outcome of step 1

Whole pod throw / heap chain signatures — follow 13-envoy-heap-saturation.md (limit bump / restart).
Specific upstream cluster failing — investigate that backend (deployment status, recent rollout, resource saturation), not the gateway.
New-class-only spike over previously silent baseline — check for a fresh routing/config rollout of a new-ish integration; a brand-new consumer can also legitimately move a class's baseline, in which case adjust expectations, not the alert.

Verify

The class's ratio expr returns nothing (pending holds cleared).
Downstream rate(envoy_http_downstream_rq_xx{envoy_response_code_class="<class>"}[15m]) back to pre-episode level on the affected cluster.

Notes

All environments — no environment="production" filter.
Group family — shares group envoy.proxy with the heap chain alerts (13/14/15); the same evaluation interval applies.
Metric-availability note — this alert relies only on envoy_http_downstream_rq_xx (scraped by synapse-client today, no chart change needed). Exact-code alerting on envoy would first require enabling per-code stats (envoy-gateway stats matcher / patch policy) in the charts.
Sibling alert — traefik (16-traefik-elevated-error-spike.md) has exact codes and most of the user-facing auth traffic.

Envoy Elevated Error Spike (4xx/5xx)