Traefik Elevated Error Spike (4xx/5xx)

Runbook for when the rate of 4xx/5xx responses through traefik rises well above its rolling 7d baseline, excluding 402 — payment/quota responses are split out to their own mass-user alert (05-observability/18-traefik-402-payment-failure-multiple-distinct-users.md; see Notes). One alert per status code, per cluster, per environment — {{ $labels.code }} is a single exact code, so authentication failures (401) never blend with authorization failures (403), and a client-abort (499) never masks a server error (5xx).

Meaning

The expr is a slope test against history:

• Numerator — increase(traefik_service_requests_total{code=~"4..|5..", code!="402"}[1h]) summed per k8s_cluster_name, environment, code: errors observed in the last hour, per exact status code.
• Denominator — increase(...[7d] offset 1h) / (7 * 24): the rolling 7d mean of the same code's hourly error count, excluding the current hour. The 7d window spans a full week so weekends and holidays — which naturally depress traffic — are only 2/7 of the baseline instead of dominating a 24h window. This prevents false spikes on Monday mornings when business-hour traffic resumes after a quiet weekend.
• Fires when the 1h count exceeds 6x that average and exceeds 10 errors/hour (the floor keeps 1–2-error speckles quiet on low-traffic codes). The 6x threshold (up from 3x in the prior 24h-based version) accounts for the wider baseline: the 7d average is smoother, so a higher ratio is needed to distinguish a real ramp from normal weekly variance. A surge over a previously silent code produces an infinite ratio and still fires.

Two properties to know before triaging:

• The 7d baseline adapts. A ramp that climbs slowly for days raises its own baseline; the alert catches sustained multi-hour increases, not day-over-day drift. Explosive ramps blow straight through it. The weekly window is long enough that a single quiet weekend does not depress the baseline enough to cause a false Monday-morning spike.
• Every 4xx/5xx code is an individual signal — including the business-protocol ones (426 Upgrade Required from websocket probing, 404 route misses, 499 client aborts). They are deliberately not excluded: a new failure mode can arrive in any code, and each code's baseline is separate, so expected background in one code cannot hide a ramp in another. The one exception is 402: it carries no per-request user attribution in the metric, so a single customer automaton in a retry loop is indistinguishable from a mass billing outage by volume alone — the code is excluded here and covered by a distinct-user companion alert (18-traefik-402-payment-failure-multiple-distinct-users.md) that only fires when 50 or more different users are failing. Expect these as the recurring daily background (see Known chronic background below).

Impact

• Authentication-failure ramps affect every protected workflow — users cannot start sessions (401) or use authorized operations (403).
• 5xx surges indicate upstream services erroring behind the gateway.
• Not a capacity alert — the gateway is up; it is a traffic-quality anomaly.

Diagnosis

The alert labels give k8s_cluster_name, environment, code.

1. Confirm the surge and attribute it to services

TOKEN=$(az keyvault secret show --vault-name DevOpsTeam --name synapse-grafana-api-token-terraform-deployments --query value -o tsv)
QUERY='topk(10, sum by (exported_service, code) (increase(traefik_service_requests_total{code=~"4..|5..", k8s_cluster_name="<cluster>", environment="<environment>"}[1h])))'
curl -s -G -H "Authorization: Bearer $TOKEN" \
  "https://synapse.corti.app/api/datasources/proxy/uid/mimir/api/v1/query" \
  --data-urlencode "query=$QUERY" | jq -r '.data.result[] | "\(.metric.code) \(.metric.exported_service): \(.value[1])"'

Then look at the surge shape (does it start exactly now, or has it been climbing?):

QUERY='sum by (code) (increase(traefik_service_requests_total{code="<code>", k8s_cluster_name="<cluster>"}[1h]))'
# range over the last 24h, 1h step

2. Read the service's logs (Loki)

The dig by the top-exported_service attribution, using the Loki streams for the cluster:

{service_name=~".*keycloak.*|assistant|authorization-service", k8s_cluster_name="<cluster>"} |= "401"

Adjust the filter to the attributed service and the exact code. Check what changed with the failure reasons found (expired sessions, invalid tokens, realm mismatches, revoked credentials — whatever the logs show).

3. Chronic background — deprioritize these patterns

Some codes carry recurring legitimate/background traffic that makes high baselines normal, not incidents:

• 426 — websocket-upgrade probing on audio-bridge/WS endpoints.
• 402 — payment/quota responses. This alert no longer fires on 402 (the code is excluded from the expr since 2026-10-28): a single customer running out of credit retried by an automated client produced a 177x "spike" from 1 distinct user on prd-par01, which is a customer-support matter, not a platform incident. 402 volume is covered instead by the companion alert 18-traefik-402-payment-failure-multiple-distinct-users.md, which counts distinct user IDs receiving 402 and only fires when 50 or more different users are failing within an hour. For manual triage of 402s: a rectangular (hard-edged) 402 plateau uniform across many tenants is usually the synthetic-testing-service probe suite in the shared namespace (client_id synthetic-testing, shared metronome customer id) hitting billing-gated endpoints (/audio-bridge/.../streams, /v2/documents/, /v2/tools/) — the platform's billing gate answers 402 for the un-metered synthetic account. Verify via traefik access logs (single ClientAddr pod IP + azp=synthetic-testing JWT) before treating it as an incident.
• 404 — route misses and probed paths.
• 499 — client aborts under load or timeouts.
• Daytime 401/400 lifts — diurnal load on tokens/sessions that expire on schedule.

A 3x/2h-or-more excursion in these codes still means elevated, just not necessarily new — check the shape (step 1) before paging anyone.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production, as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the root cause by failure class

• 401 — Keycloak health and recent claim/config changes (kubectl -n shared get pods -l app=keycloak); for embedded auth, check the session handling and any recent identity-platform changes.
• 403 — authorization-service / Casbin policy changes; a policy rollout can flip permission responses fleet-wide.
• 400/409/422 — a client bug or API contract change; contact the owner of the attributed service.
• 5xx — upstream health: check the attributed service's pods and recent deployments/dashboards before investigating the gateway itself.

Step 3 — Verify

The alert clears when the ratio falls below 6x (or the floor) — keep in mind keep_firing_for: 30m rides out small excursions after recovery.

Verify

• The ratio expr for the code returns nothing (or a pending series clears its 2h/30m hold).
• The attributed service shows no new failure lines in Loki since the fix.

Notes

• All environments — no environment="production" filter; the alert includes {{ $labels.environment }} so each fired instance names its env.
• code=~"4..|5..", code!="402" — the selector matches every 4xx/5xx code except 402, and also excludes traefik transport-level failures (<error> code) and non-numeric codes (OK, Switching Protocols, 0, 2) — gateway transport errors are a separate failure mode, not a status code. 402 is excluded deliberately (see the 402 bullet under Chronic background): it has no user attribution in the metric, so volume alone cannot distinguish one credit-exhausted customer automaton from a mass billing failure. The mass-user 402 signal lives in its own companion alert.
• Metric source — traefik_service_requests_total is scraped from traefik pods by synapse-client and remote-written to central Mimir; no chart PR is needed. The entrypoint-level variant traefik_entrypoint_requests_total has the same shape if the per-service attribution is ambiguous.
• Sibling alerts — 05-observability/17-envoy-elevated-error-spike.md covers the same slope per response-class for envoy-gateway data planes (exact codes do not exist as envoy metrics); 05-observability/18-traefik-402-payment-failure-multiple-distinct-users.md covers the 402 code that this alert excludes, keyed on distinct users rather than volume.