Traefik 402 Payment Failure (Multiple Distinct Users)

Runbook for when dozens of distinct users receive 402 Payment Required responses through traefik within an hour. Fires when at least 50 different user IDs got a 402 in the last hour on the same cluster, and deliberately does not fire while only one or a handful of users fail, no matter how aggressively their clients retry.

Meaning

Traefik logs every proxied request as a JSON access log line in Loki. Every 402 from the billing gate (Maestro/Metronome credit-balance check in copilot-api-v2 and friends) is attributable to an authenticated user — traefik preserves the X-User-Id header as request_X-User-Id in the access line, which the JSON parser surfaces as the label request_X_User_Id. The traefik_service_requests_total metric in Mimir has no user label, so by volume alone one credit-exhausted customer automation in a retry loop is indistinguishable from the same number of 402s spread across many customers. The Loki access logs carry the user identity, so this alert counts distinct user IDs that received a 402 in a 1h window, per k8s_cluster_name and environment, and fires when that count reaches 50.

Two pipeline details that matter (both verified live before this runbook was merged):

• | keep k8s_cluster_name, environment, request_X_User_Id — keep drops every label not listed, including stream labels. An earlier draft kept only the parsed fields; the count by (k8s_cluster_name, ...) then grouped on empty labels, merging all clusters into one series and rendering the summary as "… on ()". Stream labels must be repeated in the keep list to survive.
• **|= \"DownstreamStatus":402`** — a raw substring filter *before* the | jsonparse, so only actual 402 lines (a few hundred per day, ~100/h during an incident) are JSON-parsed instead of the ~410k lines/h the traefik streams carry fleet-wide.DownstreamStatus` is a unique key in the access-log JSON, so the exact substring cannot match another field. Counts were verified identical to the parse-then-filter approach.

Threshold evidence (measured 2026-10-28, live clusters):

• Single-user credit exhaustion — 1 distinct user: no page (the incident that motivated this alert, see Live examples).
• Chronic synthetic-probe background — the synthetic-testing-service probes billing-gated endpoints with a handful of accounts (~2 distinct users per hour observed on prod-weu, ≈4 per day): no page.
• Large-scale billing failure (Metronome outage, billing-gate misconfig deploy, bulk-credit expiry) — 50+ distinct users in the hour: page.

Threshold design — flat floor of 50, and what it deliberately does not cover

The floor was set at 50 (raised from an initial draft of 5 after fleet measurement, alert-owner decision 2026-10-28): the alert is the scale detector for 402s — "dozens/hundreds of users failing" — not the catch-all for any multi-user 402 event. What that costs and why it was accepted:

• A ratio to active users (402 users / active users > 0.5) was rejected first: it hides a partial outage (a 20% rejection rate on prod-weu ≈ 40 customers = a major incident) and its denominator parses every logged-in traefik line (~410k/h fleet-wide) on every evaluation minute, forever.
• A floor of 5 was the middle ground and trips a total billing outage on every cluster (prd-par01's entire active base is ~5 users/h). It was raised to 50 on the judgment that clusters this small cannot distinguish "mass event" from "a couple of customers ran out of credit in the same hour" — their whole user base is inside the noise band. At 50, the alert only means something when it fires.
• Coverage implications: a total billing outage on prd-par01 (~5 users) or prod-us (≤18 users) does not fire this alert, and a prod-weu outage during European overnight (24–43 active users/h) only fires once morning traffic crosses 50 affected users. A shared-cause billing event at that scale is still caught on prod-weu during business hours; smaller clusters rely on the envoy 4xx-class sibling slope alert (17-envoy-elevated-error-spike.md) and on the general 4xx spike alert's coverage of other codes. If small-cluster coverage is later deemed necessary, lower the floor or add per-cluster floors via a runbook PR — the expression itself is unchanged; only the comparison value moves.

Notes on the expression mechanics:

• request_X_User_Id != "" — the billing gate only answers 402 after JWT authentication, so 402s always carry a user ID; the matcher drops any hypothetically userless line rather than counting an empty label as a "user".
• >= 50 — LogQL comparison as part of the expr; the fired alert carries the distinct-user count as $value.
• for: 15m + the 1h window — a short (minutes) billing-provider blip that 402s 50+ users still fires, because the 1h window holds the count and the breach persists for 15m of evaluations; a random one-off excursion of a few users does not.
• oncall: true — P2 + production/staging + the oncall flag routes this alert to the Platform high-priority path that pages the platform on-call including outside working hours. Without the flag, a P2 pages only within working hours — wrong for a mass billing failure.

Impact

• All billing-gated workflows fail for multiple users at once — medical coding (/v2/tools/coding/), document processing (/v2/documents/), audio-bridge streams (/audio-bridge/.../streams). Every request from the affected users is rejected with 402 until the billing path recovers; this is a full customer-facing outage of the paid product surface for those customers.
• Not a capacity alert and not a gateway-availability alert — the gateway and the APIs are healthy and answering fast; the billing gate is refusing to meter/serve.
• Severity depends on where the root cause lives: inside our billing gate (deploy gone wrong — fix fast, likely revert) or upstream (Metronome outage — coordinate with finance/billing and communicate to customers).

Diagnosis

The alert labels give k8s_cluster_name and environment. From the alert, note them.

1. Confirm the user breadth and attribute 402s to services

Distinct users that got 402 in the last hour, per user (note the same keep discipline — stream labels in the keep list):

TOKEN=$(az keyvault secret show --vault-name DevOpsTeam --name synapse-grafana-api-token-terraform-deployments --query value -o tsv)
QUERY='count by (request_X_User_Id) (count_over_time({service_name="traefik", k8s_cluster_name="<cluster>"} |= `"DownstreamStatus":402` | json | keep k8s_cluster_name, request_X_User_Id | request_X_User_Id != "" [1h]))'
curl -s -G -H "Authorization: Bearer $TOKEN" \
  "https://synapse.corti.app/api/datasources/proxy/uid/loki/loki/api/v1/query" \
  --data-urlencode "query=$QUERY" | jq -r '.data.result | length'

Service attribution — which backend is rejecting (copilot-api-v2, audio-bridge, ...) and the shape over time:

QUERY='topk(10, sum by (exported_service) (increase(traefik_service_requests_total{code="402", k8s_cluster_name="<cluster>"}[1h])))'
curl -s -G -H "Authorization: Bearer $TOKEN" \
  "https://synapse.corti.app/api/datasources/proxy/uid/mimir/api/v1/query" \
  --data-urlencode "query=$QUERY" | jq -r '.data.result[] | "\(.metric.exported_service): \(.value[1])"'

2. Read the billing-gate logs — why 402

The 402-rejecting application logs the reason. Query the attributed service's logs in Loki (copilot-api-v2 shown — the usual source):

{service_name="copilot-api-v2", k8s_cluster_name="<cluster>"} |~ "out of credit balance|failed to meter success|Metronome"

Expect one of:

• Maestro: out of credit balance with a customer_id — the gate believes the customer's credit is exhausted. For a multi-user event, the same error across many different customer IDs points at a top-up/credit feed failure or an upstream Metronome problem, not at several customers independently going broke in the same hour. One shared customer_id across all affected users means a single tenant org exhausted its pooled credit.
• maestro: failed to meter success request ... missing Metronome customer ID in event attributes — metering/mapping is broken (empty Metronome customer ID); usage is not being metered, which can cascade into credit-balance errors.

3. Determine the failure class

• Upstream Metronome/Maestro outage or API breakage — check the Metronome status page; many different customers fail within the same window with balance-check errors. Open a finance/billing ticket; consider a comms update to customers.
• Billing-gate config deploy gone wrong — correlate the start of the 402 ramp with the last copilot-api-v2 / platform deploy on the cluster (kubectl -n <tenant> rollout history deployment/copilot-api-v2); if the ramp starts exactly at a rollout, treat the deploy as suspect and coordinate with the owning team before reverting.
• Credit feed / bulk-credit expiry — a mass scheduled credit expiry or a failed bulk top-up; finance/billing can confirm from Metronome. The affected users' individual balances would show the same cliff.
• Synthetic testing or a single tenant's retry storm (should NOT have fired) — verify via the traefik access logs that the users really are distinct (not a single ClientHost / azp=synthetic-testing JWT). If it was the synthetic probe suite or one customer, treat it as known background and raise the floor via a runbook PR.

4. Cross-check the sibling slope alert

16-traefik-elevated-error-spike.md excludes code 402, so it stays quiet during billing events; if 5xx or 401 is also spiked on the same cluster, a shared dependency (Keycloak, the API itself) is more likely than billing.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production, as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix by failure class

• Upstream billing outage — coordinate with finance/billing; if the platform supports a bypass/degraded mode for the credit check, that is a product decision with the copilot-api team — do not flip it ad hoc.
• Suspect deploy — coordinate with the owning team on a revert: kubectl -n <tenant> rollout undo deployment/copilot-api-v2 (confirm the tenant namespace from the service attribution first).
• Credit feed expiry — finance/billing re-issues top-ups; customers recover as balances refresh, no platform action beyond comms.

Step 3 — Verify

The alert clears when the distinct-user count falls below 50 — keep in mind the 1h window and keep_firing_for: 30m keep it firing up to 90m after the last affected user's 402.

Verify

• The distinct-user count query (Diagnosis step 1) returns fewer than 50 entities, and no new out of credit balance / metering errors appear in the attributed service's Loki stream.
• The billing gate is confirming: a spot-check request to a billing-gated endpoint (/v2/tools/coding/) from a test account succeeds and meters.

Live examples (2026-10-28)

The negative example that motivated this alert's design — everything the sibling slope alert paged on, this alert correctly ignored:

• prd-par01, tenant fr, customer 0e4ad3dc-ed1a-48df-b7e5-dc6983a2db2f: the customer's integration (axios/1.20.0, single user 6146aea3-3129-4211-89d1-27b65854aa0f, POST /v2/tools/coding/) exhausted its Maestro credit balance and retried every 1–2 minutes for hours. 241 402s in 24h (~96–109/hour peak), where the rolling-7d hourly average was ~0.5 — the sibling volume alert fired at "177.1x", a false page for platform on-call. Distinct-user count during the same window: 1. This alert: silent, as designed — including on the hours where the retrying customer was the only active user on prd-par01 (1 active user overnight).

Validation queries executed during the original PR review (2026-10-28, ~09:50 UTC): distinct-402 users in the last hour were 1 on prd-par01 (the incident, correctly silent) and 2 on prod-weu (synthetic background, correctly silent — also below the floor); active-user counts the same hour: prd-par01 5, prod-us 7, prod-weu 205, dev-weu 31, staging-eu 8.

Notes

• LogQL alert — uses logql: (not expr:); the Loki ruler evaluates it and dispatches via the Mimir Alertmanager. The signal source is the traefik JSON access logs in Loki, not a Mimir metric — traefik_service_requests_total carries no user label, which is precisely why the volume-slope sibling cannot make this distinction.
• keep drops stream labels — this is the reason the keep list includes k8s_cluster_name and environment: keep discards every unlisted label, including stream labels; without them in the list, the per-cluster count by groups on empty labels and the fired alert reports an empty cluster. The sibling LogQL alert 07-kubernetes/19-api-v2-debug-endpoint-called.md was fixed for the same bug in the same PR.
• Evaluation cost — each evaluation scans the traefik streams for the 1h window (~410k lines/h fleet-wide, measured 417,910 during validation); the |= \"DownstreamStatus":402`` line filter ensures only the ~100/h matching lines are JSON-parsed. Counts via the prefilter were verified identical to parse-then-filter.
• Sibling alerts — 16-traefik-elevated-error-spike.md covers 4xx/5xx volume slope per exact code but excludes 402 (see its Notes); 17-envoy-elevated-error-spike.md covers the same slope for envoy data planes by response class (exact codes don't exist as envoy metrics — a mass 402 shows up there as a 4xx-class spike).
• Threshold — 50 distinct users, one alert per cluster/environment — well above the measured background (~2 users/h synthetic, 1-user customer incidents) and set so the alert means scale: dozens of users failing simultaneously. A total billing outage only trips it where ≥50 users are actually active — at current fleet scale that is prod-weu business hours; smaller clusters and overnight windows are deliberately outside this alert's coverage (accepted trade-off, see Meaning). Raise/lower the floor via a runbook PR — only the comparison value moves.
• Userless 402s — hypothetically a 402 without an X-User-Id (the billing gate answers only post-auth, so none observed) would not be counted; if that ever shows up in the access logs, add a count by (... ClientHost ...) fallback arm to the expression.