Same metric as 07-kubernetes/21-platform-api-server-errors.md — traefik_service_requests_total with exported_service=~".*platform-api.*". This alert fires on 4XX client errors, excluding 404 (which is valid behavior — a user not part of any realm or tenant gets a 404).

sum by (k8s_cluster_name) (increase(traefik_service_requests_total{exported_service=~".*platform-api.*", code=~"4..", code!="404", environment="production"}[5m])) > 10 — more than 10 non-404 4XX errors in 5 minutes in production.

Scoped to environment="production" (matching DataDog). The remaining 4XX codes of concern:

401 — Unauthorized. Token validation failed; may indicate a Keycloak issue or expired client credentials.
403 — Forbidden. The user's role/scope does not allow the operation; may indicate a Casbin policy misconfiguration or a changed permission.
400 — Bad Request. Client sent malformed input; may indicate a client bug or an API contract change.
409 — Conflict. Concurrent modification or duplicate; usually valid behavior but elevated rates may indicate a client retry storm.
422 — Unprocessable Entity. Validation failure; usually valid behavior.

Impact

Clients are receiving client-side errors — some may be legitimate (409, 422), but elevated 401/403 may indicate an auth or authorization problem affecting many users.
Not an availability issue — the service is up and responding, just rejecting requests. Lower urgency than 5XX.

Diagnosis

The alert labels carry k8s_cluster_name and environment. From the alert, note both.

1. Confirm the 4XX count and identify the error codes

export KUBECONFIG=~/.kube/<cluster>.yaml

Confirm via Mimir:

QUERY='sum by (code) (increase(traefik_service_requests_total{exported_service=~".*platform-api.*", code=~"4..", code!="404", environment="production", k8s_cluster_name="<cluster>"}[5m]))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[] | "\(.metric.code): \(.value[1])"'

2. Check platform-api logs for the specific errors

kubectl -n shared logs -l app=platform-api --tail=200 | grep -iE "401|403|400|409|422|unauthorized|forbidden|bad request"

3. For 401/403 — check Keycloak and Casbin

Elevated 401/403 is the main concern:

401 — check Keycloak is healthy and the client credentials / realm are not expired (kubectl -n shared get pods -l app=keycloak).
403 — check the Casbin policy in the authorization-service; a recent policy change may have broken permissions.

4. For 400/409/422 — check for a client bug or API contract change

These are usually valid behavior, but a sudden spike may indicate:

A client sending malformed requests after an API change (400).
A retry storm causing conflicts (409).
A validation failure from a new field (422).

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production, as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the root cause

401 from Keycloak — fix Keycloak (restart, check realm/client config).
403 from Casbin — check the authorization-service Casbin policy; revert a recent policy change if it broke permissions.
400/409/422 from client bug — coordinate with the client team to fix the malformed requests.

Step 3 — Verify

The alert clears once sum by (k8s_cluster_name) (increase(traefik_service_requests_total{exported_service=~".*platform-api.*", code=~"4..", code!="404", environment="production"}[5m])) drops to <= 10 for the cluster.

Verify

The Mimir query from Diagnosis step 1 returns <= 10 for all non-404 4XX codes.
kubectl -n shared logs -l app=platform-api --tail=20 shows no new 401/403 errors.

Notes

404 excluded — code!="404": 404 is valid behavior when a user is not part of any realm or tenant.
Production only — environment="production" (matching DataDog).
for: 0m — fires immediately.
Sibling — 07-kubernetes/21-platform-api-server-errors.md (5XX server errors, all envs).