The platform-api service (github.com/corticph/platform-api) is the identity & access management API — tenant lookup, client management, quotas, soft-delete. It is deployed in the shared namespace and fronted by Traefik, which exposes traefik_service_requests_total (a counter of HTTP requests per backend service and status code). The exported_service label identifies the backend (shared-platform-api-80@kubernetes); code is the HTTP status code.

The alert fires when sum by (k8s_cluster_name) (increase(traefik_service_requests_total{exported_service=~".*platform-api.*", code=~"5.."}[5m])) > 10 — i.e. more than 10 server-side errors (500, 502, 503, 504) in 5 minutes. for: 0m fires immediately — 5XX errors are an active problem, not a trend.

Scoped to all environments (no environment= filter) — code=~"5.." matches the DataDog code:5*.

Impact

Clients calling platform-api are receiving server errors — tenant lookups, client management, and quota checks fail.
If the error rate is high, the platform-api may be down or its dependencies (Keycloak, Kafka, Redis, Postgres) are unreachable.
502/503 typically means Traefik cannot reach the backend (no ready pods); 500 means the backend is erroring internally; 504 means the backend timed out.

Diagnosis

The alert labels carry k8s_cluster_name and environment. From the alert, note both.

1. Confirm the 5XX count and identify the error codes

export KUBECONFIG=~/.kube/<cluster>.yaml

Confirm via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

QUERY='sum by (code) (increase(traefik_service_requests_total{exported_service=~".*platform-api.*", code=~"5..", k8s_cluster_name="<cluster>"}[5m]))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[] | "\(.metric.code): \(.value[1])"'

The specific 5XX code tells you where to look:

500 — the platform-api pod is erroring internally. Check logs (step 2).
502/503 — Traefik cannot reach the backend. Check if pods are running (step 3).
504 — the backend timed out. Check downstream dependencies (step 4).

2. Check platform-api logs for errors

kubectl -n shared logs -l app=platform-api --tail=100 | grep -iE "error|panic|fatal|5[0-9][0-9]"

Common causes:

Keycloak unreachable (token validation fails → 500)
Postgres/Redis connection failure → 500
Panic in a handler → 500

3. Check if platform-api pods are running and ready

kubectl -n shared get pods -l app=platform-api -o wide
kubectl -n shared describe deploy platform-api | sed -n '/^Conditions:/,$p'

If pods are CrashLoopBackOff, see 07-kubernetes/02-pod-crashloop.md. If no pods are running, see 07-kubernetes/03-deployment-no-running-replicas.md.

4. Check downstream dependencies

# Keycloak
kubectl -n shared get pods -l app=keycloak
# Postgres (CNPG)
kubectl -n shared get pods -l app.kubernetes.io/name=cloudnative-pg
# Redis
kubectl -n shared get pods -l app=redis

If a dependency is down, that is the root cause — fix the dependency, platform-api recovers.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the root cause

Based on Diagnosis:

Pods crashing — see 07-kubernetes/02-pod-crashloop.md.
No pods running — see 07-kubernetes/03-deployment-no-running-replicas.md.
Keycloak down — see the Keycloak runbooks; restart or fix Keycloak.
Postgres/Redis down — see 06-databases/ runbooks; fix the database.
Handler bug (500s from a specific endpoint) — check the logs for the stack trace, fix the code, deploy a new image.

Step 3 — Verify

The alert clears once sum by (k8s_cluster_name) (increase(traefik_service_requests_total{exported_service=~".*platform-api.*", code=~"5.."}[5m])) drops to <= 10 for the cluster.

Verify

The Mimir query from Diagnosis step 1 returns <= 10 (or 0) for all 5XX codes.
kubectl -n shared logs -l app=platform-api --tail=20 shows no new errors.
Platform-api health endpoint returns 200: kubectl -n shared exec deploy/platform-api -- curl -s localhost:8081/health

Notes

Metric — traefik_service_requests_total with exported_service=~".*platform-api.*" (Traefik's exported_service label, not service).
5XX only — code=~"5.." matches 500, 502, 503, 504.
All environments — no environment= filter; fires on any cluster where platform-api runs.
for: 0m — fires immediately; 5XX is an active problem.
Sibling — 07-kubernetes/22-platform-api-client-errors.md (4XX client errors, production only).
Related — 07-kubernetes/02-pod-crashloop.md, 07-kubernetes/03-deployment-no-running-replicas.md.