Keycloak supports the OAuth2 client_credentials grant — a machine-to-machine client authenticates with its own credentials (not a user) and gets a service-account token. For this to work, the client must have Service Accounts Roles enabled in the Keycloak admin console (Client → Settings → Service Accounts Roles → ON). When a client calls the token endpoint with grant_type=client_credentials but this setting is off, Keycloak rejects the request and logs "Client not enabled to retrieve service account".

This alert fires when that log line appears at all (> 0 in 10m) — even a single rejection means a client is misconfigured and cannot get a token.

Common causes:

Pre-platform-api client — the client was created before the managed-by=platform-api standard, which enables service accounts by default. Legacy clients may not have the setting.
Manually toggled off — someone disabled Service Accounts Roles in the Keycloak UI.
New client misconfigured — a recently created client (via API or UI) is missing the serviceAccountsEnabled flag.

Impact

The affected client cannot obtain service-account tokens — any machine-to-machine flow using client_credentials against this client fails with an error (HTTP 400 {"error":"invalid_client","error_description":"Client not enabled to retrieve service account"}).
Only the specific misconfigured client is affected — other clients on the same realm are not.
This is a configuration bug, not a Keycloak outage — Keycloak itself is healthy and serving other clients.

Diagnosis

The alert label carries k8s_cluster_name. From the alert, note it.

1. Find the offending client in the Keycloak logs

export KUBECONFIG=~/.kube/<cluster>.yaml

Query Loki for the full log line (the alert only counts it; the line has the client ID):

# via Loki (port-forward synapse-server-loki-gateway in monitoring-system, header X-Scope-OrgID: corti)
QUERY='{service_name="keycloak", k8s_cluster_name="<cluster>"} |= "Client not enabled to retrieve service account"'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:3100/loki/api/v1/query_range" \
  --data-urlencode "query=$QUERY" --data-urlencode "limit=5" | jq -r '.data.result[0].values[0][1]'

Or directly from the Keycloak pod:

kubectl -n shared logs -l statefulset=keycloak --tail=200 | grep "Client not enabled to retrieve service account"

The log line includes the clientId — note it for step 2.

2. Check the client's service-accounts setting in Keycloak

Via the Keycloak admin console (or REST API):

# Get the client's configuration (requires admin credentials)
# Realm is typically the tenant realm — check which realm the client is in
KC_URL=http://keycloak.shared:8080
REALM=<realm>
CLIENT_ID=<clientId from step 1>
TOKEN=$(kubectl -n shared exec deploy/keycloak -- curl -s "$KC_URL/realms/master/protocol/openid-connect/token" -d "grant_type=password" -d "client_id=admin-cli" -d "username=<admin>" -d "password=<pass>" | jq -r .access_token)

kubectl -n shared exec deploy/keycloak -- curl -s "$KC_URL/admin/realms/$REALM/clients" \
  -H "Authorization: Bearer $TOKEN" | jq ".[] | select(.clientId==\"$CLIENT_ID\") | {clientId, serviceAccountsEnabled}"

serviceAccountsEnabled: false confirms the misconfiguration.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Enable Service Accounts Roles on the client

In the Keycloak admin console:

Select the correct realm (the tenant realm the client belongs to).
Go to Clients → find the clientId from Diagnosis step 1.
Settings → Service Accounts Roles → toggle ON.
Save.

Or via the Keycloak REST API:

CLIENT_UUID=$(kubectl -n shared exec deploy/keycloak -- curl -s "$KC_URL/admin/realms/$REALM/clients" \
  -H "Authorization: Bearer $TOKEN" | jq -r ".[] | select(.clientId==\"$CLIENT_ID\") | .id")

kubectl -n shared exec deploy/keycloak -- curl -s -X PUT "$KC_URL/admin/realms/$REALM/clients/$CLIENT_UUID" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"serviceAccountsEnabled": true}'

Step 3 — Verify the client can get a token

kubectl -n shared exec deploy/keycloak -- curl -s "$KC_URL/realms/$REALM/protocol/openid-connect/token" \
  -d "grant_type=client_credentials" -d "client_id=$CLIENT_ID" -d "client_secret=<secret>" | jq .

A valid token response (with access_token) confirms the fix. If the response is still invalid_client, the setting did not save — recheck.

Step 4 — If the client should NOT use service accounts

If the client is not supposed to use client_credentials (a public or confidential client that should only do authorization code flow), then the caller is misconfigured, not the Keycloak client. Fix the caller to use the correct grant type instead of enabling service accounts.

Verify

The alert clears once no new "Client not enabled to retrieve service account" log line appears for 10m.
The client's token request (step 3) returns a valid access token.

Notes

LogQL alert — queries Loki, not Mimir. The alert is evaluated by the Loki ruler, not the Mimir ruler.
> 0 — fires on a single occurrence; even one rejection means a client is broken.
All environments — no environment= filter; fires on any cluster running Keycloak.
for: 0m — fires immediately; a misconfigured client is an active problem.
Notify — the DataDog message template says to notify hc@corti.ai.
Related — 07-kubernetes/21-platform-api-server-errors.md (platform-api manages clients; a 500 from platform-api may co-occur if it cannot reach Keycloak).