This check runs once per cluster, not once per tenant, because every tenant reaches the same Corti Models gateway. It logs in as one tenant: base in most clusters, fr in prd-par01 and eu-s in prd-par03. Most failures therefore affect every tenant in the cluster. Two do not: a billing failure affects only the internal account the check is billed to, and a refused access token may affect only the tenant the check logged in as. The table below says which is which.

How to use this runbook

The alert names each model that failed in the last 10 minutes, for example:

Corti Models – Corti Models is failing for tenant base on prod-us (production): corti-s1-embedding

If the alert names no model, open Why it failed: it still shows which models failed.

Open Why it failed in the alert. The error names the step the check stopped at.
If it is acquire-token, no model was called: see acquire-token below. The alert then names every model, because the check could reach none of them.
If it is call-models, the error reads step "call-models": followed by one line per model that failed, as <model>: <reason>. A model that is not listed answered correctly. Find each reason in the call-models table below.

Cluster access

Steps that use kubectl need Azure Privileged Identity Management first in production: activate the MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged) roles.

export KUBECONFIG=~/.kube/<cluster>.yaml

prd-par03 is not an AKS cluster. Its kubeconfig comes from the cluster's page in Kommodity (Copy config), and signing in with it needs the kubectl oidc-login plugin. Platform Team (privileged) is the role that grants access to it.

The journey

Feature	What the check does	Steps, in order
Corti Models	Sends a chat completion to `corti-s1` and `corti-s1-mini` and requires an answer from each, and sends an embeddings request to `corti-s1-embedding` and requires a vector	`acquire-token` → `call-models`

acquire-token

The same step as in Speech to Text, with the same errors: see the acquire-token section of the Speech to Text runbook.

call-models

Calls each model at https://ai.<region>.corti.app. Each model has 30 seconds to answer.

Path: the Corti Models gateway → authorization-service (checks the access token, then the tenant's balance) → the model's backend.

Error text after `<model>:`	What it means	Customers affected?	Check first
`the billing balance has run out`	The gateway answered 402: the internal account the check is billed to has no balance left	No	authorization-service logs: `Insufficient balance`
`answered 401 Unauthorized` with nothing after it	authorization-service refused the access token	Only if its log line concerns more than the check's own token	authorization-service logs: `Invalid JWT`, `Unknown realm` or `JWT issuer does not match realm` gives the reason
`answered 401 Unauthorized: Jwks doesn't have key to match kid or alg from Jwt`	The model's backend refused the access token	Yes, anyone using that model in the cluster	The model's backend. The error comes from the backend, not from the Corti Models gateway
`answered 404 Not Found: No matching route found. It is likely because the model specified in your request is not configured in the Gateway.`	The gateway has no route for the model	Yes, anyone using that model	The model's route in the gateway
`answered 502 Bad Gateway`	Either authorization-service could not check the balance, or the gateway got no valid answer from the model's backend	Yes	authorization-service logs: `Balance check failed`. If there is no such line, the model's backend
`answered 5` followed by any other status	The gateway or the model's backend failed	Yes	`kubectl -n shared get pods -l gateway.envoyproxy.io/owning-gateway-name=envoy-public-llm-gateway` (namespace `infra` in prd-par01), then the model's backend
`the model returned no answer`	The model answered, but with no answer text	Likely	The model's backend
`the model returned no usable vector`	The embedding model answered, but the vector was empty or all zeros	Likely	The model's backend
`calling the model:` followed by a timeout or network error	The model did not answer within 30 seconds, or the request did not reach the gateway	Yes, if it continues	The gateway pods, then the model's backend

authorization-service logs

authorization-service logs every access token it refuses and every balance check that fails. In Synapse (https://synapse.corti.app), open Explore, select the Loki data source, set the time range to a few minutes around the failure, and run:

{k8s_cluster_name="<cluster>", k8s_namespace_name="shared", service_name="authorization-service"} |~ "Invalid JWT|Unknown realm|issuer does not match|Balance check failed|Insufficient balance"

Who owns a service

The owning team is the team: value in the service's configuration in the deployments repository: config/services/<service>/base/values.yaml.