Use this runbook when a Synthetic Feature Failing alert names Speech to Text. The alert also names the feature (Transcribe, Streams or Transcripts), the tenant and the cluster.

How to use this runbook

Open Where the journey stopped in the alert. It shows which step failed.
Open Why it failed in the alert. It shows the error text, and the session's ID where one exists: interactionId for Streams and Transcripts from create-interaction onwards, sessionId for Transcribe from receive-transcript onwards (the platform assigns it when it accepts the configuration). Transcripts also shows recordingId from upload-recording onwards and transcriptId from create-transcript onwards.
Find the step below and match the error text.

Cluster access

Steps that use kubectl need Azure Privileged Identity Management first in production: activate the MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged) roles.

export KUBECONFIG=~/.kube/<cluster>.yaml

Trace the session

Every service that handles a session logs its ID (as session_id or interaction_id). For Transcripts, the speech recognition services log the transcriptId as their session ID, and searching for either the interactionId or the transcriptId finds them. Searching for the ID returns every log line about that session, from every service, in time order. In Synapse (https://synapse.corti.app), open Explore, select the Loki data source, set the time range to a few minutes around the failure, and run:

{k8s_cluster_name="<cluster>", k8s_namespace_name=~"<tenant>|shared"} |= "<id>"

Before 25 September 2026, prod-weu and prod-us logs were labelled eu and us. Use those names only for older logs.

The service_name label on each line shows which service wrote it. The last service to log the session is where it stopped. A gap of many seconds after a service's lines means that service was slow to pass the session on.

If a Transcribe or Streams failure has no ID, read copilot-audio-bridge's lines for the synthetic testing client around the time of the failure instead:

{k8s_cluster_name="<cluster>", k8s_namespace_name="<tenant>", service_name="copilot-audio-bridge"} |= "synthetic-testing"

If only some Transcribe or Streams checks fail, compare the copilot-audio-bridge pod (k8s_pod_name) across the failed sessions. A single faulty pod fails only the sessions it handles.

The journeys

Feature	What the check does	Steps, in order
Transcribe	Real-time dictation on the `/transcribe` web socket	`acquire-token` → `open-session` → `receive-transcript` → `end-session`
Streams	Real-time conversation transcript on the `/streams` web socket, for an Interaction it creates	`acquire-token` → `create-interaction` → `open-session` → `receive-transcript` → `end-session`
Transcripts	Transcription of a recording, for an Interaction it creates: uploads a short pre-recorded English clip, asks for its transcript, and waits until the transcript is complete	`acquire-token` → `create-interaction` → `upload-recording` → `create-transcript`

A check stops at the first step that fails, so later steps never run.

Transcripts is checked only for tenants that run copilot-api-v2, ml-service-routing and voice-vault. A tenant without them, such as trifork (no voice-vault), cannot transcribe recordings and is not checked.

Each section below is one step. Used by lists the checks that run it, in this runbook and others.

acquire-token

Used by: Speech to Text – Transcribe, Streams and Transcripts; Text Generation – Streams, FactsR, Documents and Templates.

Gets an access token for the tenant from Corti authentication (Keycloak, realm <tenant>), using the synthetic testing client's credentials. It first reads those credentials from the Kubernetes secret synthetic-testing-keycloak-client in the tenant's namespace.

Error text contains	What it means	Customers affected?	Check first
`no credentials in the tenant's namespace`	The secret does not exist	No, synthetic-testing-service is broken	`kubectl -n <tenant> describe secret synthetic-testing-keycloak-client`. If it is missing, check that platform-service has provisioned the `synthetic-testing` client for this tenant
`the cluster refused to return the tenant's credentials`	synthetic-testing-service is not allowed to read the secret	No	`kubectl get clusterrole synthetic-testing-service -o yaml`. It must allow `get` on the secret `synthetic-testing-keycloak-client`
`credentials are missing a field`	The secret exists but lacks `client-id` or `client-secret`	No	`kubectl -n <tenant> describe secret synthetic-testing-keycloak-client` lists the fields it has
`authentication refused: invalid_client`	Keycloak rejected the synthetic testing client: its secret was rotated, or the client was deleted or disabled	No	The `synthetic-testing` client in the tenant's realm, in the Keycloak admin console
`authentication refused:` followed by an HTTP status such as `503`, or a network error	Keycloak is not answering properly	Yes, customers cannot get access tokens either	`kubectl -n shared get pods -l app=keycloak`, then `kubectl -n shared logs -l app=keycloak --tail=200`

create-interaction

Used by: Speech to Text – Streams and Transcripts; Text Generation – Streams.

Creates an Interaction (POST /v2/interactions/). The response returns the interactionId, and the web socket URL that Streams connects to.

Path: Traefik → authorization-service (checks the access token and its permissions) → copilot-api-v2.

Error text contains	What it means	Check first
`creating an interaction answered 401` or `403`	authorization-service refused the access token	authorization-service logs
`creating an interaction answered 5xx`	copilot-api-v2 is failing	copilot-api-v2 logs
`calling the platform:`	The request got no answer	Traefik
`the created interaction named no id`	copilot-api-v2 answered without an `interactionId`	copilot-api-v2 logs
`the created interaction named no socket` (Streams)	copilot-api-v2 answered without the web socket URL	copilot-api-v2 logs

After each run the check deletes its Interaction, which also deletes the recording and transcript Transcripts created. A failed delete is logged separately as probe left data behind and is not a failed step.

open-session

Used by: Speech to Text – Transcribe and Streams; Text Generation – Streams.

Connects to the /transcribe web socket (Transcribe) or the /streams web socket (Streams), sends the configuration (primaryLanguage: en; Streams also sends mode: transcription) and waits for CONFIG_ACCEPTED.

Path: Traefik → authorization-service → copilot-audio-bridge.

Error text contains	What it means	Check first
`expected handshake response status code 101 but got 401` or `403`	authorization-service refused the access token	authorization-service logs
`… but got 502` or `503`	Traefik has no healthy copilot-audio-bridge pod to pass the connection to	`kubectl -n <tenant> get pods -l app=copilot-audio-bridge`
`… but got` any other status	copilot-audio-bridge refused the connection	Trace the session
`the platform refused the session configuration:` followed by a reason	The platform refused the configuration. The reason says why	Trace the session
`awaiting the configuration response:` … `context deadline exceeded`	The check waited for `CONFIG_ACCEPTED` until its 1-minute run limit ran out	Trace the session

receive-transcript

Used by: Speech to Text – Transcribe and Streams.

Sends the audio packets of a short pre-recorded English clip and waits for the first transcript.

Behind copilot-audio-bridge, the audio passes through a chain of speech recognition services connected by Kafka, and the transcript comes back the same way. Which services, and in what order, depends on the language and the cluster. Tracing the session shows them.

Error text contains	What it means	Check first
`awaiting transcript` … `after 0 messages including 0 transcripts: context deadline exceeded`	The audio was sent, and the check waited for a transcript until its 1-minute run limit ran out. `0 messages` means the platform sent nothing back at all	Trace the session
`the platform reported an error while awaiting transcript:` followed by details	The platform sent an error message. The details say what	Trace the session
`the platform closed the connection`	copilot-audio-bridge dropped the connection	Trace the session
`the session ended before it transcribed anything`	The session ended early, for example because the tenant ran out of balance	Trace the session

end-session

Used by: Speech to Text – Transcribe and Streams; Text Generation – Streams.

Sends the end message and waits for the platform to confirm the stream has ended (ENDED for Streams, ended for Transcribe). The end message travels through the same chain of services as the audio.

Error text contains	What it means	Check first
`awaiting ENDED` (Streams) or `awaiting ended` (Transcribe) … `context deadline exceeded`	The check sent the `end` message and waited for the platform to confirm the stream ended, until its 1-minute run limit ran out	Trace the session

upload-recording

Used by: Speech to Text – Transcripts.

Uploads the clip to the Interaction (POST /v2/interactions/<interactionId>/recordings/, as audio/ogg). The response returns the recordingId.

Path: Traefik → authorization-service → copilot-api-v2, which stores the recording in voice-vault in the tenant's namespace.

Error text contains	What it means	Check first
`uploading the recording: the platform reported an error: answered 401` or `403`	authorization-service refused the access token	authorization-service logs
`… answered 400 Bad Request: Unable to process uploaded content`	copilot-api-v2 could not read the audio. The clip never changes, so something on the platform changed	copilot-api-v2 logs for the `interactionId`
`… answered 5xx`	copilot-api-v2 or voice-vault is failing	copilot-api-v2 logs for the `interactionId`, then `kubectl -n <tenant> get deploy voice-vault`
`uploading the recording: calling the platform:`	The request got no answer	Traefik
`the upload named no recording`	copilot-api-v2 answered without a `recordingId`	copilot-api-v2 logs

create-transcript

Used by: Speech to Text – Transcripts.

Asks for a transcript of the recording (POST /v2/interactions/<interactionId>/transcripts/, with the recordingId and primaryLanguage: en) and waits until it is complete with words in it. copilot-api-v2 holds the request for up to 25 seconds. If the transcript is not ready by then, it answers processing, and the check asks GET …/transcripts/<transcriptId>/status every second until the transcript is completed or failed, within its 1-minute run limit. The check gives the request itself 30 seconds.

copilot-api-v2 asks ml-service-routing which speech recognition to use, loads the recording back from voice-vault, and sends it in chunks to Kafka (topic <tenant>-audio-offline). From there, for English, it passes through ml-service-transcription-offline (speech recognition) → ml-service-corrector → ml-service-voice-command, and back to copilot-api-v2. In dev-weu, ml-service-formatting takes the place of ml-service-corrector. Other languages can be routed to Speechmatics instead, but the check's clip is English.

Error text contains	What it means	Check first
`requesting a transcript: the platform reported an error: answered` a status, followed by a reason	copilot-api-v2 refused or failed the request. The reason says why: `failed to resolve offline routing` means ml-service-routing did not answer	copilot-api-v2 logs for the `interactionId`; for routing, `kubectl -n <tenant> get deploy ml-service-routing`
`requesting a transcript: calling the platform:` … `Client.Timeout exceeded`	No answer within 30 seconds, although copilot-api-v2 should answer `processing` after 25	copilot-api-v2 logs for the `interactionId`
`waiting for transcript` … `context deadline exceeded`	The transcript was still `processing` when the check's 1-minute run limit ran out	Trace the session by the `transcriptId`: the last service to log it is where it stopped
`the platform reported the transcript failed`	copilot-api-v2 marked the transcript `failed`	Trace the session by the `transcriptId`
`the transcript came back without words`	The transcript completed, but with no text in it	ml-service-transcription-offline's lines for the `transcriptId`: `Transcribed … (N words)` says what speech recognition heard
`asking for the transcript's status:` or `fetching the completed transcript:`	A follow-up request for the transcript failed	copilot-api-v2 logs
`the transcript request named no transcript`, or `transcript "…" is "…"` with a status other than `completed`, `processing` or `failed`	copilot-api-v2 answered in a shape the check does not recognise	copilot-api-v2 logs

Example: a Transcripts check that passed (dev-weu, 2026-10-03)

A Speech to Text – Transcripts check in dev-weu base, tracing its interactionId:

Time (UTC)	Service	What it logged	What that means
10:21:22	copilot-api-v2	`Request` for `POST /interactions/<id>/recordings/` with status `201`	The clip was uploaded
10:21:23	copilot-api-v2	`resolved offline transcript routing` (topic `base-audio-offline`), `loaded audio from voice-vault`, then `starting chunking` and `ended chunking`	Routing chose speech recognition in the cluster, and the recording went to Kafka
10:21:23	ml-service-transcription-offline	`Start event received on session`, then `Received audio … for 25.73s duration`	Speech recognition received the whole clip
10:21:24	ml-service-transcription-offline	`Submitted 1 Triton request(s)`, then `Transcribed 25.66s VAD segment on channel 0 (73 words)`	It transcribed the clip in about half a second
10:21:24–25	ml-service-formatting, then ml-service-voice-command	`Start event received on session` … `Marked session as completed`, then `Returning messages`	The transcript passed through the remaining services

The check received the completed transcript and passed, 5.4 seconds after it started. When a check fails at create-transcript, the same trace shows how far the transcript got.

Live example (2026-09-22)

A Speech to Text – Transcribe check in dev-weu started at 10:19:48 and failed at receive-transcript with after 0 messages including 0 transcripts: context deadline exceeded. Tracing its sessionId:

Time	Service	What it logged	What that means
10:19:48	ml-service-routing	`received put session request` (language `en`, session type `transcribe`), then `found routing for request`	copilot-audio-bridge asked ml-service-routing which speech recognition route to use for English, and got one
10:19:48	ml-service-transcription-online	`Start event received on session`	ml-service-transcription-online was told a new session had started
10:19:54	copilot-audio-bridge	`received audio bytes`, then `audio sent`	copilot-audio-bridge received the check's audio and passed it on to ml-service-transcription-online
		nothing for 41 seconds
10:20:29	ml-service-transcription-online	`Session started with input settings`, then `Received audio of shape (1, 0) for 0.00s duration` and `outputting 1 transcription message with event 'start' and 0 transcription segments`	ml-service-transcription-online only now started working on the session, 41 seconds after being told about it. It had processed no audio yet, so it passed on the session start with no transcript
10:20:29	ml-service-formatting, then ml-service-voice-command	`Start event received on session`	The next services passed the session start along immediately
10:20:29	copilot-audio-bridge	`received usage data` with `AudioDurationSeconds: 0`, `TranscriptTokens: 0`	copilot-audio-bridge got the session start back, with no transcript to send to the check
10:20:48	copilot-audio-bridge	`websocket closed so cannot read`	The check's 1-minute run limit ran out and it disconnected

ml-service-transcription-online (speech recognition) was 41 seconds late starting on the session and had not processed any of the audio when the check gave up. Every other service responded immediately. Next step: find out from ml-service-transcription-online's logs why it was behind (owned by speech-recognition).

Who owns a service

The owning team is the team: value in the service's configuration in the deployments repository: config/services/<service>/base/values.yaml.