Documentation navigation
Text Generation: synthetic check failing
Use this runbook when a Synthetic Feature Failing alert names Text Generation. The alert also names the feature, the tenant and the cluster.
How to use this runbook
- Open Where the journey stopped in the alert. It shows which step failed.
- Open Why it failed in the alert. It shows the error text, and for Streams
the
interactionIdfromcreate-interactiononwards. FactsR, Documents and Templates store nothing, so they have no ID: find the request as described under extract-facts, generate-document or list-templates, open-template. - Find the step below and match the error text.
Cluster access, and tracing a session by its interactionId, work as described
under Cluster access and Trace the session in the
Speech to Text runbook.
The journeys
| Feature | What the check does | Steps, in order |
|---|---|---|
| Streams | Real-time facts on the /streams web socket, for an Interaction it creates: streams a short pre-recorded English clip with facts enabled, sends flush, and waits for facts |
acquire-token → create-interaction → open-session → receive-facts → end-session |
| FactsR | Fact extraction from supplied text: sends the same consultation as text, and requires facts back. It stores nothing | acquire-token → extract-facts |
| Documents | Writes up the same consultation as a SOAP note with Corti's standard English SOAP template, and requires a written document back. It asks the platform not to store the document, and deletes it if the platform stores it anyway | acquire-token → generate-document |
| Templates | Lists Corti's standard templates and requires the standard English SOAP template among them, then opens that template and requires a published version with at least one section, each with a name. It only reads | acquire-token → list-templates → open-template |
A check stops at the first step that fails, so later steps never run.
acquire-token, create-interaction, end-session
The same steps as Speech to Text, with the same errors (FactsR, Documents and
Templates use only acquire-token):
acquire-token,
create-interaction,
end-session.
open-session
The same step as
Speech to Text – Streams,
except the configuration asks for facts: mode: facts and outputLocale: en.
receive-facts
Sends the audio packets of the clip, then sends the flush message, and waits
for facts that arrive after the flush. Each fact must have a group and text.
The audio goes through the same speech recognition services as Speech to Text.
The transcript then reaches ml-service-facts (in the tenant), which extracts
facts by calling ml-service-summarisation (in shared). The facts return to
copilot-audio-bridge, which sends them to the check. Without flush,
ml-service-facts only extracts facts once a minute, which is longer than the
check's 1-minute run limit allows.
When a session opens, ml-service-facts sends one fact straight away, before any audio has arrived (in dev-weu: Patient gender is unknown). The check ignores it.
| Error text contains | What it means | Check first |
|---|---|---|
awaiting facts … after N messages including M transcripts: context deadline exceeded, with M more than 0 |
Transcripts came back, so speech recognition works, but no facts arrived before the check's 1-minute run limit ran out | Trace the session: find Force generation flag is set and Generated N facts from ml-service-facts, and the guided/facts request in ml-service-summarisation |
| The same, with M = 0 | No transcripts came back, so the problem is in speech recognition. Other messages, such as the fact sent at the start of the session, can still arrive, so N can be more than 0 | The Speech to Text – Streams check will usually be failing too. Use the Speech to Text runbook |
the platform sent a fact without a group or text |
ml-service-facts answered, but a fact was missing its group or text | ml-service-facts logs for the session |
reading facts: |
The facts arrived in a shape the check does not recognise | copilot-audio-bridge and ml-service-facts logs for the session |
asking for facts: |
The connection broke before the flush message could be sent |
Trace the session |
the platform reported an error while awaiting facts: followed by details |
The platform sent an error message. The details say what | Trace the session |
awaiting facts: the platform closed the connection |
copilot-audio-bridge dropped the connection | Trace the session |
the session ended before it transcribed anything |
The session ended early, for example because the tenant ran out of balance | Trace the session |
extract-facts
Sends the consultation as text (POST /v2/tools/extract-facts, with
outputLanguage: en) and requires at least one fact back, each with a group
and text. The platform extracts facts without storing them.
Path: Traefik → authorization-service → copilot-api-v2 (in the tenant) →
ml-service-summarisation (in shared), which extracts the facts.
To find the request: in Synapse Explore (Loki), run the query below for a
minute around the failure, then search for its request_id.
ml-service-summarisation logs the same ID followed by -segment-0.
{k8s_cluster_name="<cluster>", k8s_namespace_name="<tenant>", service_name="copilot-api-v2"} |= "/tools/extract-facts" |= "Go-http-client"
| Error text contains | What it means | Check first |
|---|---|---|
extracting facts: the platform reported an error: answered 401 or 403 |
authorization-service refused the access token | authorization-service logs |
… answered 500 |
copilot-api-v2 failed. Its log for the request says which part: Failed to fetch fact groups for extraction (the fact groups could not be loaded) or Failed to extract facts via ML service (ml-service-summarisation failed) |
copilot-api-v2 logs for the request, then ml-service-summarisation's |
… answered any other status, followed by a reason |
copilot-api-v2 refused the request. The reason says why | copilot-api-v2 logs for the request |
extracting facts: calling the platform: … Client.Timeout exceeded |
No answer within the check's 30 seconds. copilot-api-v2 itself waits up to 120 seconds, so a slow extraction shows here rather than as an error status | copilot-api-v2 logs for the request: if Maestro: balance check passed comes long after the request started, the time went on the billing balance check. Otherwise ml-service-summarisation logs for the request: Processing fact request, then POST /v1/internal/summarisation/facts with its status and duration |
extracting facts: calling the platform: and anything else |
The request got no answer | Traefik |
the platform extracted no facts |
The extraction succeeded but found nothing in the consultation. The text never changes, so something in the extraction changed | ml-service-summarisation logs for the request |
the platform sent a fact without a group or text |
ml-service-summarisation answered, but a fact was missing its group or text | ml-service-summarisation logs for the request |
the facts could not be read |
The answer was not in the shape the check expects | copilot-api-v2 logs for the request |
generate-document
Sends the consultation as text (POST /v2/documents/, with the standard
English SOAP template, outputLanguage: en, and
X-Corti-Retention-Policy: none), and requires a document back with every
section its template lists and words in at least one of them. Which sections
are written depends on the consultation, so no section in particular is
required.
Path: Traefik → authorization-service → copilot-api-v2 (in the tenant) →
ml-service-summarisation (in shared), which writes the document.
The template is synced every 15 minutes from templates/corti-soap_en.json in
corticph/standard-templates.
Its ID is derived from that file's name, so it is the same in every cluster.
To find the request: in Synapse Explore (Loki), run the query below for a
minute around the failure, then search for its request_id.
ml-service-summarisation logs the same ID.
{k8s_cluster_name="<cluster>", k8s_namespace_name="<tenant>", service_name="copilot-api-v2"} |= "/documents/" |= "Go-http-client"
copilot-api-v2 logs every failed request as alpha structured document failed, with the underlying error and the status it answered.
| Error text contains | What it means | Check first |
|---|---|---|
generating the document: the platform reported an error: answered 401 |
authorization-service refused the access token | authorization-service logs |
… answered 403 |
authorization-service or copilot-api-v2 refused access | authorization-service logs, then copilot-api-v2's for the request |
… answered 404 Not Found with nothing after it |
Guided documents are switched off for the tenant (ALPHA_STRUCTURED_DOCS_ENABLED is not "true") |
The tenant's copilot-api-v2 values in deployments |
… answered 404 Not Found: template f901c06a-70db-59f6-8d0a-0a4bef4b8c77 lookup failed |
The standard English SOAP template is gone: it was removed or renamed in corticph/standard-templates | The history of templates/corti-soap_en.json in corticph/standard-templates |
… answered 422 followed by a reason |
The template could not be used (for example, it has no published version), or ml-service-summarisation's answer could not be read. The reason says which | copilot-api-v2 logs for the request |
… answered 429 Too Many Requests: rate limit exceeded, please try again later |
ml-service-summarisation refused the request as rate limited | ml-service-summarisation logs |
… answered 500 Internal Server Error: internal error |
copilot-api-v2 failed. If ml-service-summarisation answered with an error, copilot-api-v2 also logs guided summarization endpoint returned non 200 with its ml_service_status_code |
copilot-api-v2 logs for the request, then ml-service-summarisation's |
… answered any other status, followed by a reason |
copilot-api-v2 refused the request. The reason says why | copilot-api-v2 logs for the request |
generating the document: calling the platform: … Client.Timeout exceeded |
No answer within the check's 30 seconds. A document normally takes about 4 seconds. copilot-api-v2 itself waits up to 120 seconds, so a slow document shows here rather than as an error status | copilot-api-v2 logs for the request: if Maestro: balance check passed comes long after the request started, the time went on the billing balance check. Otherwise ml-service-summarisation logs for the request: Processing guided document request., then POST /v1/internal/summarisation/guided/document with its status and duration |
generating the document: calling the platform: and anything else |
The request got no answer | Traefik |
the platform kept the document despite being told not to |
The platform stored the document although the check asked it not to: it answered 201 Created or named the stored document. The check deletes it when the answer names it; if that fails too, a teardown failure is reported as well |
copilot-api-v2 logs for the request: Failed to resolve retention policy, default to retaining |
the document was written from another template followed by a template ID |
The document came back from a template other than the standard English SOAP template the check asked for | copilot-api-v2 logs for the request |
the document came back without sections |
The document listed no sections | copilot-api-v2 logs for the request |
the document is missing a section its template defines: followed by a heading |
A section the template lists was missing from the document | ml-service-summarisation logs for the request |
the document came back with nothing written in it |
Every section was empty. The consultation never changes, so something in writing the document changed | ml-service-summarisation logs for the request |
the document could not be read |
The answer was not in the shape the check expects | copilot-api-v2 logs for the request |
list-templates, open-template
list-templates lists Corti's standard templates
(GET /v2/documents/templates/?source=corti) and requires the standard English
SOAP template among them. open-template then opens that template
(GET /v2/documents/templates/f901c06a-70db-59f6-8d0a-0a4bef4b8c77) and requires
a published version with at least one section, each with a name. Templates
counts as working only when both succeed.
Path: Traefik → authorization-service → copilot-api-v2 (in the tenant), which
reads the templates from its own database. No other service is involved.
authorization-service lets the requests through only when the token carries
guided_templates_sections:read, which every client gets by default.
The standard templates are synced every 15 minutes from templates/ in
corticph/standard-templates;
the SOAP template is templates/corti-soap_en.json.
To find the requests: in Synapse Explore (Loki), run the query below for a
minute around the failure. Listing and opening are separate requests, each with
its own request_id.
{k8s_cluster_name="<cluster>", k8s_namespace_name="<tenant>", service_name="copilot-api-v2"} |= "/documents/templates/" |= "Go-http-client"
copilot-api-v2 logs each request with its status, but not why a template request failed: the reason is in the error text.
| Error text contains | What it means | Check first |
|---|---|---|
listing the templates: the platform reported an error: answered 401 |
authorization-service rejected the access token itself | authorization-service logs: it logs a warning naming why, such as Invalid JWT or JWT issuer does not match tenant |
… answered 403 |
The token carries no scope allowing the request. For these endpoints that is guided_templates_sections:read, which every client gets by default. authorization-service logs the refusal only at debug level, so it does not appear in production logs |
The guided_templates_sections:read entry in authorization-service's values in deployments |
… answered 404 Not Found with nothing after it, at list-templates |
Guided templates are switched off for the tenant (ALPHA_TEMPLATES_ENABLED is not "true") |
The tenant's copilot-api-v2 values in deployments |
the platform listed no standard templates |
The list came back empty: no standard templates in the tenant | Whether the standard templates sync is running in copilot-api-v2 |
a standard template is missing from the list: "f901c06a-70db-59f6-8d0a-0a4bef4b8c77" |
Other standard templates are listed, but not the SOAP template: it was removed or renamed in corticph/standard-templates | The history of templates/corti-soap_en.json in corticph/standard-templates |
opening the template: the platform reported an error: answered 404 Not Found: record not found |
The list named the SOAP template, but opening it found nothing: it was deleted between the two requests | The history of templates/corti-soap_en.json in corticph/standard-templates |
… answered any other status, followed by a reason |
copilot-api-v2 refused or failed the request. The reason is copilot-api-v2's own error | copilot-api-v2 logs for the request |
calling the platform: … Client.Timeout exceeded |
No answer within the check's 15 seconds. A healthy request takes under a second | copilot-api-v2 logs for the request, then its database |
calling the platform: and anything else |
The request got no answer | Traefik |
the template has no published version |
The SOAP template exists but has nothing published to read | The history of templates/corti-soap_en.json in corticph/standard-templates |
the template has no sections or the template has a section without a name |
The SOAP template was published without the sections a customer reads it for | The history of templates/corti-soap_en.json in corticph/standard-templates |
the templates could not be read or the template could not be read |
The answer was not in the shape the check expects | copilot-api-v2 logs for the request |
Example: a check that passed (dev-weu, 2026-09-30)
A Text Generation – Streams check in dev-weu base, tracing its interactionId:
| Time (UTC) | Service | What it logged | What that means |
|---|---|---|---|
| 17:33:06 | copilot-audio-bridge | Interaction: received streams request, then configuration validated |
The check connected and its configuration was accepted |
| 17:33:07 | ml-service-facts | Generated 1 facts with sequence id 0 |
The fact sent at the start of every session, before any audio. The check ignores it |
| 17:33:11 | copilot-audio-bridge | Flush sent |
The check sent flush after the audio |
| 17:33:13 | ml-service-facts | Force generation flag is set and unprocessed utterances exist in state, generating facts. then Calling the summarisation model |
The flush reached ml-service-facts |
| 17:33:15 | ml-service-summarisation | POST /v1/internal/summarisation/guided/facts 200 2031.36ms |
Extracting the facts took 2 seconds |
| 17:33:15 | ml-service-facts | Generated 6 facts with sequence id 1 |
|
| 17:33:15 | copilot-audio-bridge | V2 Facts: sent fact message to client |
The facts reached the check, 3.5 seconds after the flush |
When a check fails at receive-facts, the same trace shows how far the facts
got.
Example: a FactsR request that passed (dev-weu, 2026-10-05)
A FactsR request in dev-weu base, tracing its request_id:
| Time (UTC) | Service | What it logged | What that means |
|---|---|---|---|
| 12:01:44.95 | copilot-api-v2 | Maestro: balance check passed |
The tenant has balance to spend |
| 12:01:44.97 | ml-service-summarisation | Processing fact request (input_length: 398, language: en) |
copilot-api-v2 passed the consultation on for extraction |
| 12:01:46.59 | ml-service-summarisation | request: POST /v1/internal/summarisation/facts 200 1620.30ms |
Extracting the facts took 1.6 seconds |
| 12:01:47.01 | copilot-api-v2 | Request for POST /tools/extract-facts with status 200 |
The facts reached the caller, 2.1 seconds after the request arrived |
Example: a Documents request that passed (dev-weu, 2026-10-05)
A Documents request in dev-weu base, tracing its request_id:
| Time (UTC) | Service | What it logged | What that means |
|---|---|---|---|
| 15:43:54.14 | copilot-api-v2 | Maestro: balance check passed |
The tenant has balance to spend |
| 15:43:54.19 | ml-service-summarisation | Processing guided document request. (input_length: 398, number_of_sections: 4) |
copilot-api-v2 passed the consultation on with the template's four sections |
| 15:43:57.61 | ml-service-summarisation | request: POST /v1/internal/summarisation/guided/document 200 3422.82ms |
Writing the document took 3.4 seconds |
| 15:43:58.00 | copilot-api-v2 | Request for POST /documents/ with status 200 |
The document reached the caller, 3.9 seconds after the request arrived |
Example: a Templates check that passed (prod-weu, 2026-10-09)
A Templates check in prod-weu base:
| Time (UTC) | Service | What it logged | What that means |
|---|---|---|---|
| 13:59:13.81 | copilot-api-v2 | Request for GET /documents/templates/?source=corti with status 200, 0.23s |
The standard templates were listed |
| 13:59:14.05 | copilot-api-v2 | Request for GET /documents/templates/f901c06a-70db-59f6-8d0a-0a4bef4b8c77 with status 200, 0.23s |
The SOAP template was opened |
Who owns a service
The owning team is the team: value in the service's configuration in the
deployments repository: config/services/<service>/base/values.yaml.