Orca ORCA v0.174.2
Documentation navigation

Text Generation: synthetic check failing

Use this runbook when a Synthetic Feature Failing alert names Text Generation. The alert also names the feature, the tenant and the cluster.

How to use this runbook

  1. Open Where the journey stopped in the alert. It shows which step failed.
  2. Open Why it failed in the alert. It shows the error text, and for Streams the interactionId from create-interaction onwards. FactsR, Documents and Templates store nothing, so they have no ID: find the request as described under extract-facts, generate-document or list-templates, open-template.
  3. Find the step below and match the error text.

Cluster access, and tracing a session by its interactionId, work as described under Cluster access and Trace the session in the Speech to Text runbook.

The journeys

Feature What the check does Steps, in order
Streams Real-time facts on the /streams web socket, for an Interaction it creates: streams a short pre-recorded English clip with facts enabled, sends flush, and waits for facts acquire-token → create-interaction → open-session → receive-facts → end-session
FactsR Fact extraction from supplied text: sends the same consultation as text, and requires facts back. It stores nothing acquire-token → extract-facts
Documents Writes up the same consultation as a SOAP note with Corti's standard English SOAP template, and requires a written document back. It asks the platform not to store the document, and deletes it if the platform stores it anyway acquire-token → generate-document
Templates Lists Corti's standard templates and requires the standard English SOAP template among them, then opens that template and requires a published version with at least one section, each with a name. It only reads acquire-token → list-templates → open-template

A check stops at the first step that fails, so later steps never run.

acquire-token, create-interaction, end-session

The same steps as Speech to Text, with the same errors (FactsR, Documents and Templates use only acquire-token): acquire-token, create-interaction, end-session.

open-session

The same step as Speech to Text – Streams, except the configuration asks for facts: mode: facts and outputLocale: en.

receive-facts

Sends the audio packets of the clip, then sends the flush message, and waits for facts that arrive after the flush. Each fact must have a group and text.

The audio goes through the same speech recognition services as Speech to Text. The transcript then reaches ml-service-facts (in the tenant), which extracts facts by calling ml-service-summarisation (in shared). The facts return to copilot-audio-bridge, which sends them to the check. Without flush, ml-service-facts only extracts facts once a minute, which is longer than the check's 1-minute run limit allows.

When a session opens, ml-service-facts sends one fact straight away, before any audio has arrived (in dev-weu: Patient gender is unknown). The check ignores it.

Error text contains What it means Check first
awaiting facts … after N messages including M transcripts: context deadline exceeded, with M more than 0 Transcripts came back, so speech recognition works, but no facts arrived before the check's 1-minute run limit ran out Trace the session: find Force generation flag is set and Generated N facts from ml-service-facts, and the guided/facts request in ml-service-summarisation
The same, with M = 0 No transcripts came back, so the problem is in speech recognition. Other messages, such as the fact sent at the start of the session, can still arrive, so N can be more than 0 The Speech to Text – Streams check will usually be failing too. Use the Speech to Text runbook
the platform sent a fact without a group or text ml-service-facts answered, but a fact was missing its group or text ml-service-facts logs for the session
reading facts: The facts arrived in a shape the check does not recognise copilot-audio-bridge and ml-service-facts logs for the session
asking for facts: The connection broke before the flush message could be sent Trace the session
the platform reported an error while awaiting facts: followed by details The platform sent an error message. The details say what Trace the session
awaiting facts: the platform closed the connection copilot-audio-bridge dropped the connection Trace the session
the session ended before it transcribed anything The session ended early, for example because the tenant ran out of balance Trace the session

extract-facts

Sends the consultation as text (POST /v2/tools/extract-facts, with outputLanguage: en) and requires at least one fact back, each with a group and text. The platform extracts facts without storing them.

Path: Traefik → authorization-service → copilot-api-v2 (in the tenant) → ml-service-summarisation (in shared), which extracts the facts.

To find the request: in Synapse Explore (Loki), run the query below for a minute around the failure, then search for its request_id. ml-service-summarisation logs the same ID followed by -segment-0.

{k8s_cluster_name="<cluster>", k8s_namespace_name="<tenant>", service_name="copilot-api-v2"} |= "/tools/extract-facts" |= "Go-http-client"
Error text contains What it means Check first
extracting facts: the platform reported an error: answered 401 or 403 authorization-service refused the access token authorization-service logs
… answered 500 copilot-api-v2 failed. Its log for the request says which part: Failed to fetch fact groups for extraction (the fact groups could not be loaded) or Failed to extract facts via ML service (ml-service-summarisation failed) copilot-api-v2 logs for the request, then ml-service-summarisation's
… answered any other status, followed by a reason copilot-api-v2 refused the request. The reason says why copilot-api-v2 logs for the request
extracting facts: calling the platform: … Client.Timeout exceeded No answer within the check's 30 seconds. copilot-api-v2 itself waits up to 120 seconds, so a slow extraction shows here rather than as an error status copilot-api-v2 logs for the request: if Maestro: balance check passed comes long after the request started, the time went on the billing balance check. Otherwise ml-service-summarisation logs for the request: Processing fact request, then POST /v1/internal/summarisation/facts with its status and duration
extracting facts: calling the platform: and anything else The request got no answer Traefik
the platform extracted no facts The extraction succeeded but found nothing in the consultation. The text never changes, so something in the extraction changed ml-service-summarisation logs for the request
the platform sent a fact without a group or text ml-service-summarisation answered, but a fact was missing its group or text ml-service-summarisation logs for the request
the facts could not be read The answer was not in the shape the check expects copilot-api-v2 logs for the request

generate-document

Sends the consultation as text (POST /v2/documents/, with the standard English SOAP template, outputLanguage: en, and X-Corti-Retention-Policy: none), and requires a document back with every section its template lists and words in at least one of them. Which sections are written depends on the consultation, so no section in particular is required.

Path: Traefik → authorization-service → copilot-api-v2 (in the tenant) → ml-service-summarisation (in shared), which writes the document.

The template is synced every 15 minutes from templates/corti-soap_en.json in corticph/standard-templates. Its ID is derived from that file's name, so it is the same in every cluster.

To find the request: in Synapse Explore (Loki), run the query below for a minute around the failure, then search for its request_id. ml-service-summarisation logs the same ID.

{k8s_cluster_name="<cluster>", k8s_namespace_name="<tenant>", service_name="copilot-api-v2"} |= "/documents/" |= "Go-http-client"

copilot-api-v2 logs every failed request as alpha structured document failed, with the underlying error and the status it answered.

Error text contains What it means Check first
generating the document: the platform reported an error: answered 401 authorization-service refused the access token authorization-service logs
… answered 403 authorization-service or copilot-api-v2 refused access authorization-service logs, then copilot-api-v2's for the request
… answered 404 Not Found with nothing after it Guided documents are switched off for the tenant (ALPHA_STRUCTURED_DOCS_ENABLED is not "true") The tenant's copilot-api-v2 values in deployments
… answered 404 Not Found: template f901c06a-70db-59f6-8d0a-0a4bef4b8c77 lookup failed The standard English SOAP template is gone: it was removed or renamed in corticph/standard-templates The history of templates/corti-soap_en.json in corticph/standard-templates
… answered 422 followed by a reason The template could not be used (for example, it has no published version), or ml-service-summarisation's answer could not be read. The reason says which copilot-api-v2 logs for the request
… answered 429 Too Many Requests: rate limit exceeded, please try again later ml-service-summarisation refused the request as rate limited ml-service-summarisation logs
… answered 500 Internal Server Error: internal error copilot-api-v2 failed. If ml-service-summarisation answered with an error, copilot-api-v2 also logs guided summarization endpoint returned non 200 with its ml_service_status_code copilot-api-v2 logs for the request, then ml-service-summarisation's
… answered any other status, followed by a reason copilot-api-v2 refused the request. The reason says why copilot-api-v2 logs for the request
generating the document: calling the platform: … Client.Timeout exceeded No answer within the check's 30 seconds. A document normally takes about 4 seconds. copilot-api-v2 itself waits up to 120 seconds, so a slow document shows here rather than as an error status copilot-api-v2 logs for the request: if Maestro: balance check passed comes long after the request started, the time went on the billing balance check. Otherwise ml-service-summarisation logs for the request: Processing guided document request., then POST /v1/internal/summarisation/guided/document with its status and duration
generating the document: calling the platform: and anything else The request got no answer Traefik
the platform kept the document despite being told not to The platform stored the document although the check asked it not to: it answered 201 Created or named the stored document. The check deletes it when the answer names it; if that fails too, a teardown failure is reported as well copilot-api-v2 logs for the request: Failed to resolve retention policy, default to retaining
the document was written from another template followed by a template ID The document came back from a template other than the standard English SOAP template the check asked for copilot-api-v2 logs for the request
the document came back without sections The document listed no sections copilot-api-v2 logs for the request
the document is missing a section its template defines: followed by a heading A section the template lists was missing from the document ml-service-summarisation logs for the request
the document came back with nothing written in it Every section was empty. The consultation never changes, so something in writing the document changed ml-service-summarisation logs for the request
the document could not be read The answer was not in the shape the check expects copilot-api-v2 logs for the request

list-templates, open-template

list-templates lists Corti's standard templates (GET /v2/documents/templates/?source=corti) and requires the standard English SOAP template among them. open-template then opens that template (GET /v2/documents/templates/f901c06a-70db-59f6-8d0a-0a4bef4b8c77) and requires a published version with at least one section, each with a name. Templates counts as working only when both succeed.

Path: Traefik → authorization-service → copilot-api-v2 (in the tenant), which reads the templates from its own database. No other service is involved. authorization-service lets the requests through only when the token carries guided_templates_sections:read, which every client gets by default.

The standard templates are synced every 15 minutes from templates/ in corticph/standard-templates; the SOAP template is templates/corti-soap_en.json.

To find the requests: in Synapse Explore (Loki), run the query below for a minute around the failure. Listing and opening are separate requests, each with its own request_id.

{k8s_cluster_name="<cluster>", k8s_namespace_name="<tenant>", service_name="copilot-api-v2"} |= "/documents/templates/" |= "Go-http-client"

copilot-api-v2 logs each request with its status, but not why a template request failed: the reason is in the error text.

Error text contains What it means Check first
listing the templates: the platform reported an error: answered 401 authorization-service rejected the access token itself authorization-service logs: it logs a warning naming why, such as Invalid JWT or JWT issuer does not match tenant
… answered 403 The token carries no scope allowing the request. For these endpoints that is guided_templates_sections:read, which every client gets by default. authorization-service logs the refusal only at debug level, so it does not appear in production logs The guided_templates_sections:read entry in authorization-service's values in deployments
… answered 404 Not Found with nothing after it, at list-templates Guided templates are switched off for the tenant (ALPHA_TEMPLATES_ENABLED is not "true") The tenant's copilot-api-v2 values in deployments
the platform listed no standard templates The list came back empty: no standard templates in the tenant Whether the standard templates sync is running in copilot-api-v2
a standard template is missing from the list: "f901c06a-70db-59f6-8d0a-0a4bef4b8c77" Other standard templates are listed, but not the SOAP template: it was removed or renamed in corticph/standard-templates The history of templates/corti-soap_en.json in corticph/standard-templates
opening the template: the platform reported an error: answered 404 Not Found: record not found The list named the SOAP template, but opening it found nothing: it was deleted between the two requests The history of templates/corti-soap_en.json in corticph/standard-templates
… answered any other status, followed by a reason copilot-api-v2 refused or failed the request. The reason is copilot-api-v2's own error copilot-api-v2 logs for the request
calling the platform: … Client.Timeout exceeded No answer within the check's 15 seconds. A healthy request takes under a second copilot-api-v2 logs for the request, then its database
calling the platform: and anything else The request got no answer Traefik
the template has no published version The SOAP template exists but has nothing published to read The history of templates/corti-soap_en.json in corticph/standard-templates
the template has no sections or the template has a section without a name The SOAP template was published without the sections a customer reads it for The history of templates/corti-soap_en.json in corticph/standard-templates
the templates could not be read or the template could not be read The answer was not in the shape the check expects copilot-api-v2 logs for the request

Example: a check that passed (dev-weu, 2026-09-30)

A Text Generation – Streams check in dev-weu base, tracing its interactionId:

Time (UTC) Service What it logged What that means
17:33:06 copilot-audio-bridge Interaction: received streams request, then configuration validated The check connected and its configuration was accepted
17:33:07 ml-service-facts Generated 1 facts with sequence id 0 The fact sent at the start of every session, before any audio. The check ignores it
17:33:11 copilot-audio-bridge Flush sent The check sent flush after the audio
17:33:13 ml-service-facts Force generation flag is set and unprocessed utterances exist in state, generating facts. then Calling the summarisation model The flush reached ml-service-facts
17:33:15 ml-service-summarisation POST /v1/internal/summarisation/guided/facts 200 2031.36ms Extracting the facts took 2 seconds
17:33:15 ml-service-facts Generated 6 facts with sequence id 1
17:33:15 copilot-audio-bridge V2 Facts: sent fact message to client The facts reached the check, 3.5 seconds after the flush

When a check fails at receive-facts, the same trace shows how far the facts got.

Example: a FactsR request that passed (dev-weu, 2026-10-05)

A FactsR request in dev-weu base, tracing its request_id:

Time (UTC) Service What it logged What that means
12:01:44.95 copilot-api-v2 Maestro: balance check passed The tenant has balance to spend
12:01:44.97 ml-service-summarisation Processing fact request (input_length: 398, language: en) copilot-api-v2 passed the consultation on for extraction
12:01:46.59 ml-service-summarisation request: POST /v1/internal/summarisation/facts 200 1620.30ms Extracting the facts took 1.6 seconds
12:01:47.01 copilot-api-v2 Request for POST /tools/extract-facts with status 200 The facts reached the caller, 2.1 seconds after the request arrived

Example: a Documents request that passed (dev-weu, 2026-10-05)

A Documents request in dev-weu base, tracing its request_id:

Time (UTC) Service What it logged What that means
15:43:54.14 copilot-api-v2 Maestro: balance check passed The tenant has balance to spend
15:43:54.19 ml-service-summarisation Processing guided document request. (input_length: 398, number_of_sections: 4) copilot-api-v2 passed the consultation on with the template's four sections
15:43:57.61 ml-service-summarisation request: POST /v1/internal/summarisation/guided/document 200 3422.82ms Writing the document took 3.4 seconds
15:43:58.00 copilot-api-v2 Request for POST /documents/ with status 200 The document reached the caller, 3.9 seconds after the request arrived

Example: a Templates check that passed (prod-weu, 2026-10-09)

A Templates check in prod-weu base:

Time (UTC) Service What it logged What that means
13:59:13.81 copilot-api-v2 Request for GET /documents/templates/?source=corti with status 200, 0.23s The standard templates were listed
13:59:14.05 copilot-api-v2 Request for GET /documents/templates/f901c06a-70db-59f6-8d0a-0a4bef4b8c77 with status 200, 0.23s The SOAP template was opened

Who owns a service

The owning team is the team: value in the service's configuration in the deployments repository: config/services/<service>/base/values.yaml.