Synthetic Feature Failing

A customer-facing feature has kept failing for one tenant in one cluster. Each product has its own runbook, covering that product's journeys step by step and what to check at each step. Open the one for the product the alert names:

• Speech to Text
• Corti Models
• Text Generation

Meaning

synthetic-testing-service checks the platform the way a customer would. For each feature it covers, it runs that feature's customer journey every 2 minutes, for each tenant, through the cluster's public API, and records whether the feature worked. A product every tenant reaches through one shared service, such as Corti Models, is checked once per cluster instead, logged in as one tenant. The product's runbook describes each journey.

For example, the Speech to Text – Transcribe check logs in, opens a real-time dictation session, streams a short pre-recorded audio clip, waits for the transcript to come back, and then ends the session. If any of those steps fails, the check fails.

This alert fires when a feature has been failing for at least half of the last 10 minutes for one tenant, and is still failing a minute later. Each check runs every 2 minutes, so that is roughly 3 of its last 5 checks. A single failed check never fires it. The alert names the product, the feature, the tenant and the cluster. For Corti Models it also names each model that failed in those 10 minutes.

Impact

• Production: customers of that tenant most likely cannot use that feature right now. Treat it as an outage of that feature. For a product checked once per cluster, such as Corti Models, the tenant is only the one the check logged in as, and customers of every tenant in the cluster are most likely affected: the product's runbook says which failures are exceptions.
• Development: informational. It is visible but pages nobody.

Diagnosis

1. See where the journey stopped. Open Where the journey stopped in the alert. It shows the step each failed check of this feature stopped at, labelled with the probe that ran it. Steps after it never ran.

2. See why. Open Why it failed in the alert. Each failed check writes one line naming the step it stopped at and the reason. The link shows every failed check for the tenant, so read the lines whose "probe" matches the one from step 1. For example, this line from a Speech to Text – Streams check ("probe":"speech-to-text-streams"):

   step "receive-transcript": awaiting transcript … after 0 messages including 0 transcripts: context deadline exceeded

   means the audio was streamed, but no transcript and no other message came back before the check's 1-minute run limit ran out. The product's runbook explains what a failure at each step means.

3. Check what sits behind that step, using the product's runbook:

   • Speech to Text
   • Corti Models
   • Text Generation

4. Platform or synthetic-testing-service?

   • If Synthetic Monitor Silent is firing for the same cluster, synthetic-testing-service itself is unhealthy: see Synthetic Monitor Silent.
   • If it stopped at acquire-token for one tenant only, synthetic-testing-service's own login for that tenant may be broken, and customers may be fine.
   • If the same feature is failing for every tenant in the cluster, the fault is shared.
   • A product checked once per cluster, such as Corti Models, has only one tenant to compare: use its runbook instead.

Mitigation

Depends on the step that failed: follow the product's runbook, and involve the team that owns the service behind that step.

Verify

The alert clears by itself once the feature has been failing for less than half of the last 10 minutes. It stays firing for 5 minutes after that, so a shaky recovery does not page twice.