synthetic-testing-service has stopped producing results in a cluster, or one of its checks has stopped running. While it is silent, Synthetic Feature Failing cannot fire for what it no longer checks, so an outage there would go unnoticed by synthetic monitoring. This alert is not a customer-facing outage in itself, which is why it notifies during working hours only.

Meaning

synthetic-testing-service runs in the shared namespace of each cluster it is deployed to. Every 2 minutes, each check records when it last completed a run (synthetic_probe_last_run_timestamp_seconds), whether the check passed or failed.

The alert fires in one of two cases, once it has lasted 10 minutes:

No results from the cluster at all. Kubernetes still has the synthetic-testing-service Deployment in the cluster, but none of its results reach Synapse. The alert names only the cluster.
One check has stopped running. Results still arrive, but one check has not completed a run for more than 5 minutes. The alert names the check in its probe label (for example speech-to-text-streams) and the tenant.

Removing a tenant from synthetic-testing-service, or removing synthetic-testing-service from a cluster, does not fire this alert: the results disappear rather than go stale, and the Deployment is gone with them.

Impact

Synthetic monitoring is not checking that cluster (or that check and tenant).
Customers are not affected by this alert itself. An outage during the same period would not be caught by Synthetic Feature Failing.

Diagnosis

Production clusters need Azure Privileged Identity Management first: activate the MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged) roles.

export KUBECONFIG=~/.kube/<cluster>.yaml

Is synthetic-testing-service running?
```
kubectl -n shared get deployment synthetic-testing-service
kubectl -n shared get pods -l app=synthetic-testing-service
```
- READY 0/0: it has been scaled down. Check the deployments repository history for environments/<environment>/<cluster>/shared/apps/synthetic-testing-service/. If it was deliberate, for example the dev-weu alerting test, there is nothing to fix.
- READY 0/1, or the pod restarting: go to step 3.
Is Synapse collecting its metrics? If Synapse Cluster Metrics Silent is also firing for this cluster, Synapse is not collecting from the cluster at all: follow Synapse Cluster Metrics Silent. Otherwise, check the pod carries the annotations Synapse collects by:
```
kubectl -n shared get pods -l app=synthetic-testing-service -o jsonpath='{.items[0].metadata.annotations}'
```
It needs k8s.grafana.com/scrape: "true" and k8s.grafana.com/metrics.portNumber: "9090".
What does it log?
```
kubectl -n shared logs deployment/synthetic-testing-service
```
If the configuration is invalid, the service logs invalid configuration with an error naming the setting, then exits. Add --previous if the pod has already restarted. When the alert names a check, search its lines in Synapse (https://synapse.corti.app, Explore, Loki data source):
```
{k8s_cluster_name="<cluster>", k8s_namespace_name="shared", service_name="synthetic-testing-service"} |= "<probe>"
```

Mitigation

Scaled down by mistake: restore the replica count through the deployments repository.
Pod not starting: fix what the logs name. For invalid configuration, that is the named setting in environments/<environment>/<cluster>/shared/apps/synthetic-testing-service/values.yaml.
Metrics not collected: restore the annotations from step 2. They come from the service's deployment configuration (metrics.enabled, and service.targetPort for the port, in config/services/synthetic-testing-service/base/values.yaml).
One check stopped while the others run: each check has a 1-minute run limit, so a check that stops completing runs is a defect in synthetic-testing-service. Restart it (needs PIM in production):
```
kubectl -n shared rollout restart deployment/synthetic-testing-service
```
and raise it with the platform team, who own the service.

Verify

The alert clears once results arrive again. Every check's last run should be under 3 minutes old:

time() - synthetic_probe_last_run_timestamp_seconds{k8s_cluster_name="<cluster>"}