Orca ORCA v0.174.2
Welcome Developer Guide Infrastructure Platform Engineering Runbooks Archive
Documentation navigation
On Premise Customers
SFD Hamilton County CommonSpirit GMR Region Nord On-Prem Synapse API-Key Deploy
Dns
DNS Flush
Kafka
Kafka Consumer Lag in Voice-Vault Kafka Consumer Group Lag High (> 800) HPA Max Replicas Exceed Kafka Topic Partitions
Storage
Ceph OSD Full / BlueFS Allocation Wedge Query Ceph RGW (S3) Bucket Usage Ceph Mon Quorum Loss / Node-Replacement Recovery PVC Near Full Warning (>= 80%) PVC Near Full Critical (>= 90%) PVC Disk Usage Forecast > 95% Ceph Cluster Storage Utilization High (> 90%) Ceph Cluster Error Ceph Cluster Warning
Observability
Alloy Operator Wedge Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild Grafana Dashboard Provisioning Error Mimir Ingester Reaching Series Limit Mimir Ingest Storage Wedged Mimir Ingester PVC Near Full Loki PVC Near Full Mimir Self-Monitoring Scrape Stalled Synapse Cluster Metrics Silent Mimir Discarding Samples (Series Limit) Loki Discarding Log Entries Trivy Metrics Stale Envoy Proxy Heap Near Ceiling Envoy Proxy Heap Shedding Envoy Proxy Overload Rejecting Requests Traefik Elevated Error Spike (4xx/5xx) Envoy Elevated Error Spike (4xx/5xx) Scrape Targets Failing Traefik 402 Payment Failure (Multiple Distinct Users) Azure Audit Ingestion Silent
Databases
CNPG Cluster Has No Backup CNPG Replication Unhealthy CNPG Cluster Backup Stale (> 48h) CNPG Cluster Has No WAL Recovery Point
Kubernetes
Pod ImagePull / ImagePullBackOff Pod CrashLoopBackOff Deployment Has No Running Replicas Deployment Rollout Stuck DaemonSet Has No Running Pods StatefulSet Has No Running Replicas Kubernetes Job Failed Kubernetes CronJob Suspended Certificate Expiring Soon Node Memory/Disk Pressure (Kubelet Evicting Pods) Container OOMKilled Cluster Autoscaler Unhealthy Trivy Critical Vulnerability One Week Trivy Critical Vulnerability Four Weeks Trivy High Vulnerability One Week Trivy High Vulnerability Six Weeks Certificate Renewal Stuck Certificate Never Issued Trivy Critical Vulnerability Immediate API v2 Debug Endpoint Called in Production Cluster Autoscaler Unregistered Nodes Platform API Server Errors Platform API Client Errors Pod Pending Over 30 Minutes Pod Pending (GPU) Over 60 Minutes Cilium Policy Drops Burst
Capacity
CPU Throttling > 20% HPA Utilization Critically High GPUs Underutilized For 48h Container Memory Leak (Rising Near Limit) Nodegroup Utilization Critically High Nodegroup Utilization High Warning
Identity
Keycloak Client Misconfigured
Synthetic Monitoring
Synthetic Feature Failing Synthetic Monitor Silent Speech to Text – Synthetic Check Failing Corti Models – Synthetic Check Failing Text Generation – Synthetic Check Failing
Platform
Orca GitHub Rate-Limit Budget Low Email Service Minute Quota High Kyverno Admission API Server Unreachable Email Service Hourly Quota High Email Service Send Throttled Email Service Bounce Rate High

Synapse Cluster Metrics Silent

Runbook for a whole synapse-client cluster that has stopped emitting metrics. This is a dead-man's switch — it fires when a cluster that was recently sending metrics goes completely silent. The MimirSelfMonitoringScrapeStalled alert (08-mimir-self-monitoring-scrape-stalled.md) only covers clusters that run Mimir ingesters (synapse-server clusters); most clusters are client-only (dev-weu, the prod fleet) and remote_write in from afar. This alert covers them.

Meaning

The central synapse-server Mimir ingests every cluster's remote_write. If a cluster's collectors die, its remote_write breaks, or the cluster goes away, all of its series stop at once.

The alert uses the kubelet integration (job="integrations/kubernetes/kubelet") as the heartbeat — present in every synapse-client cluster. "Recently seen" = any kubelet sample in the last 6h (count_over_time(up{job="integrations/kubernetes/kubelet"}[6h])); "silent now" = no kubelet up samples right now. The unless fires for any cluster that was seen in the last 6h but is sending nothing now.

This is self-maintaining and decommission-safe: a genuinely-removed cluster clears itself after 6h of silence, so there is no hardcoded cluster list to keep in sync across dev + the prod fleet. for: 10m rides out a transient scrape gap; cluster stays in the grouping because the central Mimir ingests every cluster.

The metrics are already in the synapse-client allow-list — no chart PR is needed.

What changed from the chart

  • All environments — the chart alert had no environment filter. Kept as-is.
  • Priority 1 — a whole cluster's observability is gone; every alert scoped to that cluster is blind.

Impact

  • Every dashboard and alert scoped to the affected cluster is blind — no metrics means no error-rate, no latency, no pod-health, no capacity alerts fire for that cluster.
  • The cluster itself may be fine (serving traffic) — this is a monitoring outage, not a service outage — or it may be genuinely down. You cannot tell from metrics alone.
  • If remote_write broke (not the cluster), the cluster's collectors are still scraping and buffering locally; once remote_write recovers, the backlog may or may not be replayed depending on the collector's WAL retention.

Diagnosis

The alert label carries k8s_cluster_name. From the alert, note it.

1. Confirm the silence

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prod-weu.yaml

Confirm via Mimir (port-forward synapse-server-mimir-gateway, header X-Scope-OrgID: corti):

# Should return 0 if silent, > 0 if healthy
QUERY='count by (k8s_cluster_name) (up{job="integrations/kubernetes/kubelet", k8s_cluster_name="<cluster>"})'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[]? | "\(.metric.cluster): \(.value[1]) targets"'

# Confirm it was recently seen (within 6h)
QUERY2='count by (k8s_cluster_name) (count_over_time(up{job="integrations/kubernetes/kubelet", k8s_cluster_name="<cluster>"}[6h]))'

2. Check the synapse-client alloy pods on the affected cluster

kubectl -n monitoring-system get pods -l app.kubernetes.io/name=alloy-metrics -o wide
kubectl -n monitoring-system get pods -l app.kubernetes.io/name=alloy-singleton -o wide
kubectl -n monitoring-system get pods -o jsonpath='{range .items[?(@.metadata.labels.app\.kubernetes\.io/name=~"alloy.*")]}{.metadata.name} restarts={.status.containerStatuses[0].restartCount} phase={.status.phase}{"\n"}{end}'

If the pods are Pending, CrashLoopBackOff, or OOMKilled, the collectors are down — restart them or fix the underlying issue (see 07-kubernetes/02-pod-crashloop.md, 07-kubernetes/11-container-oomkilled.md).

3. Check remote_write egress

If the alloy pods are Running but metrics are not arriving at the central Mimir, the remote_write path is broken. For cross-cluster remote_write (remoteWrite.enabled: true), check:

  • The Envoy Gateway on the synapse-server cluster is reachable (the gatewayUrl in synapse-client values).
  • The projected ServiceAccount token is valid (not expired).
  • The NetworkPolicy egress allows HTTPS to the gateway (on Scaleway/kommodity clusters, the toFQDNs CiliumNetworkPolicy; on AKS, the external-443 rule).
# Check the alloy-metrics logs for remote_write errors
kubectl -n monitoring-system logs -l app.kubernetes.io/name=alloy-metrics --tail=50 | grep -iE 'remote_write|error|refused|timeout'

4. Is the cluster itself down?

If you cannot reach the cluster's API server (kubectl hangs), the cluster may be genuinely down or unreachable. Check the cloud provider (Azure / Scaleway) for the cluster/node status. This is a service outage, not just a monitoring outage.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Restart the collectors

If the alloy pods are CrashLoopBackOff or OOMKilled (Diagnosis step 2), restart them:

kubectl -n monitoring-system delete pod -l app.kubernetes.io/name=alloy-metrics
kubectl -n monitoring-system delete pod -l app.kubernetes.io/name=alloy-singleton

If they are OOMKilled, the root fix is synapse-client >= 0.9.2 (3Gi + GOMEMLIMIT) — see charts/charts/synapse-client/values.yaml and 08-mimir-self-monitoring-scrape-stalled.md step 3.

Step 3 — Fix the remote_write path

If the pods are Running but remote_write is broken (Diagnosis step 3), fix the egress path:

  • Verify the gatewayUrl in the synapse-client values resolves and is reachable.
  • On Scaleway/kommodity clusters, check the toFQDNs CiliumNetworkPolicy covers the gateway FQDN.
  • If the ServiceAccount token expired, the projected token refreshes automatically (1h lifetime) — a persistent auth failure means the tokenAudience or the synapse-server SecurityPolicy is misconfigured.

Step 4 — Cluster is down

If the cluster's API server is unreachable (Diagnosis step 4), this is a service outage — escalate to the platform oncall and the cloud provider. The metrics-silence is a symptom, not the problem.

Verify

  • The alert clears once count by (k8s_cluster_name) (up{job="integrations/kubernetes/kubelet", k8s_cluster_name="<cluster>"}) returns > 0, sustained for the for: 10m window.
  • Dashboards scoped to the cluster repopulate.
  • Other cluster-scoped alerts start evaluating again (they were blind during the silence).

Notes

  • Dead-man's switch — fires on absence, not a bad value. The 6h count_over_time window makes it self-maintaining (decommissioned clusters clear themselves) and decommission-safe (no hardcoded cluster list).
  • Relationship to scrape-stalled — 08-mimir-self-monitoring-scrape-stalled.md covers synapse-server clusters (the ServiceMonitor discovery wedge). This alert covers client-only clusters (remote_write silence). They are complementary.
  • Root fix for OOM — synapse-client >= 0.9.2 (3Gi + GOMEMLIMIT). Bump affected clusters to prevent recurrence.