Orca ORCA v0.174.2
Welcome Developer Guide Infrastructure Platform Engineering Runbooks Archive
Documentation navigation
On Premise Customers
SFD Hamilton County CommonSpirit GMR Region Nord On-Prem Synapse API-Key Deploy
Dns
DNS Flush
Kafka
Kafka Consumer Lag in Voice-Vault Kafka Consumer Group Lag High (> 800) HPA Max Replicas Exceed Kafka Topic Partitions
Storage
Ceph OSD Full / BlueFS Allocation Wedge Query Ceph RGW (S3) Bucket Usage Ceph Mon Quorum Loss / Node-Replacement Recovery PVC Near Full Warning (>= 80%) PVC Near Full Critical (>= 90%) PVC Disk Usage Forecast > 95% Ceph Cluster Storage Utilization High (> 90%) Ceph Cluster Error Ceph Cluster Warning
Observability
Alloy Operator Wedge Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild Grafana Dashboard Provisioning Error Mimir Ingester Reaching Series Limit Mimir Ingest Storage Wedged Mimir Ingester PVC Near Full Loki PVC Near Full Mimir Self-Monitoring Scrape Stalled Synapse Cluster Metrics Silent Mimir Discarding Samples (Series Limit) Loki Discarding Log Entries Trivy Metrics Stale Envoy Proxy Heap Near Ceiling Envoy Proxy Heap Shedding Envoy Proxy Overload Rejecting Requests Traefik Elevated Error Spike (4xx/5xx) Envoy Elevated Error Spike (4xx/5xx) Scrape Targets Failing Traefik 402 Payment Failure (Multiple Distinct Users) Azure Audit Ingestion Silent
Databases
CNPG Cluster Has No Backup CNPG Replication Unhealthy CNPG Cluster Backup Stale (> 48h) CNPG Cluster Has No WAL Recovery Point
Kubernetes
Pod ImagePull / ImagePullBackOff Pod CrashLoopBackOff Deployment Has No Running Replicas Deployment Rollout Stuck DaemonSet Has No Running Pods StatefulSet Has No Running Replicas Kubernetes Job Failed Kubernetes CronJob Suspended Certificate Expiring Soon Node Memory/Disk Pressure (Kubelet Evicting Pods) Container OOMKilled Cluster Autoscaler Unhealthy Trivy Critical Vulnerability One Week Trivy Critical Vulnerability Four Weeks Trivy High Vulnerability One Week Trivy High Vulnerability Six Weeks Certificate Renewal Stuck Certificate Never Issued Trivy Critical Vulnerability Immediate API v2 Debug Endpoint Called in Production Cluster Autoscaler Unregistered Nodes Platform API Server Errors Platform API Client Errors Pod Pending Over 30 Minutes Pod Pending (GPU) Over 60 Minutes Cilium Policy Drops Burst
Capacity
CPU Throttling > 20% HPA Utilization Critically High GPUs Underutilized For 48h Container Memory Leak (Rising Near Limit) Nodegroup Utilization Critically High Nodegroup Utilization High Warning
Identity
Keycloak Client Misconfigured
Synthetic Monitoring
Synthetic Feature Failing Synthetic Monitor Silent Speech to Text – Synthetic Check Failing Corti Models – Synthetic Check Failing Text Generation – Synthetic Check Failing
Platform
Orca GitHub Rate-Limit Budget Low Email Service Minute Quota High Kyverno Admission API Server Unreachable Email Service Hourly Quota High Email Service Send Throttled Email Service Bounce Rate High

Mimir Self-Monitoring Scrape Stalled

Runbook for the alloy-metrics ServiceMonitor-discovery wedge. This is an absence alert — it fires when metrics that should be there are not. A readiness/liveness probe cannot catch this (the alloy components stay healthy), so an absence alert is the only reliable detector.

Meaning

After an OOM restart, alloy-metrics' operator-based scraping can silently stop: all its components report healthy, but every monitoring-system/* ServiceMonitor target vanishes from up. Mimir self-monitoring metrics (cortex_ingester_active_series, cortex_ingester_memory_series, the ingester series-limit alerts) go blind while the built-in kubernetes integrations keep working.

The alert anchors on kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~".*mimir-ingester.*"} — shipped via the kubelet integration (job="integrations/kubernetes/kubelet"), which survives the wedge — and only exists on clusters that actually run Mimir ingesters. So the unless fires only when a cluster has ingester PVCs (via the kubelet path) but no monitoring-system/* targets in up (the wedged ServiceMonitor path). This scopes to synapse-server clusters and never false-fires on app-only clusters (which have no monitoring-system self-scrape by design).

The unless on by (k8s_cluster_name) yields one result per affected cluster; the central Mimir ingests every cluster, so cluster must stay in the grouping. for: 15m rides out a transient scrape gap.

The metrics are already in the synapse-client allow-list — no chart PR is needed.

What changed from the chart

  • All environments — the chart alert had no environment filter. Kept as-is.
  • Priority 2 — monitoring is blind but the cluster itself is not down; the ingester is still serving queries (you just can't see it).

Impact

  • All Mimir self-monitoring metrics for the affected cluster are missing: cortex_ingester_*, cortex_distributor_*, cortex_querier_* — the series-limit alerts (04-mimir-ingester-series-limit.md), the wedge alert (05-mimir-ingest-storage-wedged.md), and the Mimir dashboards all go blind.
  • The cluster's Mimir is still running and serving queries — this is a monitoring blind spot, not a service outage.
  • Other clusters' Mimir self-monitoring is unaffected (the wedge is per-cluster, on the alloy-metrics pod that OOMed).

Diagnosis

The alert label carries k8s_cluster_name. From the alert, note it.

1. Confirm the absence

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml

Confirm via Mimir (port-forward synapse-server-mimir-gateway, header X-Scope-OrgID: corti):

# Should return 0 if wedged, > 0 if healthy
QUERY='count by (k8s_cluster_name) (up{job=~"monitoring-system/.*", k8s_cluster_name="<cluster>"})'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[]? | "\(.metric.cluster): \(.value[1]) targets"'

# The kubelet path should still have data (confirms the cluster is up, only the ServiceMonitor path is wedged)
QUERY2='count by (k8s_cluster_name) (kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~".*mimir-ingester.*", k8s_cluster_name="<cluster>"})'

2. Check the alloy-metrics pod

kubectl -n monitoring-system get pods -l app.kubernetes.io/name=alloy-metrics -o wide
kubectl -n monitoring-system get pods -l app.kubernetes.io/name=alloy-metrics -o jsonpath='{range .items[*]}{.metadata.name} restarts={.status.containerStatuses[0].restartCount} phase={.status.phase}{"\n"}{end}'

A high restart count with phase=Running (not CrashLoop) is the signature — the pod OOMed, restarted, and its ServiceMonitor discovery did not re-attach.

3. Confirm it is the operator-discovery wedge, not the operator reconcile wedge

The alloy-operator reconcile wedge (01-alloy-operator-wedge.md) is a different failure: the operator cannot reconcile the Alloy CR (missing Helm ownership annotations) and the StatefulSet is frozen on stale config. Check whether the operator is erroring:

QUERY='sum by (k8s_cluster_name, namespace) (rate(controller_runtime_reconcile_total{controller="alloy-controller", result="error"}[15m]))'

If > 0, the operator is wedged too — see 01-alloy-operator-wedge.md for the annotate-adopt remediation. If 0, only the alloy-metrics pod's discovery wedged — the simpler recovery below is sufficient.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Restart alloy-metrics (the immediate recovery)

Delete the alloy-metrics pod — the StatefulSet recreates it and ServiceMonitor discovery re-attaches on the fresh pod:

kubectl -n monitoring-system delete pod -l app.kubernetes.io/name=alloy-metrics

Within ~1 minute, up{job=~"monitoring-system/.*"} targets reappear and Mimir self-monitoring metrics resume.

Step 3 — Root fix: prevent the OOM

The root cause is alloy-metrics OOMing under memory pressure. synapse-client >= 0.9.2 ships 3Gi memory limit + GOMEMLIMIT=2700MiB to stop the OOM (see charts/charts/synapse-client/values.yaml). If the affected cluster is on an older synapse-client, bump it — the GOMEMLIMIT soft ceiling lets the Go runtime GC before the kernel OOM-kills the process.

If the operator is also wedged (Diagnosis step 3 showed reconcile errors), follow 01-alloy-operator-wedge.md to re-attach the Helm ownership annotations before restarting — otherwise the pod restarts on the stale config.

Verify

  • The alert clears once count by (k8s_cluster_name) (up{job=~"monitoring-system/.*", k8s_cluster_name="<cluster>"}) returns > 0, sustained for the for: 15m window.
  • Mimir self-monitoring metrics reappear: cortex_ingester_memory_series{k8s_cluster_name="<cluster>"} returns data.
  • alloy-metrics restart count stops increasing.

Notes

  • Absence alert — this fires on missing metrics, not a bad value. The unless operator is the only reliable detector because the alloy components stay healthy through the wedge.
  • Relationship to the operator wedge — 01-alloy-operator-wedge.md is the operator-reconcile failure (StatefulSet frozen on stale config). This runbook is the pod-discovery failure (targets not re-attaching after an OOM restart). They can co-occur; Diagnosis step 3 distinguishes them.
  • Root fix already shipped — synapse-client >= 0.9.2 (3Gi + GOMEMLIMIT). Bumping the affected cluster to a current synapse-client prevents recurrence.