Orca ORCA v0.174.2
Welcome Developer Guide Infrastructure Platform Engineering Runbooks Archive
Documentation navigation
On Premise Customers
SFD Hamilton County CommonSpirit GMR Region Nord On-Prem Synapse API-Key Deploy
Dns
DNS Flush
Kafka
Kafka Consumer Lag in Voice-Vault Kafka Consumer Group Lag High (> 800) HPA Max Replicas Exceed Kafka Topic Partitions
Storage
Ceph OSD Full / BlueFS Allocation Wedge Query Ceph RGW (S3) Bucket Usage Ceph Mon Quorum Loss / Node-Replacement Recovery PVC Near Full Warning (>= 80%) PVC Near Full Critical (>= 90%) PVC Disk Usage Forecast > 95% Ceph Cluster Storage Utilization High (> 90%) Ceph Cluster Error Ceph Cluster Warning
Observability
Alloy Operator Wedge Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild Grafana Dashboard Provisioning Error Mimir Ingester Reaching Series Limit Mimir Ingest Storage Wedged Mimir Ingester PVC Near Full Loki PVC Near Full Mimir Self-Monitoring Scrape Stalled Synapse Cluster Metrics Silent Mimir Discarding Samples (Series Limit) Loki Discarding Log Entries Trivy Metrics Stale Envoy Proxy Heap Near Ceiling Envoy Proxy Heap Shedding Envoy Proxy Overload Rejecting Requests Traefik Elevated Error Spike (4xx/5xx) Envoy Elevated Error Spike (4xx/5xx) Scrape Targets Failing Traefik 402 Payment Failure (Multiple Distinct Users) Azure Audit Ingestion Silent
Databases
CNPG Cluster Has No Backup CNPG Replication Unhealthy CNPG Cluster Backup Stale (> 48h) CNPG Cluster Has No WAL Recovery Point
Kubernetes
Pod ImagePull / ImagePullBackOff Pod CrashLoopBackOff Deployment Has No Running Replicas Deployment Rollout Stuck DaemonSet Has No Running Pods StatefulSet Has No Running Replicas Kubernetes Job Failed Kubernetes CronJob Suspended Certificate Expiring Soon Node Memory/Disk Pressure (Kubelet Evicting Pods) Container OOMKilled Cluster Autoscaler Unhealthy Trivy Critical Vulnerability One Week Trivy Critical Vulnerability Four Weeks Trivy High Vulnerability One Week Trivy High Vulnerability Six Weeks Certificate Renewal Stuck Certificate Never Issued Trivy Critical Vulnerability Immediate API v2 Debug Endpoint Called in Production Cluster Autoscaler Unregistered Nodes Platform API Server Errors Platform API Client Errors Pod Pending Over 30 Minutes Pod Pending (GPU) Over 60 Minutes Cilium Policy Drops Burst
Capacity
CPU Throttling > 20% HPA Utilization Critically High GPUs Underutilized For 48h Container Memory Leak (Rising Near Limit) Nodegroup Utilization Critically High Nodegroup Utilization High Warning
Identity
Keycloak Client Misconfigured
Synthetic Monitoring
Synthetic Feature Failing Synthetic Monitor Silent Speech to Text – Synthetic Check Failing Corti Models – Synthetic Check Failing Text Generation – Synthetic Check Failing
Platform
Orca GitHub Rate-Limit Budget Low Email Service Minute Quota High Kyverno Admission API Server Unreachable Email Service Hourly Quota High Email Service Send Throttled Email Service Bounce Rate High

Mimir Ingester PVC Near Full

Runbook for a Mimir ingester PVC approaching full. For the generic PVC near-full runbook (any PVC, expansion procedure, StatefulSet volumeClaimTemplate convergence), see 04-storage/04-pvc-near-full.md. This runbook is the Mimir-ingester-specific overlay: the failure mode (disk-full deadlock wedges ingestion) and the scoped PVC regex (.*mimir-ingester.*) differ.

Meaning

With the filesystem blocks backend, the Mimir ingester's local PVC is the only copy of recent data — blocks are flushed to the PVC before upload to object storage. A full ingester disk causes a disk-full deadlock: the ingester cannot flush new blocks (no space) and cannot upload old ones to free space (the upload path needs temp space on the same disk). Ingestion halts.

kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes (scraped from each node's kubelet by synapse-client) is the PVC filesystem utilisation. The regex persistentvolumeclaim=~".*mimir-ingester.*" scopes to the ingester PVCs. > 0.8 sustained for 10m is the warning tier; > 0.9 for 5m is the critical tier (imminent deadlock).

The metrics are already in the synapse-client kubelet allow-list (kubelet_volume_stats_used_bytes, kubelet_volume_stats_capacity_bytes) — no chart PR is needed.

What changed from the chart

  • All environments — the chart alert had no environment filter. Kept as-is. The {{ $labels.environment }} is in the summary/description.
  • Two tiers, one runbook — the warning tier (80%) is this runbook's alert. The critical tier (90%) is the same expr with > 0.9 and severity: critical. Create a separate runbook if you want a distinct critical alert name.

Impact

  • The ingester disk is approaching full. At 100% the disk-full deadlock halts ingestion — new samples are not consumed and the ingester cannot recover without human intervention.
  • Recent data on the full ingester is at risk: if the disk corrupts under pressure, the un-uploaded blocks are lost.
  • The ingester's PVC is a volumeClaimTemplate on a StatefulSet — expansion requires the GitOps convergence procedure (see 04-storage/04-pvc-near-full.md Mitigation step 3).

Diagnosis

The alert labels carry k8s_cluster_name, environment, namespace, persistentvolumeclaim. From the alert, note all of them.

1. Confirm the utilisation and identify the ingester

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=monitoring-system
kubectl -n $NS get pvc <persistentvolumeclaim> -o wide
kubectl -n $NS get pod -o wide | grep mimir-ingester   # find the pod mounting the PVC

Confirm via Mimir (port-forward synapse-server-mimir-gateway, header X-Scope-OrgID: corti):

QUERY='kubelet_volume_stats_used_bytes{persistentvolumeclaim="<persistentvolumeclaim>"} / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="<persistentvolumeclaim>"}'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

A value > 0.8 confirms the alert.

2. Is ingestion already wedged?

Check whether the ingester is failing records (the disk-full deadlock manifests as the ingest-storage wedge):

QUERY='sum by (pod, namespace, k8s_cluster_name) (rate(cortex_ingest_storage_reader_records_failed_total[5m]))'

If > 0, see 05-mimir-ingest-storage-wedged.md.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Expand the PVC (the durable fix)

Follow 04-storage/04-pvc-near-full.md Mitigation step 3 for the full procedure — the ingester PVC is a volumeClaimTemplate on the Mimir ingester StatefulSet, so the zero-downtime orphan-delete + ArgoCD sync convergence applies:

# 1. Capacity fix (urgent, online) — expands the PVC immediately
kubectl -n monitoring-system patch pvc <persistentvolumeclaim> --patch '{"spec":{"resources":{"requests":{"storage":"<new-size>Gi"}}}}'

# 2. GitOps convergence — edit the synapse-server chart values to match, merge, then:
kubectl -n monitoring-system delete sts <statefulset> --cascade=orphan
# ArgoCD recreates the STS with the new volumeClaimTemplate and re-adopts the running pod

Step 3 — Trim retention / cardinality (reversible)

If expanding the PVC is not immediately possible, reduce the data on the disk:

  • Lower the ingester's max_series to reduce active series (carefully — see 04-mimir-ingester-series-limit.md).
  • Confirm the compactor is enabled and retention is reclaiming space.
  • Reduce the blocks retention period if the ingester holds more history than needed.

Verify

  • The alert clears once used / capacity <= 0.8 sustained for the for: 10m window.
  • kubectl -n monitoring-system get pvc <persistentvolumeclaim> shows the updated Capacity.
  • cortex_ingest_storage_reader_records_failed_total does not appear (no wedge).

Notes

  • Metrics already collected — kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes are in the synapse-client kubelet allow-list. No chart PR required.
  • Two tiers, one runbook — warning (80%) is this alert; critical (90%) is the same expr with > 0.9.
  • Filesystem blocks backend — the disk-full deadlock is specific to the filesystem blocks backend. If Mimir is configured with a different backend, the failure mode differs.
  • See also — 04-storage/04-pvc-near-full.md (generic PVC expansion), 05-mimir-ingest-storage-wedged.md (post-wedge), 04-mimir-ingester-series-limit.md (series limit that can drive disk fill).