Orca ORCA v0.174.2
Welcome Developer Guide Infrastructure Platform Engineering Runbooks Archive
Documentation navigation
On Premise Customers
SFD Hamilton County CommonSpirit GMR Region Nord On-Prem Synapse API-Key Deploy
Dns
DNS Flush
Kafka
Kafka Consumer Lag in Voice-Vault Kafka Consumer Group Lag High (> 800) HPA Max Replicas Exceed Kafka Topic Partitions
Storage
Ceph OSD Full / BlueFS Allocation Wedge Query Ceph RGW (S3) Bucket Usage Ceph Mon Quorum Loss / Node-Replacement Recovery PVC Near Full Warning (>= 80%) PVC Near Full Critical (>= 90%) PVC Disk Usage Forecast > 95% Ceph Cluster Storage Utilization High (> 90%) Ceph Cluster Error Ceph Cluster Warning
Observability
Alloy Operator Wedge Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild Grafana Dashboard Provisioning Error Mimir Ingester Reaching Series Limit Mimir Ingest Storage Wedged Mimir Ingester PVC Near Full Loki PVC Near Full Mimir Self-Monitoring Scrape Stalled Synapse Cluster Metrics Silent Mimir Discarding Samples (Series Limit) Loki Discarding Log Entries Trivy Metrics Stale Envoy Proxy Heap Near Ceiling Envoy Proxy Heap Shedding Envoy Proxy Overload Rejecting Requests Traefik Elevated Error Spike (4xx/5xx) Envoy Elevated Error Spike (4xx/5xx) Scrape Targets Failing Traefik 402 Payment Failure (Multiple Distinct Users) Azure Audit Ingestion Silent
Databases
CNPG Cluster Has No Backup CNPG Replication Unhealthy CNPG Cluster Backup Stale (> 48h) CNPG Cluster Has No WAL Recovery Point
Kubernetes
Pod ImagePull / ImagePullBackOff Pod CrashLoopBackOff Deployment Has No Running Replicas Deployment Rollout Stuck DaemonSet Has No Running Pods StatefulSet Has No Running Replicas Kubernetes Job Failed Kubernetes CronJob Suspended Certificate Expiring Soon Node Memory/Disk Pressure (Kubelet Evicting Pods) Container OOMKilled Cluster Autoscaler Unhealthy Trivy Critical Vulnerability One Week Trivy Critical Vulnerability Four Weeks Trivy High Vulnerability One Week Trivy High Vulnerability Six Weeks Certificate Renewal Stuck Certificate Never Issued Trivy Critical Vulnerability Immediate API v2 Debug Endpoint Called in Production Cluster Autoscaler Unregistered Nodes Platform API Server Errors Platform API Client Errors Pod Pending Over 30 Minutes Pod Pending (GPU) Over 60 Minutes Cilium Policy Drops Burst
Capacity
CPU Throttling > 20% HPA Utilization Critically High GPUs Underutilized For 48h Container Memory Leak (Rising Near Limit) Nodegroup Utilization Critically High Nodegroup Utilization High Warning
Identity
Keycloak Client Misconfigured
Synthetic Monitoring
Synthetic Feature Failing Synthetic Monitor Silent Speech to Text – Synthetic Check Failing Corti Models – Synthetic Check Failing Text Generation – Synthetic Check Failing
Platform
Orca GitHub Rate-Limit Budget Low Email Service Minute Quota High Kyverno Admission API Server Unreachable Email Service Hourly Quota High Email Service Send Throttled Email Service Bounce Rate High

PVC Near Full Critical (>= 90%)

Runbook for a PersistentVolumeClaim that has crossed 90% utilisation — the critical tier. The warning tier (>= 80%) is 04-storage/03-pvc-near-full-warning.md; both share the same diagnosis and mitigation, which are documented here.

Meaning

A PersistentVolumeClaim is a cluster-attached volume that one or more pods write to. kubelet_volume_stats_* reports per-PVC filesystem usage from the node's kubelet. When utilisation crosses 90% the volume is at its capacity ceiling: pod writes will begin to fail once the volume is exhausted (Kubernetes raises DiskPressure and evicts the pod). The warning tier (>= 80%) catches the trend earlier; this critical tier (>= 90%) means the volume is nearly full.

The metric is scraped from each node's kubelet (:10250) by synapse-client's alloy-metrics and remote-written to the central synapse-server Mimir, so the alert sees every cluster's PVCs in one place.

Impact

  • The affected PVC is on track to fill. If it reaches 100%, the pod(s) mounting it fail to write and Kubernetes evicts them with DiskPressure.
  • Stateful workloads (CNPG/Postgres, Kafka, Redis, Mimir ingesters, Loki) lose availability when their data volume fills — writes halt and the service degrades or stops.
  • A full PVC on a shared storage backend (e.g. Ceph RBD) can cascade: a wedged consumer blocks pool operations and can tip the whole cluster toward full (see 04-storage/01-ceph-osd-full-bluefs-wedge.md).

Diagnosis

1. Identify the PVC and its consumer

The alert labels carry k8s_cluster_name, namespace, and persistentvolumeclaim. From the alert in Grafana, note all three.

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
kubectl -n <namespace> get pvc <persistentvolumeclaim> -o wide
kubectl -n <namespace> get pods -o wide | grep <persistentvolumeclaim>   # find the pod(s) mounting it

Confirm current utilisation directly from the kubelet metric via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

# instant query: current % used for the flagged PVC
curl -s -H "X-Scope-OrgID: corti" \
  "http://localhost:9090/prometheus/api/v1/query?query=$(python3 -c 'import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))' \
    '(1 - (kubelet_volume_stats_available_bytes{persistentvolumeclaim="<persistentvolumeclaim>"} / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="<persistentvolumeclaim>"})) * 100')"

2. Find what is consuming the space

Exec into the pod (if it has a shell) and inspect the mount path:

kubectl -n <namespace> exec <pod> -- df -h <mount-path>
kubectl -n <namespace> exec <pod> -- du -sh <mount-path>/* 2>/dev/null | sort -h | tail

For stateful workloads the usual culprits are:

  • CNPG/Postgres — WAL buildup (check pg_stat_wal), an unvacuumed table, or a runaway query spilling to disk.
  • Kafka — retention not enforcing, or a lagging consumer holding segments past retention.
  • Mimir ingester — cortex_ingester_memory_series near max_series (see MimirIngesterReachingSeriesLimit), or un-flushed blocks.
  • Loki — compactor retention disabled, or chunks not compacting.

3. Check the storage backend

If the PVC is backed by Ceph RBD, confirm the backend is not itself near full (a near-full pool makes PVC expansion fail):

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph df     # per-pool %USED; RAW %USED
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df tree

See 04-storage/01-ceph-osd-full-bluefs-wedge.md if the Ceph cluster is also filling.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Free space on the volume (reversible)

Remove disposable data on the volume so the alert clears without a resize:

  • Delete old logs, temp files, or snapshots the workload no longer needs.
  • For Kafka: confirm retention is enabled and reduce retention.ms / retention.bytes for the topic, or remove lagging consumer groups holding segments.
  • For Postgres: run VACUUM / pg_repack on bloated tables; archive + purge old WAL if archive_mode is on.
  • For Mimir/Loki: confirm retention/compaction is enabled and wait for it to reclaim space.

Re-check the Mimir query — it should clear within the for: 5m window once utilisation drops below 90%.

Step 3 — Resize the PVC (the durable fix)

If the workload legitimately needs more space, expand the PVC. Kubernetes supports online expansion for most CSI drivers:

kubectl -n <namespace> patch pvc <persistentvolumeclaim> --patch '{"spec":{"resources":{"requests":{"storage":"<new-size>Gi"}}}}'
kubectl -n <namespace> get pvc <persistentvolumeclaim> -w   # wait for Capacity to update and condition FileSystemResizePending to clear
Important

If the PVC is a volumeClaimTemplate on an ArgoCD-managed StatefulSet, the template is immutable — changing it is a GitOps change (edit the chart values → PR), and the StatefulSet must be recreated for ArgoCD to converge. The capacity fix and the GitOps convergence are two separable steps:

  1. Capacity fix (urgent, online). kubectl patch pvc (above) expands the PVC immediately — the running pod sees the new size within seconds. Do this first; it does not touch the StatefulSet and clears the 90% alert right away. The live PVC can be larger than the stale StatefulSet template with no ill effect.
  2. GitOps convergence (the STS template). Once the chart-values PR merges, ArgoCD errors on sync — spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden — because the volumeClaimTemplate size changed. Fix it with a zero-downtime orphan delete + sync: 
    # 1. Orphan-delete the STS: pod + PVC survive, the pod keeps running (no ownerRef).
kubectl -n <namespace> delete sts <statefulset> --cascade=orphan
# 2. Trigger an ArgoCD sync to recreate the STS with the new template:
#    - auto-sync ON  → ArgoCD recreates it automatically within seconds.
#    - auto-sync OFF → trigger manually:
kubectl -n argocd patch application <app-name> --type=merge -p '{"operation":{"sync":{"revision":"HEAD"}}}'
    ArgoCD recreates the StatefulSet with the corrected volumeClaimTemplate, re-adopts the running pod (0 restarts), and reuses the already-expanded PVC.

Do not use the Replace=true sync option for this: it cascades pod deletion (foreground delete) and causes downtime — the orphan-delete above is the zero-downtime equivalent. PVCs cannot be shrunk in place; reducing size requires delete + recreate (destroys that volume's data).

If the PVC is Ceph-RBD-backed, confirm the Ceph pool has headroom before expanding (Diagnosis step 3) — expanding a PVC on a near-full pool can fail mid-operation.

Verify

  • The alert clears once utilisation is back below 90% for 5m.
  • The warning-tier alert (04-storage/03-pvc-near-full-warning.md) may still fire if utilisation is between 80% and 90%.
  • kubectl -n <namespace> exec <pod> -- df -h <mount-path> shows the new free space.
  • For a resize, kubectl get pvc shows the updated Capacity and no FileSystemResizePending condition.
  • For a StatefulSet volumeClaimTemplate resize, the recreated STS template matches the desired size and the ArgoCD app is Synced/Healthy: kubectl -n <namespace> get sts <statefulset> -o jsonpath='{.spec.volumeClaimTemplates[*].spec.resources.requests.storage}'.