Orca ORCA v0.174.2
Documentation navigation

PVC Near Full Warning (>= 80%)

Runbook for a PersistentVolumeClaim that has crossed 80% utilisation — the warning tier. This is the early signal; the critical tier (>= 90%) is 04-storage/04-pvc-near-full-critical.md. Both share the same diagnosis and mitigation, documented in the critical runbook.

Meaning

A PersistentVolumeClaim is a cluster-attached volume that one or more pods write to. kubelet_volume_stats_* reports per-PVC filesystem usage from the node's kubelet. When utilisation crosses 80% the volume is trending toward its capacity ceiling — not yet critical, but on track to fill if the workload keeps growing or retention/compaction is not reclaiming space. The warning tier catches this early so there is time to investigate and resize or clean up before the critical tier (90%) fires.

The metric is scraped from each node's kubelet (:10250) by synapse-client's alloy-metrics and remote-written to the central synapse-server Mimir, so the alert sees every cluster's PVCs in one place.

Impact

  • The PVC is filling but has not reached the critical 90% threshold. There is lead time.
  • If the growth trend continues, the critical-tier alert (04-storage/04-pvc-near-full-critical.md) fires at 90%, and at 100% pod writes fail (Kubernetes raises DiskPressure and evicts the pod).
  • Stateful workloads (CNPG/Postgres, Kafka, Redis, Mimir ingesters, Loki) are the most affected by a full volume.

Diagnosis

See 04-storage/04-pvc-near-full-critical.md § Diagnosis for the full procedure (identify the PVC and consumer, find what is consuming the space, check the storage backend).

The Mimir query to confirm current utilisation:

curl -s -H "X-Scope-OrgID: corti" \
  "http://localhost:9090/prometheus/api/v1/query?query=$(python3 -c 'import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))' \
    '(1 - (kubelet_volume_stats_available_bytes{persistentvolumeclaim="<persistentvolumeclaim>"} / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="<persistentvolumeclaim>"})) * 100')"

Mitigation

See 04-storage/04-pvc-near-full-critical.md § Mitigation for the full procedure (activate PIM, free space on the volume, resize the PVC, GitOps convergence for StatefulSet volumeClaimTemplate).

At the warning tier the urgency is lower — a PVC at 80-89% has time before it hits 90%. Use this window to:

  • Investigate the space consumer (Diagnosis step 2).
  • Free disposable data if the growth is transient (logs, temp files, old WAL).
  • Plan a PVC resize if the workload legitimately needs more space, before the critical tier fires.

Verify

  • The alert clears once utilisation is back below 80% for 5m.
  • kubectl -n <namespace> exec <pod> -- df -h <mount-path> shows the new free space.
  • For a resize, kubectl get pvc shows the updated Capacity and no FileSystemResizePending condition.

Notes

  • Two tiers — warning (>= 80%, this runbook, severity: warning) and critical (>= 90%, 04-storage/04-pvc-near-full-critical.md, severity: critical). Both priority 2.
  • Same diagnosis/mitigation as the critical runbook — the root causes and fixes are identical; only the threshold and urgency differ.
  • All environments — no environment="production" filter; fires for every cluster's PVCs (repo convention).