Documentation navigation
PVC Near Full Warning (>= 80%)
Runbook for a PersistentVolumeClaim that has crossed 80% utilisation — the warning tier. This is the early signal; the critical tier (>= 90%) is 04-storage/04-pvc-near-full-critical.md. Both share the same diagnosis and mitigation, documented in the critical runbook.
Meaning
A PersistentVolumeClaim is a cluster-attached volume that one or more pods write to. kubelet_volume_stats_* reports per-PVC filesystem usage from the node's kubelet. When utilisation crosses 80% the volume is trending toward its capacity ceiling — not yet critical, but on track to fill if the workload keeps growing or retention/compaction is not reclaiming space. The warning tier catches this early so there is time to investigate and resize or clean up before the critical tier (90%) fires.
The metric is scraped from each node's kubelet (:10250) by synapse-client's alloy-metrics and remote-written to the central synapse-server Mimir, so the alert sees every cluster's PVCs in one place.
Impact
- The PVC is filling but has not reached the critical 90% threshold. There is lead time.
- If the growth trend continues, the critical-tier alert (
04-storage/04-pvc-near-full-critical.md) fires at 90%, and at 100% pod writes fail (Kubernetes raisesDiskPressureand evicts the pod). - Stateful workloads (CNPG/Postgres, Kafka, Redis, Mimir ingesters, Loki) are the most affected by a full volume.
Diagnosis
See 04-storage/04-pvc-near-full-critical.md § Diagnosis for the full procedure (identify the PVC and consumer, find what is consuming the space, check the storage backend).
The Mimir query to confirm current utilisation:
curl -s -H "X-Scope-OrgID: corti" \
"http://localhost:9090/prometheus/api/v1/query?query=$(python3 -c 'import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))' \
'(1 - (kubelet_volume_stats_available_bytes{persistentvolumeclaim="<persistentvolumeclaim>"} / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="<persistentvolumeclaim>"})) * 100')"
Mitigation
See 04-storage/04-pvc-near-full-critical.md § Mitigation for the full procedure (activate PIM, free space on the volume, resize the PVC, GitOps convergence for StatefulSet volumeClaimTemplate).
At the warning tier the urgency is lower — a PVC at 80-89% has time before it hits 90%. Use this window to:
- Investigate the space consumer (Diagnosis step 2).
- Free disposable data if the growth is transient (logs, temp files, old WAL).
- Plan a PVC resize if the workload legitimately needs more space, before the critical tier fires.
Verify
- The alert clears once utilisation is back below 80% for 5m.
kubectl -n <namespace> exec <pod> -- df -h <mount-path>shows the new free space.- For a resize,
kubectl get pvcshows the updatedCapacityand noFileSystemResizePendingcondition.
Notes
- Two tiers — warning (>= 80%, this runbook, severity: warning) and critical (>= 90%,
04-storage/04-pvc-near-full-critical.md, severity: critical). Both priority 2. - Same diagnosis/mitigation as the critical runbook — the root causes and fixes are identical; only the threshold and urgency differ.
- All environments — no
environment="production"filter; fires for every cluster's PVCs (repo convention).