PVC Disk Usage Forecast > 95%

Runbook for a PVC that is forecast to exceed 95% utilization within 24h, triggered by the PVC Disk Usage Forecast > 95% alert: a PrometheusRule CRD synced to the Mimir ruler (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). Evaluations dispatch through the Mimir Alertmanager → incident.io. The rule-name label was previously PvcDiskForecastFull — the earlier Grafana-managed version of this alert was retired when the runbook alerts moved to PrometheusRule CRDs.

This is the forward-looking sibling of 04-storage/04-pvc-near-full.md (which fires when a PVC is currently >= 90% full). This alert fires when a PVC is forecast to hit 95% within 24h, giving lead time to act before the volume is exhausted.

Meaning

predict_linear(v range, t) fits a linear regression over the lookback window and extrapolates t seconds into the future. Here: the last 6h of kubelet_volume_stats_used_bytes (the used bytes on a PVC, from the kubelet's statfs on the mount) → forecast 24h ahead. Divided by kubelet_volume_stats_capacity_bytes and × 100 = forecast utilization %. The alert fires when that forecast exceeds 95% for 30m (to ride out transient trend spikes) and the PVC is currently at least half full.

The predicted bytes are wrapped in clamp_min(predict_linear(...) or vector(0), 0) so that an empty series (no samples in the window) or a negative-extrapolated forecast is clamped to 0 instead of leaking garbage (NaN/negative) values into the division.

The and on (cluster, namespace, persistentvolumeclaim) (used / capacity > 0.5) clause is a current-usage floor: the forecast alone is not enough, the PVC must already be >= 50% full. Without it, predict_linear alerted on pod restarts: when a StatefulSet pod is recreated (rolling update, reschedule), the new series starts near-empty and ramps hard while the workload rebuilds its on-disk state — e.g. a Loki ingester's WAL rebuild climbed a 10 GiB PVC from 173 MB to ~900 MB in ~10 minutes, and a 24h extrapolation of that ramp computed a >1000% "forecast" while the volume was actually never above 10% (data-loki-write-5, prd-par01). The 50% floor removes the restart artifact while losing almost no lead time: a PVC genuinely on track to pass 95% within 24h spends the second half of that day above 50%, and the 04-pvc-near-full tiers (>= 80% / >= 90%) backstop anything the guard delays.

Both metrics are scraped from the kubelet (:10250/metrics) by synapse-client's alloy-metrics and remote-written to synapse-server Mimir. They carry namespace + persistentvolumeclaim labels, so the forecast is per-PVC. Each PVC forecast to exceed 95% becomes its own alert instance with cluster, namespace, persistentvolumeclaim, and environment labels preserved.

The two knobs are [6h] (lookback) and 24*3600 (forecast horizon). 6h is a balance: long enough to smooth over hourly fluctuations, short enough to catch a sudden fill that started recently. 24h is the lead time — enough to act (resize, cleanup) before the volume is full.

How this differs from 04-pvc-near-full

A PVC that is currently 70% but filling at 1%/hour will not fire 04-pvc-near-full until it hits 90%, and will not be at 95% for another 25h. This forecast alert fires ~24h earlier, at the point the trend predicts 95% — giving a full day of lead time.

Impact

• Same as 04-storage/04-pvc-near-full.md: the PVC is on track to fill. If it reaches 100%, pod writes fail and Kubernetes evicts the pod with DiskPressure.
• Stateful workloads (CNPG/Postgres, Kafka, Mimir, Loki, redis-node) lose availability when their data volume fills — writes halt and the service degrades or stops.
• The forecast gives 24h of lead time, so this is a warning (act today), not an outage (act now). But a PVC that fills faster than the forecast (a burst) can hit 100% before the 24h — the forecast is a linear extrapolation and underestimates accelerating fills.
• Linear-only caveat: predict_linear assumes the fill rate stays constant. A single burst (backup, log spike) in the 6h window can overshoot the forecast; a PVC that already filled and stopped will keep "forecasting high" until the trend ages out of the window. Cross-check the forecast against the current utilization (Diagnosis step 1) before acting.

Diagnosis

The alert labels carry k8s_cluster_name, namespace, persistentvolumeclaim, and environment. From the incident.io alert page (or Grafana's Alerting page), note all of them.

1. Check current utilization vs forecast

The forecast is a prediction; confirm against the current state:

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=<namespace>
PVC=<persistentvolumeclaim>
kubectl -n $NS get pvc $PVC
kubectl -n $NS get pods -o wide | grep $PVC   # find the pod(s) mounting it

Confirm current utilization via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

# current % used for the flagged PVC
QUERY='(1 - (kubelet_volume_stats_available_bytes{persistentvolumeclaim="<PVC>"} / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="<PVC>"})) * 100'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY"

• current < 90% and forecast > 95% → the PVC is filling fast; the forecast is the early signal. Act now (Mitigation).
• current >= 90% → the sibling 04-pvc-near-full alert should also be firing; follow that runbook (it covers the resize/cleanup procedure in detail).
• current is low (e.g. < 50%) but forecast high → the alert expression itself now enforces this check (current usage must be >= 50% for the alert to fire), so a genuinely received alert here means the PVC is already half full: treat the forecast as the early signal, but first check the trend for a burst that will age out (Diagnosis step 3) before acting.

2. Find what is consuming the space

kubectl -n $NS exec <pod> -- df -h <mount-path>
kubectl -n $NS exec <pod> -- du -sh <mount-path>/* 2>/dev/null | sort -h | tail

For stateful workloads the usual culprits are:

• CNPG/Postgres — WAL buildup (check pg_stat_wal), an unvacuumed table, or a runaway query spilling to disk.
• Kafka — retention not enforcing, or a lagging consumer holding segments past retention.
• Mimir ingester — un-flushed blocks; Loki — compactor retention disabled, chunks not compacting.
• Logs / temp / cache — an app writing to the PVC instead of stdout, or an unbounded cache.

See 04-storage/04-pvc-near-full.md Diagnosis step 2 for the per-workload breakdown.

3. Check the fill rate / trend

Confirm the trend the forecast is based on:

# used bytes over the last 6h (is it a steady climb, or a step?)
QUERY='kubelet_volume_stats_used_bytes{persistentvolumeclaim="<PVC>"}[6h]'

A steady climb confirms a genuine fill trend. A single step (one big write at a point in time) means the forecast is dominated by that step — the PVC may not actually fill (the step won't repeat).

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Free space (reversible) or resize (durable)

The full procedure is in 04-storage/04-pvc-near-full.md Mitigation — both options apply here:

• Free space (reversible): delete disposable data (old logs, temp files, snapshots the workload no longer needs), enforce retention, vacuum/repack bloated tables, run compaction. The forecast clears once the trend drops enough that the 24h projection is under 95%.
• Resize the PVC (durable, the fix if the workload legitimately needs more space): kubectl patch pvc for the immediate capacity fix, then a chart-values PR for the volumeClaimTemplate (the GitOps convergence is documented in the sibling runbook — the orphan-delete + sync procedure).

The forecast gives lead time, so prefer the durable fix (resize via chart PR) over a reversible cleanup when the fill is legitimate growth.

Step 3 — Verify the trend breaks

After the fix, re-run the Mimir forecast query from Diagnosis step 1 — the 24h projection should drop below 95%. The alert clears once the forecast is under 95% sustained for the for: 30m window. Note: the 6h lookback means the forecast can stay elevated for up to 6h after the trend breaks (the high-fill samples age out slowly) — the alert may not clear immediately even after the fix; recheck over a few hours.

Verify

• The alert clears once the 24h forecast drops below 95% sustained for for: 30m. Given the 6h lookback, this can lag the actual fix by up to 6h.
• kubectl -n $NS exec <pod> -- df -h <mount-path> shows the current utilization dropping (for a cleanup) or the new capacity (for a resize).
• For a resize, kubectl get pvc shows the updated Capacity and no FileSystemResizePending condition.
• The sibling 04-pvc-near-full alert (current >= 90%) does not fire — the PVC stays under 90%.
• The Mimir forecast query from Diagnosis step 1 returns a value < 95% within a few hours of the fix.