For the specific failure mode where high utilization is accompanied by crashlooping OSDs and unavailable PGs, see sibling 01-ceph-osd-full-bluefs-wedge.md — that is the BlueFS-allocation-wedge runbook. This one is the pre-failure warning: the cluster is filling but still serving I/O. The sibling 04-pvc-near-full.md is the per-PVC analogue; a single PVC filling does not fire this alert (it fires 04-pvc-near-full.md).

Meaning

ceph_cluster_total_used_bytes and ceph_cluster_total_bytes are emitted by the rook-ceph-mgr exporter (job="rook-ceph-mgr", scraped into Mimir). Their ratio is the aggregate RADOS utilization — one shared pool of all OSD raw capacity across every consumer: RBD PVCs (postgres/CNPG, kafka, redis, Mimir), CephFS, and the RGW S3 bucket. > 90 (percent) sustained for 5m means the cluster is between the backfillfull ratio (0.90) and the full_ratio (default 0.95): rebalancing is constrained, and a small uptick sets the full flag and blocks writes cluster-wide.

The expr groups by k8s_cluster_name, environment so each Ceph cluster is its own alert instance. The cluster label is the Mimir-standard cluster label (the DataDog kube_cluster_name maps to it). Ceph runs only on the par fleet (dev-par01, stg-par01, prd-par01) — the AKS clusters use Azure-managed disks, so this alert is effectively par-only in practice, but the expr is environment-agnostic.

What changed from the DataDog monitor

All environments — the DataDog query hardcoded environment:development. Per repo convention this alert is generic across all environments (dev/staging/prod); the environment label is in the summary/description so the fired alert names it. The par fleet has Ceph on dev-par01 (development), stg-par01 (staging), and prd-par01 (production) — all three are covered.
Threshold 90% — matches the DataDog query * 100 > 90. Ceph's own backfillfull ratio is 0.90 and nearfull is 0.85; 90% is the right "act now" threshold (above backfillfull, below full). Adjust the > 90 in the expr to tighten/loosen.
Smoothing — DataDog used avg(last_5m); this alert uses the instant ratio with for: 5m (fire only if sustained > 90% for 5m). Functionally equivalent for alerting.
Routing — DataDog did not route per-product. This runbook routes to team: platform (Ceph is platform-owned infra), matching the sibling 04-pvc-near-full.md and 01-ceph-osd-full-bluefs-wedge.md.

Impact

The cluster is between backfillfull and full. At full (default 0.95) Ceph sets the full flag and blocks all writes cluster-wide — every RBD PVC (postgres, kafka, redis, Mimir), CephFS, and RGW write fails.
OSDs that physically approach 100% on a fragmented device hit a BlueFS allocation-alignment bug on the next restart and crashloop (the 01-ceph-osd-full-bluefs-wedge.md failure mode). High utilization + fragmentation is the trigger; staying under 90% avoids it.
Recovery from a full cluster is slow: you must free space (delete data, lower retention) or add OSD capacity, neither of which is instant on a filled EC pool.

Diagnosis

The alert labels carry k8s_cluster_name and environment. From the alert in Grafana / incident.io, note both.

1. Confirm the utilization and see the breakdown

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=rook-ceph
kubectl -n $NS exec deploy/rook-ceph-tools -- ceph df      # RAW %USED + per-pool %USED (find the consumer)
kubectl -n $NS exec deploy/rook-ceph-tools -- ceph osd df tree   # per-OSD %use, up/down, host
kubectl -n $NS exec deploy/rook-ceph-tools -- ceph health detail
kubectl -n $NS exec deploy/rook-ceph-tools -- ceph osd dump | grep -E 'full_ratio|flags'

Confirm the live ratio via Mimir (via the Grafana datasource proxy, or port-forward synapse-server-mimir-gateway in monitoring-system with header X-Scope-OrgID: corti):

QUERY='(sum by (k8s_cluster_name,environment) (ceph_cluster_total_used_bytes{k8s_cluster_name="<cluster>"}) / sum by (k8s_cluster_name,environment) (ceph_cluster_total_bytes{k8s_cluster_name="<cluster>"})) * 100'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

A value > 90 confirms the alert. ceph df shows which pool is consuming the space (block, CephFS, RGW).

2. Is it the BlueFS wedge?

If any OSD is crashlooping or the cluster shows BLUESTORE_FREE_FRAGMENTATION / pgs down, you are in the 01-ceph-osd-full-bluefs-wedge.md failure mode — stop and follow that runbook (its remediation is different and data-loss-sensitive: do not purge/zap a down OSD before reviving it).

kubectl -n $NS get pods -l app=rook-ceph-osd -o jsonpath='{range .items[*]}{.metadata.name} restarts={.status.containerStatuses[0].restartCount} phase={.status.phase}{"\n"}{end}'
kubectl -n $NS exec deploy/rook-ceph-tools -- ceph health detail | grep -E 'BLUESTORE_FREE_FRAGMENTATION|OSD_DOWN|PG_AVAILABILITY'

3. Identify the consuming pool / workload

ceph df identifies the pool. Map the pool to its consumers:

ceph-block-ec-pool (RBD) → PVCs. Find the biggest PVCs: kubectl -n $NS exec deploy/rook-ceph-tools -- rbd diff --pool <pool> stat or check per-PVC usage via the kubelet_volume_stats_* metric (see 04-pvc-near-full.md). Usual culprits: postgres/CNPG (WAL buildup, unvacuumed tables), kafka (retention not enforcing), redis, Mimir ingester.
CephFS → file-system mounts. Check ceph fs status and the largest subvolumes.
RGW (S3) → the object store. See 04-storage/02-ceph-rgw-bucket-usage.md for bucket-level accounting.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Free space (reversible, do first)

Delete disposable data to drop below 90% without a capacity change:

Kafka — lower retention.ms / retention.bytes for the largest topics, or remove lagging consumer groups holding segments past retention.
Postgres/CNPG — VACUUM / pg_repack bloated tables; archive + purge old WAL if archive_mode is on.
Mimir/Loki — confirm retention/compaction is enabled and reclaiming; the Mimir compactor and Loki compactor delete expired blocks/chunks on their schedule.
RGW — enable or lower bucket lifecycle expiration for the largest buckets (see 04-storage/02-ceph-rgw-bucket-usage.md).

Re-check ceph df and the Mimir query — it should clear within the for: 5m window once aggregate utilization drops below 90%.

Step 3 — Add OSD capacity (the durable fix)

If the workload legitimately needs the space, add capacity. On the par fleet (bare-metal Scaleway/kommodity) this is a node/OSD addition, not a cloud-disk resize — coordinate with the platform team. Do not attempt to shrink a pool to free space (EC pools cannot be shrunk without data loss).

Step 4 — If OSDs are crashlooping

You are in the BlueFS-wedge failure mode — stop and follow 04-storage/01-ceph-osd-full-bluefs-wedge.md. Freeing space alone does not revive a crashlooping OSD; the wedge runbook covers the revive procedure and the data-loss caveats.

Verify

The alert clears once (used / total) * 100 <= 90 sustained for the for: 5m window.
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph df shows RAW %USED under 90 and the consuming pool dropping.
ceph health detail no longer shows BLUESTORE_FREE_FRAGMENTATION / OSD_BACKFILLFULL warnings (unless you are intentionally running near full).
Re-run the Mimir query from Diagnosis step 1 to confirm the ratio is under 90.

Notes

Metrics already collected — ceph_cluster_total_used_bytes and ceph_cluster_total_bytes are both scraped from rook-ceph-mgr into Mimir, so this alert fires as soon as the runbook merges and the Terraform module syncs the PrometheusRule CRD. No chart PR is required.
Par fleet only — Ceph runs only on dev-par01, stg-par01, prd-par01. The AKS clusters (prod-weu, prod-us, dev-weu, staging-eu) use Azure-managed disks; this alert has no series there and will not fire. The expr is environment-agnostic regardless.
Relationship to the wedge runbook — 01-ceph-osd-full-bluefs-wedge.md is the post-failure runbook (OSDs crashlooping, PGs down) and shares this alert's name in DataDog. This runbook is the pre-failure warning. If OSDs are crashlooping, follow the wedge runbook instead.
ceph_cluster_total_used_raw_bytes — an alternative numerator that reports raw EC usage (before k+m coding amplification). ceph_cluster_total_used_bytes (used here) matches the DataDog monitor and the ceph df RAW %USED. Switch to _raw_bytes if you want the post-coding-amplification view.