Runbook for the CEPH Cluster Storage Utilization High DataDog alert (kube_cluster_name:<cluster>) when it is accompanied by crashlooping OSDs and unavailable data.

First observed 2026-07-10 on dev-par01 (rook-ceph, Ceph 19.2.3 "squid"): the aggregate RADOS backend filled to ~93%, three OSDs crashlooped, 80 PGs went down, and Grafana/Mimir could no longer read from the object store.

Meaning

The rook-ceph cluster is one shared RADOS pool of all OSD raw capacity. Every consumer — RBD PVCs (postgres/CNPG, kafka, redis, Mimir), CephFS, and the RGW S3 bucket — draws from that single pool. When aggregate usage crosses Ceph's full_ratio (default 0.95; nearfull 0.85, backfillfull 0.90) the cluster sets the full flag and blocks writes cluster-wide.

Worse, OSDs that physically approach 100% on a fragmented device hit a BlueFS allocation-alignment bug on the next restart and crashloop:

/…/src/os/bluestore/BlueFS.cc: 3871: FAILED ceph_assert(cut_off == p->length)
… in BlueFS::truncate(FileWriter*, uint64_t) …
*** Caught signal (Aborted) **   (exit code 134)

This is a known Ceph 19.2.3 (squid) issue, triggered by high utilisation (~75 %+) plus high free-space fragmentation (score > 0.75). The truncate cannot obtain a 64 KiB-aligned allocation and asserts. It is not general store corruption — an offline fsck of the same OSD passes; only the live-boot rocksdb WAL replay hits the assert.

References:

Proxmox forum — Ceph squid OSD crash ceph_assert(cut_off == p->length)
Ceph PR #45171 — Fix data corruption in bluefs truncate()
Red Hat — ODF OSD CrashLoopBackOff "bluefs enospc"
Ceph tracker #49110; BlueFS.cc source

Impact & data-loss risk

The rook block pool ceph-block-ec-pool is erasure-coded k=4 m=2, failure-domain=host (6 shards per PG, one per host; 6 hosts × 1 OSD). It tolerates losing 2 hosts. Losing 3 OSDs/hosts leaves only 3 shards per PG — below k=4 — so every PG in the pool goes down and its data is unavailable (postgres, keycloak, kafka, redis, Mimir — not just metrics).

The data is not lost while the down OSDs' disks are intact. Recovery requires bringing at least one down OSD back (its shards restore the ≥k=4 needed to reconstruct).

⚠️ Do NOT purge/zap/recreate a down OSD before reviving at least one. With 3/6 down, every PG's surviving shards may live only on the down disks — zapping is permanent EC data loss for the whole pool. Revive first; rebuild only as a last resort.

Diagnosis

kubectl/ceph from the mgr and osd pods hangs while the cluster is this sick (mon quorum is fine, but client sessions stall). Use a mon pod with an explicit admin key.

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. dev-par01
NS=rook-ceph
MON=$(kubectl -n $NS get pods -l app=rook-ceph-mon -o name | head -1)
# mon container has no client config; pass the admin key + a mon IP explicitly:
K=$(kubectl -n $NS get secret rook-ceph-mon -o jsonpath='{.data.ceph-secret}' | base64 -d)
MONIP=$(kubectl -n $NS get cm rook-ceph-mon-endpoints -o jsonpath='{.data.data}' | sed 's/.*=\([0-9.]*\):.*/\1/' | head -1)
CEPH="kubectl -n $NS exec $MON -c mon -- ceph --name client.admin --key $K -m $MONIP"

$CEPH -s                 # overall: look for 'pgs down', 'backfillfull', 'osds down'
$CEPH df                 # RAW %USED + per-pool %USED (find the consumer)
$CEPH osd df tree        # per-OSD %use, up/down, host/zone
$CEPH health detail      # BLUESTORE_FREE_FRAGMENTATION, OSD_BACKFILLFULL, PG_AVAILABILITY
$CEPH osd dump | grep -E 'full_ratio|flags'

Confirm the crash signature on a down OSD (works from the OSD pod even while crashlooping):

kubectl -n rook-ceph logs <rook-ceph-osd-N-pod> -c osd --previous | grep 'FAILED ceph_assert'

Confirm the store is otherwise intact (read-only) via the rook-ceph plugin maintenance pod:

kubectl krew install rook-ceph      # once, client-side
kubectl rook-ceph maintenance start rook-ceph-osd-N      # stops OSD, keeps disk attached
POD=$(kubectl -n rook-ceph get pods -l ceph_daemon_id=N -o name | head -1)
kubectl -n rook-ceph exec $POD -c osd -- ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-N
# → "fsck success" = store consistent; the crash is the BlueFS alloc bug, not corruption
kubectl rook-ceph maintenance stop rook-ceph-osd-N       # restore

Remediation

1. Get the OSDs booting again (the BlueFS alloc workaround)

Lower the BlueFS shared allocation unit from 64 KiB to 4 KiB so truncate can satisfy the alignment on a fragmented device, then restart the OSD. The key is bluefs_shared_alloc_size (the Proxmox thread's bluestore_shared_alloc_size is a typo — that key does not exist).

# per-OSD (targeted); repeat for each affected id, or set 'osd' for all daemons:
$CEPH config set osd.N bluefs_shared_alloc_size 4096
$CEPH config get osd.N bluefs_shared_alloc_size          # verify -> 4096
kubectl -n rook-ceph delete pod -l ceph_daemon_id=N       # restart to pick up config

Watch it boot past the assert (rocksdb WAL replay takes minutes; restarts stays 0):

watch "$CEPH osd stat; $CEPH pg stat | grep -o '[0-9]* down'"

Apply to every OSD, not just the first. All OSDs at high utilisation are vulnerable; healthy ones will hit the same assert under recovery load. Setting it on the still-up OSDs (osd.6, osd.7 in the dev-par01 incident) is a no-op until they restart but protects them if they flap.

Bringing back one down OSD returns the down PGs to active (≥k=4 shards); bringing back the rest restores redundancy and capacity. fsck-clean OSDs boot fine with the 4 KiB setting — no repair needed.

2. Free space (the actual root cause)

The alloc workaround only masks fragmentation; the cluster is still full. Reclaim the top consumer — check ceph df per-pool %USED. On dev-par01 it was the Mimir ingest-storage RBD PVCs (data-0-mimir-ingest-broker-{0,1,2}, 500 Gi each on rook-ceph-block), not the RGW object bucket (which was ~3 GiB).

PVCs cannot be shrunk in place (Kubernetes allows expansion only). To reduce a PVC you must delete + recreate it (destroys that volume's data) — acceptable on dev for reproducible data (Mimir WAL, un-flushed samples).
The StatefulSet volumeClaimTemplate is ArgoCD-managed and immutable → the resize is a GitOps change (edit chart values in this repo → PR), plus an STS recreate (Replace=true syncOption or manual STS+PVC delete).

2a. Breaking a full-cluster write block (stuck deletes + rbd trash)

If the cluster crosses full_ratio (0.95) — HEALTH_ERR, 1 full osd(s), N pool(s) full, a pool showing %USED 100.00 / MAX AVAIL 0 — all writes are blocked cluster-wide. Symptoms cascade far from Ceph: any pod that writes hangs (e.g. a Kafka broker's fsGroup chown stays at processed 0 files; the VolumePermissionChangeInProgress event never completes), and new PVCs sit Pending because provisioning an RBD image is itself a write.

The trap: deleting PVCs to free space does not work at 100% full, because the delete operations are also writes:

Deleting a PVC leaves the PV Released; CSI's DeleteVolume then moves the RBD image to rbd trash (deferred delete — this part succeeds and the PV object disappears), but the actual object reclaim only happens on trash purge, which is blocked while the cluster is full.
So you see PVs drain from Released → gone, yet ceph df STORED does not drop and %RAW USED stays pinned. The space is all sitting in rbd trash.

Break-out (Ceph admin; all reversible). Verify the target Released PVs are disposable first (kubectl get pv → check claimRef; on dev-par01 all 20 were monitoring-system Mimir/Loki/Kafka — never postgres/keycloak):

# 1. Stop recovery/backfill so the headroom you free is used by DELETES, not new backfill data.
$CEPH osd set nobackfill; $CEPH osd set norebalance; $CEPH osd set norecover

# 2. Nudge full_ratio just above the fullest OSD (check `ceph osd df`; e.g. fullest 95.03% → 0.96).
#    This clears the `full` flag so the queued rbd operations can run. Keep it SMALL.
$CEPH osd set-full-ratio 0.96

# 3. The Released-PV deletes now flush (PV count drops), but the images land in rbd trash —
#    STORED still won't drop until you PURGE. The RBD image lives in the *metadata* pool.
$CEPH osd dump | grep full_ratio           # confirm 0.96 took
rbd -p ceph-block-ec-pool-metadata trash ls | wc -l    # count trashed images
rbd -p ceph-block-ec-pool-metadata trash purge         # reclaim — SLOW without fast-diff on big images

# 4. Once `ceph df` shows healthy headroom, RESTORE immediately.
$CEPH osd set-full-ratio 0.95
$CEPH osd unset norecover; $CEPH osd unset norebalance; $CEPH osd unset nobackfill

Notes:

rbd trash purge on multi-GiB images without fast-diff is slow and will time out a client exec (~minutes) while still running server-side — it is idempotent, just re-run to finish the remainder. Watch ceph df AVAIL climb.
On dev-par01 (2026-07-10) this reclaimed ~220 GiB and took the cluster 94.82% → 70.76%; the write block cleared and the dependent Kafka brokers/cruise-control started on their own the moment writes were allowed.
This only removes the full flag risk; it does not add capacity. Recovery/backfill after down OSDs rejoin reconstructs degraded EC shards, which writes more data and can re-tip a near-full cluster into full — that is exactly how dev-par01 went from 93 % to full. Freeing disposable data is a stopgap; the durable fix is more OSD capacity (below).

3. Recover fully & prevent recurrence

After space is freed, BLUESTORE_FREE_FRAGMENTATION and OSD_BACKFILLFULL clear as backfill completes.
Consider reverting bluefs_shared_alloc_size to the default (65536) once utilisation and fragmentation are healthy: $CEPH config rm osd bluefs_shared_alloc_size (or per-OSD). Leaving 4 KiB is safe but slightly less efficient.
Archive the crash backlog once stable: $CEPH crash archive-all.
Add a %-full alert well below full_ratio (e.g. page at 80 % RAW USED) so capacity is addressed before OSDs wedge.
Capacity: OSDs are raw disks (useAllDevices: true) — expansion means adding storage nodes to the ceph-storage-nodepool via kommodity/terraform, not resizing a PVC.
Restart both mgr pods after any mon regeneration. When the mons are recreated (as on 2026-07-10, when s/r/q → t/u/v), a long-running mgr pod's watch-active sidecar keeps an emptyDir /etc/ceph/ceph.conf written once at pod start, pointing at the dead mon addresses. The mgr daemon is unaffected (it holds the live monmap in memory, so :9283 metrics stay green), but the sidecar's ceph mgr stat hangs every 15 s cycle, so the dashboard Service endpoints never reconcile and the dashboard (:8443) becomes unreachable. Symptom: failed to reconcile services. unable to get active mgr: failed to get mgr stat … timed out every 15 s in watch-active logs, with prev active mgr was (empty) on the active daemon. Fix: kubectl -n rook-ceph delete pod <mgr> — the emptyDir regenerates a fresh ceph.conf from the current rook-ceph-mon-endpoints ConfigMap. This triggers a brief mgr failover (seconds) to the standby; do both mgrs.

ArgoCD notes

OSD/mon/mgr deployments are NOT in any ArgoCD Application — they are created by the rook operator and owned by the CephCluster CR (no argocd.argoproj.io/instance label). Scaling/deleting them for maintenance is not reverted by ArgoCD; the rook-ceph-cluster app shows Degraded (reflecting Ceph health) but that is expected.
The rook operator (app rook-operator, selfHeal: true) coordinates OSDs; the kubectl rook-ceph maintenance plugin is the supported way to take an OSD offline for ceph-bluestore-tool.
Only the PVC/StatefulSet resize (step 2) is genuinely ArgoCD-bound.

Tooling gotchas

The OSD/mgr containers inject -m <mons> via CEPH_ARGS; this breaks ceph daemon … (local admin-socket) commands with unused arguments: ['-m', …]. Prefix with env CEPH_ARGS= for admin-socket calls, keep it for network calls.
ceph -s/df hang from mgr/osd pods when the cluster is full/degraded — run from a mon pod with --name client.admin --key <ceph-secret> -m <mon-ip> as shown above.
rbd/radosgw-admin also stall while the cluster is this sick; get RBD→PVC mapping from kubectl get pv -o json (match spec.csi.volumeHandle's csi-vol-… to claimRef) instead.
A stale ceph.conf can mimic a network/mon outage. CLI calls (ceph -s, ceph mgr stat) hang silently when /etc/ceph/ceph.conf points at mon addresses that no longer exist — the connection hangs rather than erroring. This is per-pod (an emptyDir written at start), so one mgr can be broken while another is fine: compare /etc/ceph/ceph.conf across mgr pods (the mon host line) against the rook-ceph-mon-endpoints ConfigMap. If they disagree, the pod predates a mon regeneration — restart it (see "Restart both mgr pods" above). Same trap bit the watch-active sidecar.