Ceph Cluster Warning

Runbook for a Ceph RADOS cluster in HEALTH_WARN, triggered by the Ceph Cluster Warning alert.

This is the generic Ceph health alert. It tells you the cluster is sick; it does not tell you which failure mode. Use Diagnosis to identify it, then reach for the specific runbook: 01-ceph-osd-full-bluefs-wedge.md (OSDs crashlooping / PGs down on a filled, fragmented pool) or 03-ceph-mon-quorum-loss-node-replacement.md (mons lost quorum after node loss).

Meaning

ceph_health_status is emitted by the rook-ceph-mgr exporter (job="rook-ceph-mgr", scraped into Mimir). It is a single integer per cluster:

The threshold is == 1 (HEALTH_WARN). ceph_health_detail in Mimir (a per-check series) carries the per-check names for a richer view.

The expr uses max by (k8s_cluster_name, environment) to collapse the (mgr replica, instance) series to one value per cluster. for: 5m rides out a transient bounce (an OSD restarting clears a transient WARN within a scrape or two). The expr is environment-agnostic; Ceph runs only on the par fleet (dev-par01, stg-par01, prd-par01), so in practice only those three clusters are covered.

Impact

• A HEALTH_ERR cluster is failing, not just degraded. Data may be unavailable (PGs down past the pool's failure tolerance) or the cluster may be unable to reach quorum.
• Every consumer of the RADOS pool is at risk: RBD PVCs (postgres/CNPG, kafka, redis, Mimir), CephFS, and the RGW S3 bucket. Writes may be blocked; reads may fail or return stale data.
• A HEALTH_ERR that coincides with crashlooping OSDs on a filled, fragmented pool is the 01-ceph-osd-full-bluefs-wedge.md failure mode — data-loss-sensitive (do not purge/zap a down OSD before reviving it; the wedge runbook covers this).

Diagnosis

The alert labels carry k8s_cluster_name and environment. From the alert in Grafana / incident.io, note both.

1. Get the health detail and overall status

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=rook-ceph
kubectl -n $NS exec deploy/rook-ceph-tools -- ceph -s             # overall: health, mons, OSDs, PGs
kubectl -n $NS exec deploy/rook-ceph-tools -- ceph health detail  # the failing checks (PG_AVAILABILITY, OSD_DOWN, ...)

If kubectl/ceph from the tools or mgr pod hangs (client sessions stall when the cluster is very sick), use a mon pod with an explicit admin key, as in 01-ceph-osd-full-bluefs-wedge.md Diagnosis.

Confirm the live status via Mimir (via the Grafana datasource proxy, or port-forward synapse-server-mimir-gateway in monitoring-system with header X-Scope-OrgID: corti):

QUERY='max by (k8s_cluster_name,environment) (ceph_health_status{k8s_cluster_name="<cluster>"})'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

A value of 1 confirms HEALTH_WARN. ceph_health_detail in Mimir carries the per-check names (PG_AVAILABILITY, OSD_DOWN, MON_DOWN, ...) for a richer view.

2. Identify the failure mode

The failing checks from ceph health detail map to specific runbooks:

• OSD_DOWN + PG_AVAILABILITY + crashlooping OSDs + high utilization → BlueFS wedge on a filled, fragmented pool. Follow 04-storage/01-ceph-osd-full-bluefs-wedge.md (data-loss-sensitive: revive a down OSD before any purge/zap; EC k=4 m=2 tolerates losing 2 hosts, not 3).
• MON_DOWN / mon quorum loss (often after a node loss) → Follow 04-storage/03-ceph-mon-quorum-loss-node-replacement.md.
• OSD_DOWN without crashloop / without high utilization → a single OSD bounced; check ceph osd df tree for which OSD, its pod events, and the node. Often self-heals; if it does not, investigate the OSD pod (kubectl -n rook-ceph describe pod <osd-pod>).
• PG_AVAILABILITY without OSD_DOWN → PGs are scrubbing/deep-scrubbing or backfilling after a prior event; check ceph pg dump for degraded/recovering PGs and wait for recovery, or investigate a stuck PG.
• SLOW_OPS / SLOW_OPS_HIGH_OPS_BLOCKING → I/O latency; check for a saturated OSD (ceph osd df tree, ceph daemon ... performance_counters), a noisy neighbour, or a disk going bad.
• BLUESTORE_FREE_FRAGMENTATION + nearfull → the pre-wedge warning; see 06-ceph-cluster-storage-utilization-high.md.

3. Check OSD and mon health

kubectl -n $NS exec deploy/rook-ceph-tools -- ceph osd df tree      # per-OSD %use, up/down, host
kubectl -n $NS get pods -l app=rook-ceph-osd -o jsonpath='{range .items[*]}{.metadata.name} restarts={.status.containerStatuses[0].restartCount} phase={.status.phase}{"\n"}{end}'
kubectl -n $NS get pods -l app=rook-ceph-mon -o wide                # mon pods and their nodes

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Follow the matching specific runbook

This alert is intentionally generic. Use Diagnosis step 2 to identify the failure mode and follow the matching runbook:

• OSDs crashlooping / PGs down on a filled pool → 04-storage/01-ceph-osd-full-bluefs-wedge.md.
• Mon quorum loss after node loss → 04-storage/03-ceph-mon-quorum-loss-node-replacement.md.
• High utilization driving the failure → also 04-storage/06-ceph-cluster-storage-utilization-high.md.

Do not take generic remediation actions (restart all OSDs, purge a down OSD, recreate the cluster) — Ceph ERR is data-loss-sensitive and the wrong action can cause permanent EC data loss. Diagnose first.

Step 3 — Verify

• The alert clears once ceph_health_status returns to 0 (HEALTH_OK) sustained for the for: 5m window.
• kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s shows HEALTH_OK, all OSDs up, all PGs active+clean.
• Re-run the Mimir query from Diagnosis step 1 to confirm the status is 0.

Notes

• Metric already collected — ceph_health_status (and ceph_health_detail) are scraped from rook-ceph-mgr into Mimir, so this alert fires as soon as the runbook merges and the Terraform module syncs the PrometheusRule CRD. No chart PR is required.
• Par fleet only — Ceph runs only on dev-par01, stg-par01, prd-par01. The AKS clusters use Azure-managed disks; this alert has no series there and will not fire. The expr is environment-agnostic regardless.
• WARN only, not ERR — the == 1 threshold pages on HEALTH_WARN (a single OSD bounce sets WARN and self-heals). HEALTH_ERR (== 2) is paged by the sibling runbook 07-ceph-cluster-health-error.md. Use ceph_health_detail in a Grafana panel for observability.
• Non-prescriptive — this runbook intentionally does not prescribe a fix. Ceph ERR remediation is failure-mode-specific and data-loss-sensitive; the two concrete procedures we have are the BlueFS wedge (01-...) and mon quorum loss (03-...). Add per-failure-mode runbooks as procedures are proven.
• Relationship to the utilization alert — 06-ceph-cluster-storage-utilization-high.md is the pre-failure warning (filling). This alert is the failure (sick). They often fire together on the wedge failure mode.