Ceph Mon Quorum Loss / Node-Replacement Recovery

Runbook for rook-ceph going HEALTH_ERR with mon quorum lost and/or OSD hosts gone after one or more ceph-storage nodes were replaced on a Scaleway/Kommodity (Gen3) cluster.

First observed 2026-07-21/22 on prd-par01 (PLA-6237 / INC-50, total EC loss of 3 hosts) and again 2026-07-23 on stg-par01 (PLA-6288, 2 hosts lost + a 3rd mid-deletion — recovered with zero data loss). Same root cause both times.

This is a different failure class to 01-ceph-osd-full-bluefs-wedge. That runbook is for a full/fragmented cluster whose OSDs crashloop but whose disks are intact. This runbook is for node/instance replacement, where the OSD/mon data is physically gone with the old instance.

Root cause — MachineHealthCheck remediation wipes machine-bound storage

The ceph-storage nodepool runs OSDs and mons on block volumes declared inline in the CAPI ScalewayMachine (additionalVolumes, no volume-ID ref). Per kommodity/pkg/provider/crds/scaleway/ScalewayMachine.yaml, these volumes are "deleted during instance deletion" — they are lifecycle-bound to the Machine, effectively ephemeral w.r.t. replacement. OSDs also hard-pin to a node via nodeSelector: kubernetes.io/hostname.

The kommodity-cluster chart emits a MachineHealthCheck per nodepool per zone (templates/provider/capi/machinehealthcheck.yaml) when healthCheck.enabled: true. Default: maxUnhealthy 40%, remediate on Ready=Unknown or Ready=False for 300s.

The failure chain:

1. A ceph node has a transient Ready blip (kubelet flap, brief network/etcd instability).
2. MHC declares its Machine unhealthy → CAPI deletes the Machine → Scaleway destroys the instance → the OSD/mon block volumes are deleted with it → fresh empty volumes on the replacement → Rook (useAllDevices: true) builds empty OSDs → that host's data is gone.
3. MHC is Ceph-blind — it counts machines, unaware each holds 1 of 6 EC shards. The per-zone maxUnhealthy 40% breaker does fire (RemediationRestricted) when ≥2 nodes are unhealthy at once, but it does not protect Ceph against a staggered cascade: when nodes flap Ready=Unknown sequentially (each remediated before the next fails) the breaker keeps re-opening and remediates one-at-a-time, walking through the whole zone → total EC loss. Confirmed on stg-par01: 3 sequential ceph-machine replacements in ~30 min despite the 40% breaker.

Trigger: on both incidents the node flaps were caused by control-plane / etcd instability in one zone — the drain errors literally read failed to cordon Node: etcdserver: request timed out. A transient etcd blip → nodes report Ready=Unknown → MHC fires. Also note: a FailedDrainNode error does not stop the deletion — CAPI still reaches NodeVolumesDetached and replaces the machine (the Rook PDB/drain path does not guard against this).

Data-loss math (know this before touching anything)

Data pools are EC 4+2 (k=4, m=2), failureDomain=host — 6 shards per PG, one per host.

Corollary that drives the whole recovery: never let an Nth host go while (N-1) are already gone and PGs haven't rebuilt. On stg-par01, 2 hosts were already gone and a 3rd was mid-deletion — pausing that deletion (below) was the difference between full recovery and permanent loss.

The Kommodity management layer (CAPI) — how it works

The Scaleway/Kommodity clusters are Cluster API (CAPI) clusters. There are two Kubernetes clusters involved and it is essential to know which one you are talking to:

• The management cluster (kubeconfig ~/.kube/kommodity-<env>.yaml, context kommodity) runs the CAPI controllers and holds the declarative model of the workload cluster: Cluster, MachineDeployment, MachineSet, Machine, MachineHealthCheck, and the Scaleway infra objects (ScalewayMachine). The nodes live here as Machine objects, not as Node objects.
• The workload cluster (kubeconfig ~/.kube/<cluster>-admin.yaml) is stg-par01/prd-par01 itself — where Node, Pod, rook-ceph and ceph live.

flowchart TB
  subgraph MGMT["Management cluster — kommodity-ENV.yaml (ctx: kommodity)"]
    C[Cluster<br/>stg-par01]
    MD[MachineDeployment<br/>ceph-storage-nodepool-fr-par-1]
    MS[MachineSet<br/>...-4v5fc]
    MHC[MachineHealthCheck<br/>maxUnhealthy 40%, Ready=Unknown/False 5m]
    M1[Machine 7jnvz]
    M2[Machine 9bj4g]
    M3[Machine trzkc<br/>PHASE=Deleting]
    SM[ScalewayMachine<br/>additionalVolumes: inline,<br/>deleted on instance delete]
    C --> MD --> MS --> M1 & M2 & M3
    MHC -. watches Nodes, marks Machines unhealthy .-> M1 & M2 & M3
    M3 --> SM
  end
  subgraph SCW["Scaleway (IaaS)"]
    INST[instance ac3897d1...<br/>+ block volumes 200Gi/100Gi]
  end
  subgraph WL["Workload cluster — CLUSTER-admin.yaml"]
    N[Node trzkc]
    OSD[rook-ceph OSD/mon pods<br/>hostPath on the block volume]
    N --> OSD
  end
  SM -->|provisions| INST
  INST -->|joins as| N
  MHC -->|unhealthy → delete Machine| M3
  M3 -->|delete → drain Node → detach+DELETE volumes → terminate instance| INST
  INST -. volumes deleted → OSD data gone .-> OSD

Machine deletion lifecycle — and where pause intervenes

When MHC (or anything) sets a deletionTimestamp on a Machine, the CAPI Machine controller runs an ordered, finalizer-gated sequence. A deletionTimestamp cannot be unset — but the cluster.x-k8s.io/paused annotation makes the controller skip reconciliation entirely, freezing the sequence at whatever step it's on:

stateDiagram-v2
  [*] --> Running
  Running --> Deleting: MHC marks unhealthy → Machine deleted
  Deleting --> Draining: cordon + evict pods
  Draining --> VolumesDetaching: drain done (or drainTimeout)
  VolumesDetaching --> InfraDelete: detach volumes
  InfraDelete --> [*]: Scaleway deletes instance + volumes (DATA GONE)

  Draining --> Paused: annotate cluster.x-k8s.io/paused=true
  VolumesDetaching --> Paused
  Paused --> Draining: annotation removed (paused-)
  note right of Paused
    Reconcile halted. Instance + volumes
    stay alive. This is the safe hold.
  end note

Key properties observed in the incidents:

• nodeDrainTimeout: None on the ceph Machines → CAPI drains forever; a Machine can sit in Deleting for days (trzkc: 35h) as long as the drain never completes. That is why pausing works — nothing force-completes it.
• The drain wedges on the Rook OSD PodDisruptionBudget (maxUnavailable 0). But a FailedDrainNode error does not block the sequence — CAPI still advances to volume-detach + instance-delete (confirmed on b5dht). Do not rely on the drain failing to save you; pause explicitly.

What to inspect (management plane)

export KC_MGMT=~/.kube/kommodity-<env>.yaml

# 1. Machine phases — the first thing to check. Running / Deleting / Provisioning.
kubectl --kubeconfig $KC_MGMT -n default get machines.cluster.x-k8s.io -o wide | grep ceph

# 2. Why a Machine is being/was deleted — conditions + owner + drain timeout.
kubectl --kubeconfig $KC_MGMT -n default get machine <name> -o json | python3 -c "
import json,sys; m=json.load(sys.stdin)
print('deletionTimestamp:', m['metadata'].get('deletionTimestamp'))
print('paused:', m['metadata'].get('annotations',{}).get('cluster.x-k8s.io/paused'))
print('nodeDrainTimeout:', m['spec'].get('nodeDrainTimeout'))
print('owner:', [(o['kind'],o['name']) for o in m['metadata'].get('ownerReferences',[])])
[print('cond:', c['type'], c['status'], c.get('reason'),'|',(c.get('message') or '')[:100]) for c in m.get('status',{}).get('conditions',[])]
"
# Look for: OwnerRemediated=True (MHC did it), DrainingSucceeded=False (stuck), Ready=False/UnhealthyNode.

# 3. The MHC — thresholds + current verdict.
kubectl --kubeconfig $KC_MGMT -n default get mhc | grep ceph      # MAXUNHEALTHY / CURRENTHEALTHY
kubectl --kubeconfig $KC_MGMT -n default get mhc <name> -o jsonpath='{.spec.unhealthyConditions}{"\n"}{.status}{"\n"}'

# 4. The timeline — Machine/MHC events carry the trigger (e.g. 'etcdserver: request timed out').
kubectl --kubeconfig $KC_MGMT -n default get events --sort-by=.lastTimestamp \
  -o custom-columns='TIME:.lastTimestamp,OBJ:.involvedObject.name,REASON:.reason,MSG:.message' \
  | grep -iE 'ceph|remediat|drain|unhealthy|VolumesDetached'
#   Reasons to expect: RemediationRestricted, MachineMarkedUnhealthy, FailedDrainNode,
#   NodeVolumesDetached, SuccessfulCreate (= a replacement was made → old volumes gone).

Pause / unpause (the safe hold)

# PAUSE — freeze a Deleting Machine whose node/data must survive (reversible).
kubectl --kubeconfig $KC_MGMT -n default annotate machine <name> \
  cluster.x-k8s.io/paused=true --overwrite
# then on the WORKLOAD cluster the node stops being re-cordoned:
kubectl --kubeconfig ~/.kube/<cluster>-admin.yaml uncordon <node>
# verify it holds (node.spec.unschedulable should stay empty/false):
kubectl --kubeconfig ~/.kube/<cluster>-admin.yaml get node <node> -o jsonpath='{.spec.unschedulable}{"\n"}'

# UNPAUSE — only once Ceph is active+clean and losing this host is safe (≥ min_size).
kubectl --kubeconfig $KC_MGMT -n default annotate machine <name> cluster.x-k8s.io/paused-
#   → deletion resumes → instance+volumes deleted → MachineSet provisions a fresh replacement.

To prevent a Machine from being deleted in the first place (rather than pausing after), annotate the healthy ceph Machines with cluster.x-k8s.io/skip-remediation=true so MHC will not target them. The durable equivalent is healthCheck.enabled: false on the ceph nodepool in GitOps (tracked in PLA-6287).

Access — the kubeconfigs you need

Three planes, three kubeconfigs. You need all three.

# 1. WORKLOAD cluster, full admin — where rook-ceph / ceph itself lives.
#    ~/.kube/<cluster>-admin.yaml may be a CAPI kubeconfig *Secret wrapper*, not a
#    usable kubeconfig. Detect: `kind: Secret` at the top. Extract the real one:
yq -r '.data.value' ~/.kube/<cluster>-admin.yaml | base64 -d > /tmp/<cluster>-real.yaml
# validate: apiVersion/kind should be Config, has current-context + client-cert
export KUBECONFIG=~/.kube/<cluster>-admin.yaml     # (after overwriting with the real one)
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph auth can-i '*' '*'   # -> allow

# 2. WORKLOAD cluster, limited (sel@corti.ai default ~/.kube/<cluster>.yaml).
#    Lacks cluster-scoped node/CephCluster reads on *-par* clusters. Impersonate the
#    owning operator SA when you only have this:
kubectl --as=system:serviceaccount:rook-ceph:rook-ceph-system get nodes

# 3. MANAGEMENT (CAPI) plane — where the Machines / MachineHealthChecks live.
#    ~/.kube/kommodity-<env>.yaml  (context "kommodity"; has cluster.x-k8s.io CRDs).
export KC_MGMT=~/.kube/kommodity-<env>.yaml

# VERIFY you're on the right mgmt plane BEFORE trusting it — it must (a) expose the
# CAPI CRDs and (b) actually know about your cluster's Machines:
kubectl --kubeconfig $KC_MGMT get machines.cluster.x-k8s.io -A | grep <cluster>
#   OK  - lists your cluster's Machines (with a PHASE column) -> correct plane.
#   BAD - "error: the server doesn't have a resource type machines" -> wrong cluster
#         (not a CAPI mgmt plane); or an empty list -> right CRDs but wrong context.
kubectl --kubeconfig $KC_MGMT get machinehealthchecks.cluster.x-k8s.io -A

If the mgmt kubeconfig uses an interactive OIDC exec-plugin, it is unusable headless ("exec plugin cannot support interactive mode") — run it from a human terminal.

Toolbox for ceph commands

ceph from mgr/osd pods hangs while the cluster is this sick. Use the rook-ceph-tools deployment (deploy it if absent — it needs the admin secret wired in or it crashloops on a missing keyring):

NS=rook-ceph
POD=$(kubectl -n $NS get pod -l app=rook-ceph-tools -o jsonpath='{.items[0].metadata.name}')
CEPH="kubectl -n $NS exec $POD -- ceph"
$CEPH -s

If the toolbox is missing/crashlooping, the mon-pod-with-explicit-key pattern from 01-ceph-osd-full-bluefs-wedge also works and does not need a healthy mgr.

When the standing toolbox helps — and when it doesn't (important for this failure class)

The toolbox is deployed permanently on every ceph cluster (rook_ceph_cluster.toolbox.enabled: true, PLA-6288). For a mon-quorum-loss / node-replacement incident specifically, understand its timing:

• Early (quorum lost / no active mgr): mostly useless. Almost every read the toolbox issues (ceph -s, df, osd df, pg dump, osd purge's safe-to-destroy check) is served by the mgr and needs a mon quorum to even connect. With quorum collapsed or no active mgr, these hang rather than error. So in the first phase of this exact incident the standing toolbox does not save you — use the mon-pod-with-explicit-admin-key fallback for the handful of quorum-independent facts you need (mon map, who's in quorum).
• After steps 1–2 (quorum + mgr restored): it becomes the workhorse. Once an mgr is active, the toolbox is the fastest way to drive and verify the rest of the recovery.

Theoretical uses across the recovery, in order:

Rule of thumb for this runbook: the standing toolbox is a verification and remediation tool for phases 3–7, not an initial-triage tool for phase 0–2. If ceph in the toolbox hangs, that is itself a signal (no mgr / no quorum) — don't wait on it; go restore quorum + mgr first, then the toolbox lights up.

Diagnosis

$CEPH -s                        # mon quorum count? mgr active? osd up/in? pgs unknown/down?
$CEPH osd tree                  # which hosts/OSDs are down; reweight 0 = already-out
$CEPH osd df tree               # 0 B on a host = data gone (wiped); real bytes = intact
$CEPH health detail | head -40  # OSD_HOST_DOWN / OSD_ZONE_DOWN / PG_AVAILABILITY

Then classify each down host on the mgmt plane:

kubectl --kubeconfig $KC_MGMT -n default get machines.cluster.x-k8s.io | grep ceph
#  PHASE=Running   + node gone      -> instance already replaced, data WIPED (purge)
#  PHASE=Deleting                   -> deletion IN PROGRESS — this is the live threat (pause it)
#  PHASE=Running   + node Ready     -> healthy, keep

Distinguish wiped (host bucket present in osd tree but 0 B in osd df, node absent from kubectl get nodes) from temporarily-down-but-data-intact (node Ready, OSD pod present, real bytes on disk). Only the former gets purged.

Recovery

Order matters. Stop the bleeding first, then restore control plane (mon → mgr), then OSDs, then data.

0. STOP further node loss (do this before anything else)

a. Disable MHC on the ceph nodepool so it cannot remediate another node mid-recovery. Durable fix is GitOps (healthCheck.enabled: false in clusters/<cluster>/values.yaml → PR; tracked in PLA-6287). For an immediate stopgap on the mgmt plane, annotate the healthy ceph Machines to skip remediation:

kubectl --kubeconfig $KC_MGMT -n default annotate machine <ceph-machine> \
  cluster.x-k8s.io/skip-remediation=true --overwrite

b. Pause any ceph Machine already stuck Deleting whose node must survive. A deletionTimestamp cannot be unset, but pausing halts the deletion reconcile (drain + infra-instance removal) indefinitely and reversibly, keeping the instance — and its OSD data — alive:

kubectl --kubeconfig $KC_MGMT -n default annotate machine <trzkc-machine> \
  cluster.x-k8s.io/paused=true --overwrite
# then, on the workload cluster, uncordon the node — it now stays uncordoned:
kubectl uncordon <node>

Why this is the keystone (stg-par01): the stuck-Deleting Machine had nodeDrainTimeout: None (drain forever). Its drain wedged on the Rook OSD PDB (maxUnavailable 0). Rook's disruption controller saw the persistent cordon, treated it as a node drain, and repeatedly relaxed the OSD PDB to cooperate → the external eviction landed → OSD and mgr pods were force-recreated every ~15s ("the OSDs keep getting terminated" / recurring MGR_DOWN). Pausing the Machine stops the drain reconcile → the churn stops → OSDs rejoin and the cluster heals itself.

1. Restore mon quorum

Pick a mon whose data survived on a healthy node ($CEPH won't work yet — check the mon pods / rook-ceph-mon-endpoints). Preferred: the rook-ceph krew plugin:

kubectl krew install rook-ceph                      # once, client-side
kubectl rook-ceph mons restore-quorum <good-mon>    # e.g. n
# confirmation prompt wants the exact literal string: yes-really-restore

If the plugin is flaky (it churns when the surviving mon is on a CAPI-draining node — add an unschedulable toleration to the mon deployment so it can schedule on a cordoned node, or pause the Machine per step 0b first), fall back to manual monmap surgery. Critical gotcha: the monmap has two sources — fixing only the ConfigMap lets phantom mons reappear:

# The ACTUAL mon list Rook feeds the daemons is in the SECRET, not just the CM:
kubectl -n rook-ceph get secret rook-ceph-config \
  -o jsonpath='{.data.mon_host}' | base64 -d       # + mon_initial_members
# Patch rook-ceph-config (mon_host + mon_initial_members) AND rook-ceph-mon-endpoints
# to list only the surviving mon(s); delete leftover per-mon Services + Deployments.

Once a single-mon base quorum exists, the simplest path to a clean 3-mon quorum is to let the operator rebuild it (keep rook-ceph-operator at replicas=1): it will recreate mons on healthy nodes across both zones. This beats hand-fighting monmap injection for all three.

2. Restore the mgr

ceph -s, df, pg, and osd purge all hang without an active mgr. If both mgr pods are Pending:

• CPU saturation (POP2-4C-16G ceph nodes are easily starved — prd/stg both hit 89-99%). Free CPU by scaling down non-essential workloads on the cluster (e.g. monitoring-system):

  kubectl -n monitoring-system scale deploy,sts --all --replicas=0
  

  If ArgoCD auto-sync is on, it will revert this — disable auto-sync on those apps first, or the scale-down flaps.

• mgr has a required nodeAffinity: workload In [ceph] — it only runs on ceph nodes. If the only ceph node with CPU headroom is cordoned, uncordon it (after step 0) or add an unschedulable toleration to the mgr deployment.

Verify: $CEPH mgr stat → available: true.

3. Purge wiped OSDs + dead host buckets (needs an active mgr)

Only for hosts confirmed wiped (step Diagnosis). osd purge's safe-to-destroy check needs the mgr, so it hangs (exit 124) until step 2 is done.

for id in <wiped-osd-ids>; do $CEPH osd purge $id --yes-i-really-mean-it; done
$CEPH osd crush rm <dead-host-bucket>       # for each wiped host
# delete the orphaned Rook OSD deployments (their pods are stuck Pending, no node):
kubectl -n rook-ceph delete deploy rook-ceph-osd-<id>

4. Rejoin recoverable OSDs

OSDs on surviving nodes that show down but up/in historically often just need a clean restart after the mon surgery (they wedge at a stale osdmap epoch, mon marks them down). Restart them once the cluster is quiet:

kubectl -n rook-ceph rollout restart deploy rook-ceph-osd-<id>
$CEPH osd in <id>       # if an OSD shows autoout/weight 0

Confirm all recoverable OSDs reach up/in and PGs leave unknown/down for active/backfilling.

5. Backfill → active+clean

On an idle non-prod cluster, speed up recovery (revert after):

$CEPH tell 'osd.*' injectargs '--osd-max-backfills=4 --osd-recovery-max-active=8'
$CEPH crash archive-all

Watch objects degraded (%) fall and active+clean PG count climb to the pool total.

6. If a pool is below k (3+ hosts lost) — accept-loss triage

Only when redundancy is genuinely gone (permanent). See the prd-par01 detail in [PLA-6237] and the memory note; summary:

• exactly-4-shard PGs: temporarily ceph osd pool set <pool> min_size 4 → backfill → restore min_size 5.
• unknown PGs with empty acting set (last acting []): ceph osd force-create-pg <pgid> --yes-i-really-mean-it (works here; no-op on incomplete PGs that still have a full acting set).
• whole-pool loss: ceph osd pool delete <pool> <pool> --yes-i-really-really-mean-it, then let the Rook operator recreate the pool empty from the still-present ArgoCD-managed CR — do NOT delete the CephBlockPool/CephObjectStore CR (ArgoCD fights it; delete the underlying ceph pool, keep the CR). For RGW, fix .rgw.root pg_num to satisfy the operator's pg_num_min then restart the operator so it rebuilds the realm/zonegroup/zone + buckets.
• RGW metadata split-brain (stale zonegroup/zone IDs → 301 self-redirect loop): delete the CephObjectStore CR (bypass the dependent-bucket finalizer), purge the stale *.rgw.* metadata pools, re-apply → Rook rebuilds a clean realm. Restart rook-ceph-operator if OBC provisioning stalls afterward (stale provisioner state — was the last blocker on prd-par01).

7. Graceful, deliberate node replacement (after active+clean)

To retire the paused/doomed node cleanly once redundancy is restored:

1. Confirm HEALTH_OK / active+clean at full host count.
2. Unpause the Machine (kubectl --kubeconfig $KC_MGMT annotate machine <m> cluster.x-k8s.io/paused-) → it deletes. Losing 1 of 6 hosts leaves ≥ min_size → PGs stay available, just degraded.
3. The MachineSet provisions a fresh replacement → new empty OSDs → backfill → active+clean.
4. Never replace >1 storage node before the cluster is active+clean.

Notes / gotchas

• cluster.x-k8s.io/paused=true on a Deleting Machine is the safe, reversible way to freeze a doomed node whose data you still need. A deletionTimestamp can never be unset — pause is the only lever.
• The Rook operator (replicas=1) is helpful during recovery (rebuilds mons, manages OSDs). It does not need to be scaled to 0 if you pause the Machine at the CAPI source instead — that stops the drain the operator was cooperating with.
• ceph-storage nodes ship no logs to Datadog (only default/stateful nodepools do), so the node's first-cause NotReady trigger is usually not in DD. The rook operator log (retains ~3 days) is the best local record of the mon-drop / control-plane instability timeline. Kubernetes events age out (~1h TTL) — capture them early.
• OSD/mon/mgr deployments are not in any ArgoCD Application (operator-owned via the CephCluster CR) — scaling/patching them for recovery is not reverted by ArgoCD. Only pool CRs and PVC/STS resizes are ArgoCD-bound.