Welcome Developer Guide Infrastructure Platform Engineering Runbooks Archive

Documentation navigation

Networking

Compute

Data

Operations Manuals

Ceph Cookbook CNPG Runbook AKS Upgrades AKS Cluster Overview Cluster Autoscaler Kubeconfig Resync Deploying a New LLM Model Troubleshooting Talos Control-Plane down in a Private Network

Engineering Guidelines

Platform Standards and Guidelines

Ceph Cookbook

Warning

Always set the kubectl context (kubectl --context <ctx>). This prevents destructive operations from running against the wrong cluster.

Resizing Ceph Nodes

This procedure covers replacing Ceph storage nodes with larger ones (e.g. scaling up disk size). It uses Kommodity to roll the nodepool while Rook handles OSD lifecycle.

Background: What is an OSD?

An OSD (Object Storage Daemon) is the Ceph process responsible for storing data on a single disk. Each disk in the cluster runs one OSD, and each OSD is pinned to the node it was provisioned on (portable=false in host-based clusters). Rook manages OSDs as Kubernetes Deployments with a hardcoded nodeSelector for their host.

When a node is removed, its OSD loses its home — the pod becomes unschedulable and Ceph marks it down. Until the OSD is also marked out, Ceph holds its data in place and does not rebalance to other OSDs.

OSD States

State	Meaning
`up`	The OSD process is running and reachable
`down`	The OSD process is unreachable (pod crashed, node gone, etc.)
`in`	The OSD is part of the CRUSH map — Ceph expects data to be on it
`out`	The OSD has been removed from the CRUSH map — Ceph rebalances its data away to other OSDs
`noout`	A flag set on a failure domain (host) that prevents Ceph from automatically marking its OSDs `out` after the `mon_osd_down_out_interval` timeout (default 10 min). Used during planned maintenance to avoid unnecessary rebalancing.

An OSD is normally up+in (healthy) or transitions through down+in → down+out when it fails. The noout flag interrupts this transition, keeping the OSD in even while down.

Erasure-Coded (EC) Pools: Minimum Chunks for Read and Write

EC pools split each object into k data chunks and m parity chunks across independent failure domains (typically hosts). The total pool size is k + m, and each chunk is placed on a different host.

How `k`, `m`, and `min_size` Relate

Parameter	Meaning	Default
`k`	Number of data chunks — minimum needed to reconstruct an object	Set in EC profile
`m`	Number of parity chunks — number of failures the pool tolerates	Set in EC profile
`size`	Total chunks per object (`k + m`)	`k + m`
`min_size`	Minimum chunks required to accept writes	`k + 1`

Read vs Write Thresholds

Available chunks	Operation	Explanation
`k + m` (all)	Read + Write	Full health, optimal performance
`>= k + 1`	Read + Write	`min_size` met — writes accepted. Degraded but functional.
`k`	Read only	Enough to reconstruct data, but not enough margin to safely accept new writes
`< k`	Unavailable	Cannot reconstruct objects — data loss risk, pool blocks all I/O

Prerequisites

Helm chart configuration

Ensure the following is set in the rook-ceph-cluster Helm values before starting:

rook_ceph_cluster:
  cephClusterSpec:
    removeOSDsIfOutAndSafeToRemove: true

This allows Rook to automatically purge OSD deployments once their data has been safely rebalanced to other OSDs. Without this, stale OSD deployments must be cleaned up manually.

`rook-ceph` kubectl plugin

You will need to install the kubectl plugin for Rook to run Ceph commands:

kubectl krew install rook-ceph

How it works

Rook's disruption management cannot distinguish between a temporary node drain and a permanent node deletion. When a node is deleted, Rook detects the OSD as down and assumes it is a drain — so it sets a noout flag on that failure domain to protect data.

This noout flag prevents Ceph from automatically marking the OSD as out, which in turn prevents data from rebalancing and blocks removeOSDsIfOutAndSafeToRemove from triggering. The result is a stuck loop that requires manual intervention per node.

Procedure

1. Start the nodepool resize in Kommodity

Trigger the node replacement through Kommodity. New nodes will join the cluster and Rook will automatically provision OSDs on them.

Monitor new nodes joining:

kubectl get nodes -l workload=ceph -w

2. For each old node being deleted, unblock the OSD

When Kommodity deletes an old node, Rook will enter the noout loop for the OSD on that node. You will see this in the operator logs:

osd "rook-ceph-osd-X" is down on node "<deleted-node>" and a possible node drain is detected
Set noout on draining Failure Domain: "true"

To break the loop, manually mark the OSD as out. Manual ceph osd out overrides noout:

# Find the OSD ID for the deleted node
kubectl rook-ceph -n rook-ceph ceph osd tree

# Mark the OSD out
kubectl rook-ceph --context <ctx> -n rook-ceph ceph osd out <id>

3. Wait for rebalancing and automatic cleanup

Once the OSD is marked out, Ceph will rebalance the data to the remaining OSDs. Monitor progress:

kubectl rook-ceph -n rook-ceph ceph -s

When all PGs reach active+clean, Rook detects the OSD as safe-to-destroy and automatically:

Purges the OSD from Ceph
Deletes the stale OSD deployment
Clears the noout flag
Unblocks the PodDisruptionBudget for the next node, allowing Kommodity to proceed with the next deletion

Rook processes one node at a time — repeat step 2 for each subsequent node deletion.

4. Verify completion

Once all old nodes have been replaced, confirm the cluster is healthy:

kubectl rook-ceph -n rook-ceph ceph -s
kubectl rook-ceph -n rook-ceph ceph osd tree

All OSDs should be up and in, and cluster health should be HEALTH_OK. If not, see section below.

Cleaning Up Orphaned OSDs After Nodepool Replacement

When a storage nodepool is fully replaced (e.g. migrating from 6j6hc → 2nfsx), the old OSDs and their CRUSH host buckets are not automatically removed from Ceph. Rook removes the OSD pod and deployment, but intentionally leaves the OSD entries in Ceph's OSD map — purging destroys PG history and triggers rebalancing, so it is left as a manual step.

The result is stale OSD entries (down, reweight: 0) and empty CRUSH host buckets in the map, potentially with noout flags set, causing a persistent HEALTH_WARN.

Symptoms

HEALTH_WARN
6 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set

kubectl rook-ceph ceph osd tree
# Shows old hosts with weight 0 and OSDs in "down" state

Orphaned OSDs Cleanup Procedure

Important

This procedure is not optional! Stale host entries in the CRUSH map can block Ceph from properly operating.

1. Identify dead OSDs

kubectl rook-ceph ceph osd tree

Look for hosts with weight: 0 and OSDs with status down. Note all dead OSD IDs.

2. Verify data is fully migrated off dead OSDs

kubectl rook-ceph ceph osd safe-to-destroy <id1> <id2> ...

This should confirm all OSDs are safe to remove. If not, wait for rebalancing to complete (ceph -s will show recovery progress).

3. Purge each dead OSD

for id in <id1> <id2> ...; do
  kubectl rook-ceph --context <ctx> -n rook-ceph ceph osd purge $id --yes-i-really-mean-it
done

This removes the OSD from the auth map and OSD map, but leaves the CRUSH host bucket if it has flags set on it.

4. Check for orphaned CRUSH host buckets

After purging, check if empty CRUSH host buckets remain:

kubectl rook-ceph ceph health detail
# Look for: host <name> has flags noout

kubectl rook-ceph ceph osd tree
# Empty hosts appear with weight 0 and no OSDs

5. Remove orphaned CRUSH host buckets

for host in \
  <old-host-1> \
  <old-host-2>; do
  kubectl rook-ceph --context <ctx> ceph osd crush remove $host
done

Removing the bucket also removes any flags set on it (e.g. noout).

6. Verify cleanup

kubectl rook-ceph ceph -s
kubectl rook-ceph ceph osd tree

Cluster health should return to HEALTH_OK once all orphaned entries are removed and any in-progress PG backfill completes.

Troubleshooting

OSD deployment is stuck Pending after node deletion

The OSD pod has a hardcoded nodeSelector pointing to the deleted node. This is normal — it will be cleaned up automatically once ceph osd out <id> is run and Rook finishes rebalancing.

`noout` loop persists after deleting the OSD deployment

Deleting the deployment is not sufficient — Rook reads failure domains from the Ceph CRUSH map, not from Kubernetes pods. Run ceph osd out <id> to break the loop.

Next node drain is blocked (ALLOWED DISRUPTIONS: 0)

Rook sets all OSD PodDisruptionBudgets to maxUnavailable: 0 when the cluster is degraded. This is intentional — it prevents cascading failures. The drain will unblock automatically once the current OSD's rebalancing completes and PGs return to active+clean.

Cluster remains in `HEALTH_WARN`

Check the cluster status:
```
kubectl rook-ceph ceph status
```
If it is caused by recent OSDs crashes, which are recovered, clean up the crash logs:
```
kubectl rook-ceph ceph crash ls
kubectl rook-ceph ceph crash archive-all
```

`BLUESTORE_SLOW_OP_ALERT` on an OSD

Ceph reports OSD(s) experiencing slow operations in BlueStore when an OSD observed slow IO. The alert is sticky — it stays after the condition clears. Investigate disk health first (smartctl, dmesg, node IO metrics); the alert flags a real event, clearing only hides the symptom.

To clear once the underlying cause is ruled out, restart the OSD:

kubectl -n rook-ceph rollout restart deploy rook-ceph-osd-<id>