Runbook for measuring how much Ceph RGW object-store space a given S3 bucket is consuming — e.g. the Mimir (synapse-server-mimir) and Loki (synapse-server-loki) buckets created by synapse-server's ObjectBucketClaims on the gen3 Paris (Scaleway/Rook-Ceph) clusters.

This is the S3-side view (logical bytes + RGW chunking overhead). It is not the disk-consumption view: the raw RADOS footprint on the pool is higher again, multiplied by the data pool's replication/EC factor. For the RBD PVC view (Mimir ingest-storage, etc.) see 01-ceph-osd-full-bluefs-wedge.md.

Prerequisites

Cluster-admin (or rook-ceph namespace exec) RBAC.
KUBECONFIG pointing at the target par cluster, e.g. ~/.kube/dev-par01.yaml.

There is no rook-ceph-tools toolbox pod on these clusters. Run radosgw-admin from a rook-ceph-rgw-* pod's rgw container instead.

The gotchas (read before copy-pasting)

Empty ceph.conf + no mon SRV. The rgw container's /etc/ceph/ceph.conf is empty and DNS SRV for _ceph-mon._tcp fails: unable to get monitor info from DNS SRV with service name: ceph-mon. You must pass --mon-host and --mon-initial-members on the CLI.
Realm/zone flags are mandatory. Without --rgw-realm / --rgw-zonegroup / --rgw-zone (all ceph-objectstore on this fleet) radosgw-admin bucket list returns [] even though buckets exist and the RGW daemon serves them. The daemon gets these from its own --rgw-* args; the CLI does not inherit them.
Admin key, not the rgw keyring. The pod's /etc/ceph/keyring-store/keyring is [client.rgw.ceph.objectstore.a] — read-only for RGW. Use client.admin (key from the rook-ceph-mon secret) written to a temp keyring file.

Quick start: copy-paste script

Paste this function into your shell, then call it with a kubeconfig path (or with no arg to use the current kubectl context / $KUBECONFIG). It auto-discovers the RGW pod, mon endpoints, and admin key, then prints a table of every bucket:

ceph-bucket-usage() {
  # Usage: ceph-bucket-usage [kubeconfig-path]
  # If no arg given, uses $KUBECONFIG or the current kubectl context.
  local NS=rook-ceph
  [ -n "$1" ] && export KUBECONFIG="$1"

  local RGW EP MONS MEMBERS KEY
  RGW=$(kubectl -n "$NS" get pods -l app=rook-ceph-rgw -o name | head -1)
  [ -z "$RGW" ] && { echo "no rook-ceph-rgw pod found in $NS"; return 1; }
  EP=$(kubectl -n "$NS" get cm rook-ceph-mon-endpoints -o jsonpath='{.data.data}')
  MONS=$(printf '%s' "$EP" | grep -oE '[0-9.]+:[0-9]+' | tr '\n' ',' | sed 's/,$//')
  MEMBERS=$(printf '%s' "$EP" | grep -oE '^[a-z]=|[a-z]=' | tr -d '=' | tr '\n' ',' | sed 's/,$//')
  KEY=$(kubectl -n "$NS" get secret rook-ceph-mon -o jsonpath='{.data.ceph-secret}' | base64 -d)

  local O="--mon-host=$MONS --mon-initial-members=$MEMBERS --keyring=/tmp/admin.keyring --name client.admin --rgw-realm=ceph-objectstore --rgw-zonegroup=ceph-objectstore --rgw-zone=ceph-objectstore"

  printf '%-32s %12s %12s %12s\n' BUCKET LOGICAL_GiB ACTUAL_GiB OBJECTS
  printf '%-32s %12s %12s %12s\n' "------" "----------" "---------" "-------"
  for B in $(
    kubectl -n "$NS" exec "$RGW" -c rgw -- sh -c \
      "printf '[client.admin]\n\tkey = $KEY\n' > /tmp/admin.keyring; radosgw-admin $O bucket list" \
    | python3 -c 'import sys,json; print(" ".join(json.load(sys.stdin)))'
  ); do
    kubectl -n "$NS" exec "$RGW" -c rgw -- sh -c \
      "radosgw-admin $O bucket stats --bucket=$B" \
    | python3 -c 'import sys,json
d=json.load(sys.stdin)
u=d.get("usage",{})
size=sum(v.get("size",0) for v in u.values())
actual=sum(v.get("size_actual",0) for v in u.values())
objs=sum(v.get("num_objects",0) for v in u.values())
name=d.get("bucket","")
print(f"{name:32s} {size/1024**3:12.2f} {actual/1024**3:12.2f} {objs:12,}")'
  done
}

Example output (prd-par01, 2026-07-14):

$ ceph-bucket-usage ~/.kube/prd-par01.yaml
BUCKET                            LOGICAL_GiB   ACTUAL_GiB      OBJECTS
------                             ----------    ---------      -------
synapse-server-mimir                    36.99        36.99           92
fr-voice-vault                           0.01         0.01           15
synapse-server-loki                      8.40         8.85      209,209

To query a single bucket instead, use the lower-level commands in the step-by-step section below.

Step-by-step

The quick-start function above is self-contained. These are the individual commands it runs, broken out for adapting or debugging. They set the shell variables ($NS, $RGW, $K, $RGWOPTS) that the Parsing section below references.

export KUBECONFIG=~/.kube/<cluster>.yaml        # e.g. dev-par01, prd-par01
NS=rook-ceph
RGW=$(kubectl -n $NS get pods -l app=rook-ceph-rgw -o name | head -1)

# Mon endpoints: "t=100.64.x.x:6789,u=100.64.y.y:6789,v=100.64.z.z:6789"
EP=$(kubectl -n $NS get cm rook-ceph-mon-endpoints -o jsonpath='{.data.data}')
MONS=$(printf '%s' "$EP" | grep -oE '[0-9.]+:[0-9]+' | tr '\n' ',' | sed 's/,$//')
MEMBERS=$(printf '%s' "$EP" | grep -oE '^[a-z]=|[a-z]=' | tr -d '=' | tr '\n' ',' | sed 's/,$//')

# Admin key (40 chars, starts with AQ...)
K=$(kubectl -n $NS get secret rook-ceph-mon -o jsonpath='{.data.ceph-secret}' | base64 -d)

RGWOPTS="--mon-host=$MONS --mon-initial-members=$MEMBERS \
  --keyring=/tmp/admin.keyring --name client.admin \
  --rgw-realm=ceph-objectstore --rgw-zonegroup=ceph-objectstore --rgw-zone=ceph-objectstore"

# List every bucket
kubectl -n $NS exec $RGW -c rgw -- sh -c \
  "printf '[client.admin]\n\tkey = $K\n' > /tmp/admin.keyring; radosgw-admin $RGWOPTS bucket list"

# Stats for one bucket
kubectl -n $NS exec $RGW -c rgw -- sh -c \
  "printf '[client.admin]\n\tkey = $K\n' > /tmp/admin.keyring; radosgw-admin $RGWOPTS bucket stats --bucket=synapse-server-mimir"

Parsing `bucket stats` JSON

usage is a map keyed by storage class (rgw.main for object data, rgw.multimeta for multipart metadata). Sum across all categories — do not assume a single key:

kubectl -n $NS exec $RGW -c rgw -- sh -c \
  "printf '[client.admin]\n\tkey = $K\n' > /tmp/admin.keyring; radosgw-admin $RGWOPTS bucket stats --bucket=$BUCKET" \
  | python3 -c '
import sys, json
d = json.load(sys.stdin)
u = d.get("usage", {})
size   = sum(v.get("size", 0)        for v in u.values())
actual = sum(v.get("size_actual", 0) for v in u.values())
objs   = sum(v.get("num_objects", 0) for v in u.values())
print(f"logical size : {size:>14,} bytes  ({size/1024**3:.2f} GiB)")
print(f"actual size  : {actual:>14,} bytes  ({actual/1024**3:.2f} GiB)")
print(f"num_objects  : {objs:>14,}")'

Interpreting the numbers

Field	Meaning
`size` (logical)	Sum of object content-lengths as the S3 client sees them.
`size_actual`	RGW-side accounting after per-object chunking/alignment overhead. Slightly larger than logical; the gap grows with many small objects.
RADOS footprint	Not in `bucket stats`. ≈ `size_actual` × pool replication/EC factor. Use `ceph df` (from a mon pod — see runbook 01) for real disk consumption.

Workload signatures observed 2026-07-14:

Mimir (synapse-server-mimir): few, large objects (compacted TSDB blocks). Low object count (~57 dev / ~104 prd), size ≈ size_actual.
Loki (synapse-server-loki): many small chunks + index files. High object count (~20 k dev / ~207 k prd), noticeable size_actual > size gap.

Bucket → OBC mapping

The bucket names match the ObjectBucketClaim bucketName in charts/charts/synapse-server/values.yaml (objectStore.bucketName and lokiObjectStore.bucketName). The OBC-generated ConfigMap (synapse-server-mimir / synapse-server-loki in the monitoring-system namespace) exposes BUCKET_HOST, BUCKET_PORT, BUCKET_NAME to the workloads — read it to confirm the live bucket name if the chart was renamed:

kubectl -n monitoring-system get cm synapse-server-mimir synapse-server-loki \
  -o jsonpath='{.items[*].data.BUCKET_NAME}{"\n"}'

Query Ceph RGW (S3) Bucket Usage