Runbook for when a Mimir (Prometheus-compatible) query returns different series counts on repeated identical queries — e.g. 0, 9, 12, 21 for the same instant query, 2s apart — and Grafana panels flip between data and "No data".

This is not a network/Cilium issue and not a query-frontend/ring problem. It is a half-empty ingester set: some ingesters hold zero series because their Kafka consumer-group offsets were left pointing past the end of the recreated mimir-ingest topic after a Ceph RBD pool rebuild.

Observed on prd-par01 (2026-07-22) after the Ceph EC data pool (pool 1) was destroyed and recreated. The RBD PVC rebuild + this offset fix together make up the recovery; see the storage runbooks for the PVC/RGW side.

Prerequisites

Cluster-admin RBAC, KUBECONFIG pointing at the affected par cluster, e.g. ~/.kube/prd-par01.yaml.
The mimir stack is deployed in monitoring-system (Strimzi Kafka mimir-ingest in the same namespace).
ArgoCD auto-sync for the synapse-server app is off (verify before pausing Strimzi — see Gotchas).

How Mimir ingests here (context you need)

Writes go to the mimir-ingest Kafka topic, not via gRPC to ingesters (push_grpc_method_enabled: false). Distributors produce; ingesters consume.
Each ingester is its own Kafka consumer group, named after the pod, e.g. synapse-server-mimir-ingester-zone-a-1. Its committed offset lives in Kafka's __consumer_offsets topic — not on the ingester PVC.
There are 6 ingesters across 3 zones (a-0/a-1, b-0/b-1, c-0/c-1), RF-3 zone-aware. The mimir-ingest topic has 6 partitions; only partitions 0 and 1 carry data on this fleet.

The failure mode

A Ceph RBD EC data pool is destroyed and recreated (e.g. pool 1 → pool 18). All Mimir ingester PVCs die; they are rebuilt fresh and blank on the new pool.
The mimir-ingest Kafka topic is also recreated, but shorter than before (new topic starts at offset 0; the old one had ~24M records).
Each ingester's committed offset survived (it's in Kafka, not on the PVC). For some ingesters the old committed offset (~24.4M) is now past the end of the recreated topic (~1.04M): LAG = -23,363,181 (negative).
Those ingesters "catch up" instantly (lag 0 because they are beyond the end), read nothing, and sit idle with 0 active series.
The ring still reports 6 ACTIVE / 0 Unhealthy — the lifecycler state recovered, but the data did not. The querier fans out to all 6, gets data from only the ingesters that hold series, and returns a varying subset per query → random results.

The split tracks which partition each ingester is bound to (deterministic per pod identity), so it lands on a clean half (3 of 6), not a random per-query subset.

Gotchas

The ring lies. cortex_ring_members{name="ingester",state="ACTIVE"} 6 and 0 Unhealthy/Leaving/Joining looks healthy. Ring state does not reflect whether an ingester holds data. You must check cortex_ingester_active_series per ingester (see Diagnosis).
Don't chase the network. Memberlist WriteTo failed ... 7946 i/o timeout and Cilium Unsupported L3 protocol drops are red herrings — the drops are IPv6 neighbor-discovery baseline noise on the IPv4 VXLAN, and the memberlist timeouts are occasional (~3/10min) and cannot explain a per-query swing. Direct pod-to-pod probes (7946, 9095) will be clean. Skip them.
Don't reset offsets to --to-earliest first. Replaying the full backlog (~670k records / ~7.5M samples) OOM-kills the ingester (10Gi limit, exit 137). Reset to --to-latest to skip the stale backlog — the gap is already lost (Kafka retention has moved on). If you must replay, raise the memory limit or set GOMEMLIMIT (see Fix).
Negative lag is the signature. kafka-consumer-groups --describe shows LAG as a negative number for the broken ingesters. A consumer parked past the topic end never recovers on its own.

Confirm ArgoCD auto-sync is off before you restart ingesters or pause Strimzi, else ArgoCD reverts manual changes:

kubectl -n argocd get application monitoring-system-synapse-server -o jsonpath='{.spec.syncPolicy.automated.enabled}'
# must be empty / absent (off)

Diagnosis

1. Confirm the symptom against Mimir directly

Query Mimir the way Grafana does (gateway /prometheus, X-Scope-OrgID: corti) and watch the series count swing. Run from a pod that has egress to the gateway (e.g. a mimir-query-frontend pod's network namespace via an ephemeral container):

QFP=$(kubectl -n monitoring-system get pod -l app.kubernetes.io/component=query-frontend \
  --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
kubectl -n monitoring-system debug $QFP --image=curlimages/curl:8.12.0 -- sh -c '
  GW=http://localhost:8080; H="X-Scope-OrgID: corti"
  for i in $(seq 1 10); do
    n=$(curl -s --max-time 8 -H "$H" "$GW/prometheus/api/v1/query?query=up%7Bcluster%3D%22prd-par01%22%7D" | grep -o "\"value\":" | wc -l)
    echo "R$i=$n"; sleep 2
  done'

A swing (e.g. 0,21,0,9,12,0,21) confirms the symptom. Querying a query-frontend directly avoids gateway/nginx artifacts. (Queries from a debug container on the gateway pod, or from pods whose egress NetworkPolicy doesn't allow the gateway port, return HTTP=000 — a test artifact, not a Mimir failure.)

2. Find the empty ingesters (the root cause)

For each ingester, read cortex_ingester_active_series from its own /metrics. Healthy ingesters show ~700k–930k; empty ones omit the metric (= 0). This is the decisive check — the ring looks fine, this does not.

for p in $(kubectl -n monitoring-system get pods -l app.kubernetes.io/component=ingester \
  -o jsonpath='{.items[*].metadata.name}'); do
  dc=$(kubectl -n monitoring-system debug $p --image=curlimages/curl:8.12.0 -- sh -c \
    "curl -s --max-time 5 http://localhost:8080/metrics" 2>&1 | grep -oE "debugger-[a-z0-9]+" | tail -1)
  sleep 3
  echo -n "$p: "
  kubectl -n monitoring-system logs $p -c $dc 2>&1 | grep -E "^cortex_ingester_active_series\{user" | tr "\n" " "
  echo
done

Expected when broken: 3 ingesters print a series count, 3 print nothing.

3. Confirm with Kafka consumer-group lag

The empty ingesters have negative lag — committed offset past the recreated topic end. Run kafka-consumer-groups.sh from any mimir-ingest broker pod:

B=$(kubectl -n monitoring-system get pod -l app.kubernetes.io/name=kafka,strimzi.io/kind=Kafka \
  --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
for cg in $(kubectl -n monitoring-system get pods -l app.kubernetes.io/component=ingester \
  -o jsonpath='{.items[*].metadata.name}'); do
  echo "--- $cg ---"
  kubectl -n monitoring-system exec $B -- sh -c \
    "/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server mimir-ingest-kafka-bootstrap:9092 \
     --describe --group $cg 2>/dev/null" | awk "NR==1 || /mimir-ingest    0 |mimir-ingest    1 /"
done

A broken ingester shows e.g.:

GROUP                                  TOPIC        PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
synapse-server-mimir-ingester-zone-a-1 mimir-ingest 1          24404401        1041220         -23363181

Negative LAG = parked past the end = the ingester will never consume again.

Fix

1. Reset the empty ingesters' offsets to latest

For each ingester that had 0 active series (from Diagnosis step 2), reset its consumer group to the topic's current end. This skips the stale, OOM-causing backlog; the ~hours-old gap is unrecoverable anyway.

B=$(kubectl -n monitoring-system get pod -l app.kubernetes.io/name=kafka,strimzi.io/kind=Kafka \
  --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
for cg in synapse-server-mimir-ingester-zone-a-1 \
          synapse-server-mimir-ingester-zone-b-0 \
          synapse-server-mimir-ingester-zone-b-1; do   # the empty ones
  kubectl -n monitoring-system exec $B -- sh -c \
    "/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server mimir-ingest-kafka-bootstrap:9092 \
     --group $cg --topic mimir-ingest --reset-offsets --to-latest --execute"
done

NEW-OFFSET should land near the topic end (~1.04M on partitions 0/1; 0 on the empty partitions 2–5).

2. Restart the empty ingesters so they pick up the reset offset

for p in synapse-server-mimir-ingester-zone-a-1 \
         synapse-server-mimir-ingester-zone-b-0 \
         synapse-server-mimir-ingester-zone-b-1; do
  kubectl -n monitoring-system delete pod $p
  # wait for 1/1 before the next (ingesters are StatefulSets; keep quorum)
  until kubectl -n monitoring-system get pod $p -o jsonpath='{.status.containerStatuses[0].ready}' | grep -q true; do
    sleep 10
  done
done

Restart one at a time. The ingester ring is RF-3 zone-aware; rolling a single pod at a time keeps queries served by the other zones.

3. Prevent OOM during any backlog replay (if you reset to earliest)

If you chose --to-earliest instead of --to-latest (e.g. to recover recent data), the ingester replays the full backlog and will OOMKill (exit 137) under the 10Gi memory limit. Set GOMEMLIMIT on the ingester StatefulSet containers so the Go GC targets a soft ceiling below the cgroup limit, e.g. GOMEMLIMIT=8GiB against a 10Gi limit. Roll the StatefulSet for it to take effect. Prefer --to-latest unless you specifically need the backlog.

Verification

All 6 ingesters hold data and are ingesting:

for p in $(kubectl -n monitoring-system get pods -l app.kubernetes.io/component=ingester \
  -o jsonpath='{.items[*].metadata.name}'); do
  dc=$(kubectl -n monitoring-system debug $p --image=curlimages/curl:8.12.0 -- sh -c \
    "curl -s --max-time 5 http://localhost:8080/metrics" 2>&1 | grep -oE "debugger-[a-z0-9]+" | tail -1)
  sleep 3
  echo -n "$p: "
  kubectl -n monitoring-system logs $p -c $dc 2>&1 | grep -E "^cortex_ingester_active_series\{user|^cortex_ingester_ingestion_rate_samples_per_second" | tr "\n" " "
  echo
done

Every ingester must show active_series > 0 and a non-zero ingestion rate.

The query is stable. Re-run Diagnosis step 1 — the series count must be constant across all 10 queries (no swing, no zeros). A query returning 21 ten times in a row is fixed; 0,21,0,21 is not.
Lag is zero or small-and-positive for all 6 consumer groups (never negative): re-run Diagnosis step 3.

What not to do

Do not restart all 6 ingesters at once before fixing offsets — the empty ones come back empty (the stale committed offset persists in Kafka) and you just re-trigger the OOM/backlog cycle.
Do not chase Cilium drops or memberlist timeouts — they are unrelated (IPv6 ND baseline noise / occasional gossip hiccups). See Gotchas.
Do not delete the ingester PVCs to "fix" the empty ingesters — the data is blank anyway and the problem is the Kafka offset, not the PVC.

Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild