Runbook for a Mimir ingest-storage reader that has wedged on a failing record. This is the post-failure companion to 04-mimir-ingester-series-limit.md — that runbook is the pre-failure warning (80% of max_series); this one fires once the wedge has begun (a record is rejected and the reader is stuck retrying the same Kafka offset).

Meaning

In ingest-storage mode (push_grpc_method_enabled: false), Mimir ingesters read from the mimir-ingest Kafka topic instead of receiving pushes via gRPC. Each ingester is its own Kafka consumer group, named after the pod. When a record is rejected by a per-instance limit (max_series or max_ingestion_rate), the reader cannot skip the record — it retries the same Kafka offset indefinitely, stalling ingestion for the entire partition.

cortex_ingest_storage_reader_records_failed_total counts the failed records. The metric only appears once a failure has happened — absence is healthy. rate(...[5m]) > 0 sustained for 2m means the reader is actively failing and not recovering — the wedge has begun.

The expr groups by pod, namespace, k8s_cluster_name (the k8s_cluster_name disambiguates identically-named pods across clusters — see 04-mimir-ingester-series-limit.md). The metrics are scraped from the Mimir ingester pods — no chart PR is needed.

What changed from the chart

All environments — the chart alert had no environment filter. Kept as-is; these metrics only exist on synapse-server clusters.
Added k8s_cluster_name to the grouping — the chart grouped by (pod, namespace) only; adding k8s_cluster_name disambiguates across clusters into the same Mimir tenant (the same fix the series-limit alert uses).
Priority 1 — ingestion is stalled; this is a service-down condition for Mimir.

Impact

Ingestion for the affected Kafka partition is completely stalled. New samples on that partition are not consumed — they back up in Kafka.
Dashboards and alerts scoped to the affected tenant/cluster go blind (no new samples ingested).
The wedge does not self-heal: raising the limit unblocks the reader, but the reader does not skip the rejected record on its own.
If left unresolved, the Kafka retention on the mimir-ingest topic expires the backed-up messages — permanent data loss for that window.

Diagnosis

The alert labels carry pod, namespace, k8s_cluster_name. From the alert, note all of them.

1. Confirm the wedge

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml

Confirm via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

QUERY='sum by (pod, namespace, k8s_cluster_name) (rate(cortex_ingest_storage_reader_records_failed_total[5m]))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[]? | "\(.metric.k8s_cluster_name)/\(.metric.pod): \(.value[1]) fails/s"'

A value > 0 confirms the wedge. The failing pod is the wedged ingester.

2. Identify which limit was hit

Check the ingester's series utilisation and ingestion rate:

# series utilisation
QUERY='cortex_ingester_memory_series{pod="<pod>"} / on(pod, namespace, k8s_cluster_name) cortex_ingester_instance_limits{limit="max_series", pod="<pod>"}'
# ingestion rate vs limit
QUERY2='cortex_ingester_ingestion_rate_samples_per_second{pod="<pod>"} / on(pod, namespace, k8s_cluster_name) cortex_ingester_instance_limits{limit="max_ingestion_rate", pod="<pod>"}'

Whichever is at/near 1.0 is the limit that triggered the rejection.

3. Check the Kafka consumer-group lag

The wedged ingester's consumer group will show rising lag on the stuck partition. See 02-mimir-random-query-results-after-storage-rebuild.md for the kafka-consumer-groups.sh procedure (that runbook is the negative-lag inverse; here the lag is positive and rising — the reader is stuck):

B=$(kubectl -n monitoring-system get pod -l app.kubernetes.io/name=kafka,strimzi.io/kind=Kafka --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
kubectl -n monitoring-system exec $B -- sh -c "/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server mimir-ingest-kafka-bootstrap:9092 --describe --group <pod>" | awk 'NR==1 || /mimir-ingest/'

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Raise the hit limit in runtimeConfig

This is the unblock action — do it immediately to stop the data-loss clock (Kafka retention is expiring the backed-up messages).

If max_series was hit (Diagnosis step 2 showed series utilisation at 1.0): raise ingester_limits.max_series. See 04-mimir-ingester-series-limit.md Mitigation step 2 for the procedure. The runtimeConfig is hot-reloaded — the reader resumes on the next retry.
If max_ingestion_rate was hit: raise ingester_limits.max_ingestion_rate in the runtimeConfig.

The runtimeConfig is a ConfigMap mounted into the ingester pods; Mimir hot-reloads it (no restart needed). The values are managed via GitOps in the synapse-server chart / deployments overlay.

Step 3 — Verify the reader un-wedges

After the limit is raised, the reader consumes the previously-rejected record on its next retry and resumes. Confirm:

# failures stop
QUERY='sum by (pod, namespace, k8s_cluster_name) (rate(cortex_ingest_storage_reader_records_failed_total{pod="<pod>"}[5m]))'
# lag drains
kubectl -n monitoring-system exec $B -- sh -c "/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server mimir-ingest-kafka-bootstrap:9092 --describe --group <pod>"

Step 4 — Address the root cause

Raising the limit unblocks the wedge but does not prevent recurrence. Follow 04-mimir-ingester-series-limit.md Mitigation step 3 (trim cardinality) or step 4 (raise the global per-user limit) for the durable fix.

Verify

The alert clears once rate(cortex_ingest_storage_reader_records_failed_total[5m]) returns to 0 (metric may disappear) sustained for the for: 2m window.
Kafka consumer-group lag for the wedged ingester drains to near-zero.
cortex_ingester_memory_series / max_series drops below 0.8 (or ingestion rate drops below the rate limit).
New samples are flowing: cortex_ingester_ingestion_rate_samples_per_second is non-zero.

Notes

Metric only appears on failure — cortex_ingest_storage_reader_records_failed_total is absent when healthy. The alert fires when it appears with a positive rate.
Relationship to series-limit runbook — 04-mimir-ingester-series-limit.md is the pre-failure warning (80%). This runbook is the post-failure (wedge). They share the mimir.ingester group.
Data-loss clock — while the reader is wedged, backed-up messages in mimir-ingest are subject to Kafka retention. Raise the limit fast to minimize the gap.
k8s_cluster_name not cluster — Mimir self-monitoring metrics natively carry k8s_cluster_name; annotations use {{ $labels.k8s_cluster_name }}.