MimirIngesterReachingSeriesLimit (warning, 80%) — this runbook's alert.
MimirIngesterReachingSeriesLimitCritical (critical, 90%) — the critical tier; same expr, > 0.9. Add a second alert: block in a separate runbook if you want a separate alert name, or lower this runbook's threshold to > 0.9 and change severity to critical.
MimirSeriesLimitRejectingSamples (critical) — the distributor-side limit: series over the global cap are rejected before reaching an ingester, so ingester head series stay flat while clients' samples are silently dropped. expr: sum by (k8s_cluster_name, reason) (rate(cortex_discarded_samples_total{reason=~".*series.*"}[10m])) > 0.

For the post-wedge state (ingestion already stalled), see 05-mimir-ingest-storage-wedged.md.

Meaning

Mimir limits the number of active series per ingester via ingester_limits.max_series (set in the Mimir runtimeConfig). cortex_ingester_memory_series is the current active series count per ingester; cortex_ingester_instance_limits{limit="max_series"} is the configured ceiling. The ratio memory_series / max_series is the utilisation; > 0.8 means the ingester is at 80% of its series ceiling.

The join uses on(pod, namespace, k8s_cluster_name) because multiple clusters run identically-named synapse-server ingester pods (e.g. synapse-server-mimir-ingester-zone-c-0 exists on both dev-par01 and prd-par01 into the same Mimir tenant) — joining on (pod, namespace) alone fails with "duplicate series for the match group".

In ingest-storage mode (Mimir reads from the mimir-ingest Kafka topic, push_grpc_method_enabled: false), a full ingester does not just refuse new series — it wedges: the Kafka reader rejects a new-series record, retries the same offset indefinitely, and stalls ingestion for the entire partition. This is why the series-limit alert exists: catch it at 80% before the wedge at 100%.

The distributor-side limit (limits.max_global_series_per_user) is separate: series over the global cap are rejected at the distributor before reaching any ingester, so ingester head series stay flat. cortex_discarded_samples_total{reason=~".*series.*"} only appears once a discard has happened — absence is healthy. The 2026-08-10 incident saw 8.38M active vs 8M limit, 2.8M samples/24h discarded with reason=per_user_series_limit while ingester series looked fine.

The metrics are scraped from the Mimir ingester/distributor pods by synapse-client and remote-written to the central Mimir — no chart PR is needed.

What changed from the chart

All environments — the chart alert had no environment filter (already environment-agnostic). Kept as-is; these metrics only exist on synapse-server clusters (dev-par01, stg-par01, prd-par01, dev-neu).
Label namespace — the alert carries k8s_cluster_name (not cluster) because that is the label the Mimir self-monitoring metrics natively carry and the join requires it for disambiguation. Annotations use {{ $labels.k8s_cluster_name }}.
Three-in-one — this runbook covers the warning tier (80%). The critical tier (90%) and the distributor-side rejection are noted above; create separate runbooks if you want distinct alert names per tier.

Impact

At 80% the ingester is approaching its ceiling. Without action it hits 100% and wedges (see 05-mimir-ingest-storage-wedged.md).
At 100% the ingester rejects new-series records and the Kafka partition reader stalls — all ingestion for the affected partition halts until the limit is raised.
The distributor-side limit silently drops samples — dashboards go blind for the affected tenant with no ingester-side signal.

Diagnosis

The alert labels carry pod, namespace, k8s_cluster_name. From the alert, note all of them.

1. Confirm the utilisation

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml

Confirm via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

QUERY='cortex_ingester_memory_series{pod="<pod>"} / on(pod, namespace, k8s_cluster_name) cortex_ingester_instance_limits{limit="max_series", pod="<pod>"}'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

A value > 0.8 confirms the alert.

2. Check the distributor-side discard (MimirSeriesLimitRejectingSamples)

QUERY='sum by (k8s_cluster_name, reason) (rate(cortex_discarded_samples_total{reason=~".*series.*"}[10m]))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[]? | "\(.metric.reason): \(.value[1]) samples/s"'

If this returns values > 0, the global per-user series limit is also being hit — raise limits.max_global_series_per_user in addition to ingester_limits.max_series.

3. Is the ingester already wedged?

If cortex_ingest_storage_reader_records_failed_total is present and rising, ingestion has stalled — see 05-mimir-ingest-storage-wedged.md.

QUERY='sum by (pod, namespace) (rate(cortex_ingest_storage_reader_records_failed_total[5m]))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[]? | "\(.metric.pod): \(.value[1]) fails/s"'

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Raise max_series in runtimeConfig

The Mimir runtimeConfig is a ConfigMap mounted into the ingester pods. Raise ingester_limits.max_series so the utilisation drops below 80%. The runtimeConfig is hot-reloaded (Mimir watches the file) — no restart needed.

NS=monitoring-system
kubectl -n $NS get configmap -l app.kubernetes.io/component=ingester -o yaml | grep -A5 max_series
# edit the runtimeConfig ConfigMap (GitOps: the values live in the synapse-server chart / deployments overlay)

Before raising, verify ingester headroom: each series costs ~2.2KB of RAM. Check the ingester memory limit vs current usage:

kubectl -n $NS get pod <pod> -o jsonpath='{.spec.containers[0].resources.limits.memory}{"\n"}'

If the ingester is also near its memory limit, raise the memory limit (and set GOMEMLIMIT at ~85-90% — see 07-kubernetes/11-container-oomkilled.md) first, then raise max_series.

Step 3 — Trim cardinality (the durable fix)

If the series count is driven by high-cardinality labels (e.g. an unbounded request_id label, a per-user metric), trim the cardinality at the source (the service emitting the metric) rather than endlessly raising the limit. Identify the top series contributors via Mimir:

QUERY='topk(10, count by (__name__) (cortex_ingester_memory_series))'

Step 4 — Raise the global per-user limit (if distributor-side rejection)

If Diagnosis step 2 showed cortex_discarded_samples_total{reason=~".*series.*"} > 0, raise limits.max_global_series_per_user in the runtimeConfig. Verify ingester headroom first (the global limit feeds the ingesters).

Verify

The alert clears once memory_series / max_series <= 0.8 sustained for the for: 5m window.
cortex_discarded_samples_total{reason=~".*series.*"} stops increasing (rate drops to 0 / metric disappears).
cortex_ingest_storage_reader_records_failed_total does not appear (no wedge).
Re-run the Mimir query from Diagnosis step 1 to confirm the ratio is under 0.8.

Notes

Metrics already collected — cortex_ingester_memory_series, cortex_ingester_instance_limits, and cortex_discarded_samples_total are scraped from the Mimir pods into Mimir. No chart PR required.
Three alerts, one runbook — the warning tier (80%) is this runbook's alert. The critical tier (90%) and MimirSeriesLimitRejectingSamples are documented here; create separate runbooks if distinct alert names are wanted.
Relationship to the wedge — at 100% the ingester wedges. See 05-mimir-ingest-storage-wedged.md for the post-wedge runbook.
k8s_cluster_name not cluster — Mimir self-monitoring metrics natively carry k8s_cluster_name; the join requires it to disambiguate identically-named pods across clusters. Annotations use {{ $labels.k8s_cluster_name }}.