Mimir Discarding Samples (Series Limit)

Runbook for Mimir actively discarding metric samples due to the global series cap, triggered by the Mimir Discarding Samples (Series Limit) alert.

This is the data-loss companion to 04-mimir-ingester-series-limit.md. That runbook is the capacity pre-warning (80% of max_series); this one fires when the cap is hit and new series are being rejected.

Scope: The alert only fires on reason=per_user_series_limit — the global series cap (limits.max_global_series_per_user). Other discard reasons (sample-out-of-order, new-value-for-timestamp, sample-timestamp-too-old) are caused by producer clock skew or WAL replay and are not alerted — they are a producer-side issue, not a Mimir capacity problem. They are still visible in the cortex_discarded_samples_total metric if you want to investigate them manually (see Notes).

Meaning

cortex_discarded_samples_total is a Mimir distributor/ingester counter that increments every time a sample is rejected. The metric only appears once a discard has happened — absence is healthy.

The reason label distinguishes the failure mode:

Why only per_user_series_limit is alerted

per_user_series_limit is the only reason that causes permanent, ongoing data loss — once the cap is hit, every new series is rejected until the limit is raised, and the affected dashboards/alerts go blind permanently. The timestamp/ordering reasons are intermittent — only the out-of-order/stale samples are dropped; subsequent in-order samples are ingested normally. They are caused by producer clock skew or WAL replay, not a Mimir capacity problem, so alerting on them would page oncall for a producer issue that Mimir config cannot fix.

Live context (2026-08-26)

• prd-par01: ~2070 samples/hr discarded with reason=sample-out-of-order — a producer cluster is remote-writing samples with clock skew or replaying a backlog out of order. Not alerted.
• dev-par01: ~284 samples/hr discarded with reason=sample-out-of-order. Not alerted.
• per_user_series_limit: not currently active on any cluster — the series-limit pre-warning (04-mimir-ingester-series-limit.md) catches it at 80% before the cap is hit.

Impact

• Dashboards go blind for the rejected series — the data is lost, not delayed.
• Alerts scoped to the rejected series do not fire — if the discarded samples were the only signal for an alert, that alert is blind.
• The series itself is rejected, so ALL future samples for that series are discarded until the limit is raised — the blindness is permanent for that series until the cap is increased.

Diagnosis

The alert labels carry k8s_cluster_name (the synapse-server cluster where Mimir runs) and reason (always per_user_series_limit for this alert). From the alert, note the cluster.

1. Confirm the discard rate

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml

Confirm via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

QUERY='sum by (k8s_cluster_name, reason) (rate(cortex_discarded_samples_total{reason="per_user_series_limit"}[5m]))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[]? | "\(.metric.k8s_cluster_name) \(.metric.reason): \(.value[1]) samples/s"'

A value > 0 confirms the alert.

2. Check the series count

QUERY='sum by (k8s_cluster_name) (cortex_ingester_memory_series)'
QUERY2='topk(10, count by (__name__) (cortex_ingester_memory_series))'

See 04-mimir-ingester-series-limit.md for the full diagnosis and mitigation.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Raise the series limit

Raise limits.max_global_series_per_user in the Mimir runtimeConfig. See 04-mimir-ingester-series-limit.md Mitigation step 4. The runtimeConfig is hot-reloaded — no restart needed. Verify ingester headroom first (RAM ~2.2KB/series).

Step 3 — Verify the fix

QUERY='sum by (k8s_cluster_name, reason) (rate(cortex_discarded_samples_total{reason="per_user_series_limit"}[5m]))'

The rate should drop to 0 (or the metric may disappear) once the fix takes effect.

Verify

• The alert clears once rate(cortex_discarded_samples_total{reason="per_user_series_limit"}[5m]) returns to 0 (metric may disappear) sustained for the for: 5m window.
• cortex_ingester_memory_series is below the max_global_series_per_user cap.
• Dashboards for the rejected series repopulate with new samples.

Notes

• Metric only appears on failure — cortex_discarded_samples_total is absent when healthy. The alert fires when it appears with reason=per_user_series_limit and a positive rate.
• Relationship to series-limit runbook — 04-mimir-ingester-series-limit.md is the capacity pre-warning (80% of max_series). This runbook is the active data-loss alert (series cap hit). When this fires, the fix is the same — this runbook is the "it's already happening" signal.
• Unalerted reasons — sample-out-of-order, new-value-for-timestamp, sample-timestamp-too-old are NOT alerted. They are caused by producer clock skew or WAL replay and are a producer-side issue. They are visible in the cortex_discarded_samples_total metric (filter by reason) if you want to investigate them manually. The steady-state out-of-order discards on prd-par01/dev-par01 are low-rate and by-design — Mimir rejects out-of-order samples from producers with clock skew.
• k8s_cluster_name not cluster — Mimir self-monitoring metrics natively carry k8s_cluster_name; annotations use {{ $labels.k8s_cluster_name }}.