Loki Discarding Log Entries

Runbook for Loki actively discarding log entries, triggered by the Loki Discarding Log Entries alert.

This is the log-data-loss companion to 10-mimir-discarding-samples.md (which covers metric data loss). Both fire when the central synapse-server is throwing away data it should be storing.

Meaning

loki_discarded_samples_total is a Loki distributor counter that increments every time a log entry is rejected. The metric only appears once a discard has happened — absence is healthy. rate(...[5m]) > 0.1 sustained for 5m means Loki is actively losing log data at a meaningful rate (>1 entry every 10s).

The reason label distinguishes the failure mode:

Live examples (2026-10-05)

• prd-par01: too_far_behind discards at ~0.006 entries/s (rounds to "0.0/s" in the alert annotation) caused by stale Kubernetes series events from loki.source.kubernetes_events in synapse-client-alloy-singleton. The mlservices namespace has 6 ml-service-medical-coding-* pods stuck Pending whose FailedScheduling events have eventTime values 4–19h old. The event handler re-sends them every ~30 min; Loki discards them as too_far_behind (older than max_chunk_age = 2h default). Also: edge_function_invocation_logs from prd-supabase occasionally arrive 20–80 min behind wall-clock.
• prd-par02: same too_far_behind pattern from stale K8s events in mlservices.
• engagestaging / try: stale K8s events in redis namespace (events from 2026-10-03).
• dev-par01 (2026-08-26): ~1004 entries/hr discarded with reason=rate_limited — a producer exceeding the per-tenant ingestion rate.

Impact

• Log queries return gaps for the affected streams during the discard window — the log lines are lost, not delayed.
• LogQL alerts scoped to the affected streams do not fire — if the discarded lines contained the error pattern an alert watches for, that alert is blind.
• For rate_limited: all streams from the affected tenant are subject to the rate cap, so the impact is fleet-wide for that tenant, not just one stream.
• For too_far_behind: only entries older than max_chunk_age are dropped; newer entries are ingested normally — the gap is in the historical window.

Diagnosis

The alert labels carry k8s_cluster_name (the synapse-server cluster where Loki runs) and reason. From the alert, note both.

1. Confirm the discard rate

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml

Confirm via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

QUERY='sum by (k8s_cluster_name, reason) (rate(loki_discarded_samples_total[5m]))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[]? | "\(.metric.k8s_cluster_name) \(.metric.reason): \(.value[1]) entries/s"'

A value > 0 confirms the alert. The reason label tells you which failure mode.

2. For rate_limited — check the ingestion rate vs limit

# Current ingestion rate
QUERY='sum by (k8s_cluster_name) (rate(loki_distributor_lines_received_total[5m]))'
# Current bytes rate vs the configured limit (ingestion_rate_mb in runtimeConfig)
QUERY2='sum by (k8s_cluster_name) (rate(loki_distributor_bytes_received_total[5m])) / 1024 / 1024'

Compare the bytes rate against the ingestion_rate_mb override (16MB/s on prd-par01, set in the runtimeConfig). If the rate is at or above the limit, raise the limit (see Mitigation).

3. For too_far_behind — check for stale K8s events, consumer OOM, or external log drains

too_far_behind means entries are arriving with timestamps older than Loki's max_chunk_age (default 2h). There are three distinct scenarios:

3a. Stale Kubernetes series events (most common on prd-par01)

The synapse-client-alloy-singleton pod runs loki.source.kubernetes_events "cluster_events" which polls the Kubernetes API every ~30 minutes and forwards events using the event's eventTime field as the log timestamp. When pods are stuck in Pending or FailedScheduling states for hours, their series events (with count=0 and eventTime set) have timestamps >2h old. Every poll cycle re-sends these stale events, and Loki discards them.

Check for stale events in the affected namespace:

# Find K8s events with eventTime >2h old (series events)
kubectl -n <namespace> get events -o json | jq '[.items[] | select(.eventTime != null) | {eventTime: .eventTime, reason: .reason, object: .involvedObject.kind + "/" + .involvedObject.name, count: .series.count}] | sort_by(.eventTime) | .[0:10]'

Check loki-write logs for the exact streams being discarded:

kubectl -n monitoring-system logs loki-write-0 --since=2h | grep "too far behind" | sed 's/.*for stream //' | sort | uniq -c | sort -rn

Fix: resolve the underlying scheduling issue (stuck pods) or delete the stale events. A longer-term fix is to filter old events in the Alloy cluster_events feature config before forwarding.

3b. alloy-loki-consumer OOMKill replay

The most common cause from the runbook history is the alloy-loki-consumer falling behind on Kafka and replaying old entries on restart:

# Consumer restarts (OOM loop = stale entries on replay)
QUERY='kube_pod_container_status_restarts_total{pod=~"synapse-server-alloy-loki-consumer.*"}'

# Consumer memory vs limit
QUERY2='container_memory_working_set_bytes{pod=~"synapse-server-alloy-loki-consumer.*"} / on(pod) group_left kube_pod_container_resource_limits{pod=~"synapse-server-alloy-loki-consumer.*", resource="memory"}'

If the consumer was OOM-killing, it replays its Kafka offset on restart — entries that were buffered during the outage are now older than max_chunk_age and get discarded. The fix is to prevent the OOM (chart 0.18.1 caps batch size), not to change Loki config.

3c. External log drains with stale timestamps (prd-supabase)

External sources like Supabase Logflare send logs through the alloy-gateway enriched receiver. Some entries (e.g. edge_function_invocation_logs) may arrive with timestamps 20–80 minutes behind wall-clock. At normal ingestion rates this is usually within max_chunk_age, but bursts after a Logflare backlog can exceed it.

Note: max_chunk_age is an ingester-level config and cannot be overridden per-tenant or per-stream. It is set in the Loki limits_config (defaults to 2h) and applies globally to all streams.

4. For line_too_long — identify the chatty producer

# Which streams are producing the most bytes?
QUERY='topk(10, sum by (k8s_cluster_name, service_name) (rate(loki_distributor_bytes_received_total[5m])))'

The stream with the highest byte rate is likely the one emitting long lines. Check the producer's log format — a stack trace or a large JSON payload dumped as a single line will exceed the 256KB default.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix based on the reason

reason=rate_limited:

Raise ingestion_rate_mb (and optionally ingestion_burst_size_mb) in the Loki runtimeConfig overrides. On prd-par01 the current override is ingestion_rate_mb: 16, ingestion_burst_size_mb: 32 (in the deployments overlay). The runtimeConfig is hot-reloaded (Loki watches the file every 10s) — no restart needed.

Before raising, verify the Loki write-path has headroom (check loki_write pod memory vs limit). If the write path is already near its memory limit, raising the ingestion rate will just move the OOM from the distributor (discarding) to the write pod (OOMKilled).

reason=stream_limit:

Raise the per-stream rate limit in the runtimeConfig overrides (max_streams_per_user or the per-stream rate limit). Alternatively, identify the chatty stream (Diagnosis step 4) and reduce its volume at the source.

reason=too_far_behind:

This is a consumer/producer issue, not a Loki config issue. Do NOT raise max_chunk_age — that would let stale data in and bloat the index. max_chunk_age is an ingester-level config and cannot be overridden per-tenant or per-stream. Instead:

• If the stale entries are from loki.source.kubernetes_events (stuck pods with old FailedScheduling events): resolve the scheduling issue or delete the stale events (kubectl -n <ns> delete events --field-selector reason=FailedScheduling). A longer-term fix is to add a timestamp filter in the Alloy cluster_events feature to drop entries >2h old.
• If the alloy-loki-consumer was OOM-killing: fix the consumer (chart 0.18.1 caps batch size; see the synapse-server chart).
• If a producer cluster's synapse-client was down and is replaying: wait for the replay to finish. The stale entries will be discarded, but newer entries will flow normally.
• If the stale entries are from an external log drain (e.g. Supabase Logflare): investigate whether the source can use wall-clock timestamps. If the backlog is temporary, wait for it to drain.
• If the stale entries are from an intentional historical backfill: raise max_chunk_age temporarily and lower it after the backfill completes.

reason=line_too_long:

Raise max_line_size in the Loki limits_config if the long lines are legitimate (e.g. structured JSON payloads). Alternatively, truncate logs at the source (synapse-client's loki.process stage can add a truncate stage to cap line length before sending).

Step 3 — Verify the fix

QUERY='sum by (k8s_cluster_name, reason) (rate(loki_discarded_samples_total[5m]))'

The rate should drop to 0 (or the metric may disappear) once the fix takes effect.

Verify

• The alert clears once rate(loki_discarded_samples_total[5m]) returns below 0.1 (metric may disappear) sustained for the for: 5m window.
• For rate_limited: loki_distributor_bytes_received_total rate is below the new ingestion_rate_mb limit.
• For too_far_behind: the alloy-loki-consumer is running without OOM restarts and Kafka consumer-group lag is near zero.
• Log queries for the affected streams return continuous data (no gaps).

Notes

• Threshold tuned to >0.1 (PLA-7201) — the original > 0 threshold fired on a single discarded entry per 5m window, which is cosmetic data loss (e.g. stale K8s events from stuck pods). > 0.1 (1 entry every 10s sustained for 5m) filters out single-entry noise while still alerting on meaningful data loss: rate-limited ingestion, consumer OOM replays, or a producer flooding Loki with stale entries.
• Metric only appears on failure — loki_discarded_samples_total is absent when healthy. The alert fires when it appears with a positive rate (> 0.1 filter ensures low-volume series from the sum by do not produce false alerts).
• max_chunk_age is global — it is an ingester-level config (default 2h) and cannot be overridden per-tenant or per-stream via runtimeConfig. The runtimeConfig overrides support ingestion_rate_mb, ingestion_burst_size_mb, etc., but not max_chunk_age. Do NOT raise max_chunk_age globally to fix a single chatty producer — that would let stale data into all streams and bloat the index.
• Relationship to Mimir discard runbook — 10-mimir-discarding-samples.md covers metric data loss; this runbook covers log data loss. Both are "the observability pipeline is throwing away data" — they share the same severity and urgency.
• Consumer vs producer — rate_limited and stream_limit are Loki config issues (raise limits). too_far_behind and line_too_long are consumer/producer issues (fix the source) — changing Loki config won't help and may mask the problem.
• k8s_cluster_name not cluster — Loki self-monitoring metrics natively carry k8s_cluster_name; annotations use {{ $labels.k8s_cluster_name }}.