Kafka Consumer Group Lag High (> 800)

Runbook for any Kafka consumer group falling behind on a topic, triggered by the KafkaConsumerGroupLagHigh alert.

This is the generic companion to the specific 03-kafka/01-voice-vault-consumer-lag.md runbook. That runbook covers one known culprit (the voice-vault consumer) end-to-end with a concrete fix. This alert fires on any consumer group / topic crossing the threshold and does not prescribe a fix — the remediation depends on which consumer and why. Use this runbook to identify the owning team and the failure mode, then reach for the specific runbook or the owning team.

Meaning

kminion_kafka_consumer_group_topic_lag (from kminion, the Kafka exporter) is the per-(consumer group, topic, partition) lag: the gap between the partition's high-water-mark offset and the consumer group's last committed offset, summed across partitions by the sum by (k8s_cluster_name, environment, group_id, topic_name) in the expr. > 800 sustained for 5m means a consumer group is > 800 messages behind the topic end and the gap is not closing.

The alert is scoped to production (matching the Datadog monitor) and groups by k8s_cluster_name, environment, group_id, topic_name so each lagging (consumer, topic) pair is its own alert instance with labels intact. The group_id encodes the tenant and the consumer (e.g. base-voice-vault, medatixx-ml-service-transcription-online, aleris-copilot-audio-bridge-*); the topic_name encodes the tenant and the stream (e.g. base-audio-online, medatixx-transcription).

What changed from the Datadog monitor

• No product label — the Datadog query grouped by product (co-pilot vs mission-control) and the message routed co-pilot to @slack-feed-copilot-alerts / @slack-feed-ai-platform-platform-prod and everything else (mission-control) to @slack-prod-mc-alerts. Mimir has no product label — the product is encoded in the consumer group name (copilot-* = co-pilot; ml-service-*, *-service mission-control topics = mission-control). On-call infers the product from group_id and reroutes. To split routing, create per-product runbooks scoped by group_id=~".*copilot.*" (team: text-generation / speech-recognition) and the rest (team: platform), and retire this one.
• Threshold 800 — matches the Datadog query > 800. The specific voice-vault runbook uses 1000 (its dashboard alert threshold); this generic alert uses the Datadog monitor's 800. Adjust the > 800 in the expr to tighten/loosen.
• Exclusions carried — Datadog excluded !consumergroup:asr, !consumergroup:*audio-bridge*, !consumergroup:*speechmatics-adapter*. These are translated to group_id!="asr", group_id!~".*audio-bridge.*|.*speechmatics-adapter.*". The asr exact match is a no-op in Mimir today (no such group) but kept for parity; audio-bridge and speechmatics-adapter are excluded because they are known-noisy or covered by their own alerting. If a consumer in those groups starts lagging for a real reason, it is not caught here — remove the exclusion to cover it.
• Smoothing — Datadog used avg(last_5m); this alert uses the instant sum with for: 5m (fire only if sustained > 800 for 5m). Functionally equivalent for alerting. To match the rolling average exactly, wrap the sum in avg_over_time(... [5m:1m]) (subquery) — heavier to evaluate, rarely necessary.
• Routing — Datadog routed per-product to separate Slack channels. The Mimir path routes by a single team label → incident.io. This runbook routes to on-call as the generic catch-all (on-call triages and reroutes to the owning team).

Impact

• Messages on the topic are queuing faster than the consumer is processing them — the consumer is stuck, crashing, or unable to keep up with ingest throughput.
• Lag compounds over time if not resolved; hours of backlog can accumulate.
• For a user-facing stream (transcription, facts, audio-online) the downstream product sees stale or missing data while the backlog clears.
• At extreme lag, the consumer may never catch up, or on recovery may re-process messages it already handled (offset skew).

Diagnosis

The alert labels carry k8s_cluster_name, environment, group_id, topic_name. From the alert in Grafana / incident.io, note all of them.

1. Confirm the lag

Open the System Load dashboard in Grafana (the Consumer Group Lag panel), or query Mimir directly (via the Grafana datasource proxy, or port-forward synapse-server-mimir-gateway in monitoring-system with header X-Scope-OrgID: corti):

QUERY='sum by (k8s_cluster_name,environment,group_id,topic_name) (kminion_kafka_consumer_group_topic_lag{k8s_cluster_name="<cluster>",group_id="<group_id>",topic_name="<topic_name>"})'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

A value > 800 confirms the alert. Check whether it is rising (consumer stuck / too slow) or plateauing (steady-state backlog).

2. Inspect the consumer in Kafbat UI

Open Kafbat UI for the affected cluster. Each production cluster has its own public alias:

Sign in with your Corti SSO account (the MFA prompt uses Microsoft Authenticator / Microsoft Live, not Google Authenticator).

1. Click Consumers in the left sidebar.
2. Sort the Consumer Lag column descending to confirm the lagging group_id.
3. Click the consumer group name.
4. Expand the topic row; sort partitions by Consumer Lag descending to find the stuck partition(s) and note the Current Offset.

3. Identify the owning workload and team

The group_id is usually <tenant>-<deployment>-<pod-hash> or <tenant>-<consumer-name>. The consumer is a Deployment/StatefulSet in the <tenant> namespace on the named cluster:

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prod-weu.yaml
# strip the tenant prefix and pod hash to get the deployment name
kubectl -n <tenant> get deploy,sts | grep <consumer-name>
kubectl -n <tenant> get pods -l app=<consumer-name> -o wide

Check pod health and recent history:

kubectl -n <tenant> get pods -l app=<consumer-name> -o jsonpath='{range .items[*]}{.metadata.name} restarts={.status.containerStatuses[0].restartCount} phase={.status.phase}{"\n"}{end}'
kubectl -n <tenant> describe pod <pod> | sed -n '/^Events:/,$p'

The owning team is the service's corti.team label in its chart values (config/services/<service>/base/values.yaml). On-call reroutes to that team.

4. Failure mode (investigate, do not fix here)

This runbook does not prescribe a fix — the remediation depends on which consumer and why. Common patterns to identify:

• Restarting / CrashLoopBackOff → the consumer is crashing on a bad message or bad config. Check the pod logs (kubectl logs ... --previous) for the crash reason. The voice-vault runbook (01-voice-vault-consumer-lag.md) covers the stuck-on-a-bad-message case with an offset-reset procedure.
• No restarts, lag rising → the consumer is running but too slow (throughput < ingest rate). Check for a recent deploy that regressed processing, a downstream dependency that slowed (DB, ASR, LLM), or a partition skew (one partition getting all the traffic). Hand to the owning team.
• No restarts, lag flat at a high value → steady-state backlog; the consumer is keeping up with new messages but not draining the backlog. May be sizing (scale up replicas) or a stuck partition (see the voice-vault offset-reset procedure).
• Consumer pod absent → the Deployment was scaled to 0 or deleted; lag will be the full topic length. Restart / scale up the Deployment.

Hand the identified failure mode to the owning team; apply the specific runbook if one exists (voice-vault is the only one today).

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-voice-vault-consumer-lag.md.

Step 2 — Investigate, do not fix

This alert is intentionally non-prescriptive. Use Diagnosis to identify the owning team and the failure mode, then:

• If it is the voice-vault consumer, follow 03-kafka/01-voice-vault-consumer-lag.md end-to-end (restart → scale-to-zero → reset the stuck partition offset → scale back up).
• For any other consumer, hand to the owning team (from Diagnosis step 3) with the failure-mode finding (Diagnosis step 4). Do not blindly restart or reset offsets — a wrong offset reset can re-process or skip messages.

Step 3 — Verify

• The alert clears once kminion_kafka_consumer_group_topic_lag for the (group_id, topic_name) drops below 800 sustained for the for: 5m window.
• Re-run the Mimir query from Diagnosis step 1 to confirm the lag is falling.
• Kafbat UI → Consumers → the group: partition lags are draining.