Kafka Consumer Lag in Voice-Vault

A fix for this issue has been deployed Fri Aug 21.

Runbook for investigating and resolving Kafka consumer lag, typically triggered by the System Load dashboard alert (threshold: ≥ 1 000 messages behind).

The most common culprit is the voice-vault consumer in the base tenant namespace. It occurs on any production cluster — most often prod-weu (EU), but the same failure and remediation apply on prod-us (US) and prd-par01 (FR). Substitute the matching cluster, kubectl context, and Kafbat UI alias throughout this runbook.

Meaning

A Kafka consumer group is falling behind the partition end offsets on one or more partitions, meaning messages are not being processed fast enough. When lag crosses the alert threshold it indicates a consumer that is stuck, crashing, or unable to keep up with ingest throughput.

Impact

• Voice recordings queued in Kafka are not being stored to the vault.
• Lag compounds over time if not resolved — hours of backlog can accumulate.
• At extreme lag, partition offsets may advance past the consumer's last committed position, causing message re-processing when the consumer recovers.

Diagnosis

1. Confirm the alert

Open the System Load dashboard in Grafana and locate the Consumer Group Lag panel. Identify:

• Which consumer group has elevated lag (e.g. base-voice-vault)
• Which cluster the consumer is running on (e.g. prod-weu)

2. Inspect the consumer in Kafbat UI

Open Kafbat UI for the affected cluster. Each production cluster has its own public alias following the pattern https://tools.<alias>.corti.app/kafka:

Sign in with your Corti SSO account.

1. Click Consumers in the left sidebar.
2. Sort the Consumer Lag column descending to confirm the lagging consumer.
3. Click the consumer group name (e.g. base-voice-vault).
4. In the consumer detail view, expand the topic row by clicking the + badge.
5. Sort partitions by Consumer Lag descending to identify the stuck partition(s).
6. Note the Partition number and Current Offset of the highest-lag partition.

Mitigation

Step 1 — PIM activation

Activate the required Azure Privileged Identity Management (PIM) roles before touching production:

• MultiTenantEuOnlyAccessPIMRoles (required for production EU clusters)
• DevOpsPimRoles
• Platform Team (privileged)

Activate through the Azure Portal PIM blade → select each group → Activate.

Step 2 — Ensure correct kubectl context

kubectl config current-context          # confirm you are on the affected cluster
kubectl config use-context prod-weu     # switch if needed (prod-weu / prod-us / prd-par01)

Step 3 — Restart the deployment

A restart is often enough to recover a stuck consumer. Try this first before the more invasive steps below.

kubectl rollout restart deployment/voice-vault -n base

Watch the Grafana Consumer Group Lag panel for a few minutes. If lag is dropping, the restart was sufficient — skip to Step 7 (verify).

If lag stays high or keeps increasing, continue.

Step 4 — Disable ArgoCD auto-sync

Prevent ArgoCD from reverting the replica scale-down while you work:

autosync list                    # confirm current auto-sync state
autosync disable infra-apps      # disable for the infra-apps ApplicationSet (otherwise next autosync disable will be reverted)
autosync disable base-apps       # disable for the base ApplicationSet

Step 5 — Scale to zero

kubectl scale deployment voice-vault -n base --replicas=0
# wait for pods to terminate
kubectl get pods -n base -l app=voice-vault -w

Step 6 — Reset the stuck partition offset

Back in Kafbat UI:

1. Navigate to Consumers → base-voice-vault.
2. Confirm the consumer group now has 0 members (pods are down).
3. Click the ⋮ (three-dot menu) in the top-right and select Reset offset.
4. Fill in the form:
   • Topic: base-voice-recordings
   • Reset Type: OFFSET
   • Partitions: select the stuck partition number noted in Step 2
   • Partition #N Offset: enter the Current Offset value noted in Step 2, then add 1
5. Click Reset Offsets.

Step 7 — Scale back up

kubectl scale deployment voice-vault -n base --replicas=2

Watch Kafbat UI — lag should drop to zero within a minute or two as the consumer catches up.

Step 8 — Re-enable ArgoCD auto-sync

autosync enable base-apps
autosync enable infra-apps
autosync list                    # confirm ENABLED

Verify

• Grafana → System Load → Consumer Group Lag: confirm base-voice-vault is back to near-zero.
• Kafbat UI → Consumers → base-voice-vault: confirm all partition lags are ≤ a few messages.
• The alert should clear automatically once lag drops below the 1 000-message threshold.

autosync script reference

The autosync script (installed at ~/.local/bin/autosync) manages ArgoCD ApplicationSet auto-sync via kubectl patch. Copy it to ~/.local/bin/autosync and chmod +x if it is not already present on your machine.

#!/usr/bin/env bash
# Simple CLI for managing ArgoCD ApplicationSet auto-sync
# Usage:
#   autosync list
#   autosync disable <appset-name-pattern>
#   autosync enable <appset-name-pattern>

set -euo pipefail

NAMESPACE="argocd"

usage() {
  echo "Usage:"
  echo "  autosync list"
  echo "  autosync disable <name-pattern>"
  echo "  autosync enable <name-pattern>"
  exit 1
}

list_autosync() {
  echo "Listing ApplicationSets in namespace '${NAMESPACE}' and their auto-sync status:"
  kubectl get applicationsets -n "${NAMESPACE}" -o json \
    | jq -r '.items[] | [.metadata.name,
       (if .spec.template.spec.syncPolicy.automated then "ENABLED" else "DISABLED" end)] | @tsv' \
    | column -t
}

disable_autosync() {
  local pattern="$1"
  echo "Disabling auto-sync for ApplicationSets matching pattern '${pattern}'..."
  for appset in $(kubectl get applicationsets -n "${NAMESPACE}" -o name | grep "$pattern" || true); do
    echo "→ $appset"
    kubectl patch "$appset" -n "${NAMESPACE}" \
      --type='json' \
      -p='[{"op": "remove", "path": "/spec/template/spec/syncPolicy/automated"}]' \
      2>/dev/null || echo "   (already disabled or missing field)"
  done
}

enable_autosync() {
  local pattern="$1"
  echo "Enabling auto-sync for ApplicationSets matching pattern '${pattern}'..."
  for appset in $(kubectl get applicationsets -n "${NAMESPACE}" -o name | grep "$pattern" || true); do
    echo "→ $appset"
    kubectl patch "$appset" -n "${NAMESPACE}" \
      --type='merge' \
      -p='{"spec":{"template":{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true}}}}}}'
  done
}

# --- main ---
if [[ $# -lt 1 ]]; then usage; fi

cmd="$1"
case "$cmd" in
  list)    list_autosync ;;
  disable) [[ $# -eq 2 ]] || usage; disable_autosync "$2" ;;
  enable)  [[ $# -eq 2 ]] || usage; enable_autosync "$2" ;;
  *)       usage ;;
esac