Runbook for the Azure Audit Ingestion Silent alert, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts (LogQL → Loki ruler → Mimir Alertmanager → incident.io).

Meaning

Two independent absence checks, one per pipeline, joined with or — each fires only when its own pipeline is fully silent:

absent_over_time({service_name="azure-audit"}[15m]) — the audit pipeline (Entra AuditLogs/SignInLogs, subscription activity logs via the corti-audit-logs hub).
absent_over_time({service_name="azure-diagnostics", source="AuditEvent"}[15m]) — the diagnostics pipeline, probed via AuditEvent (KeyVault audit events) as a cheap low-volume proxy for its overall health. A single combined selector would be blind to one pipeline stopping while the other flows, and would scan ~766k lines per ruler evaluation; this shape scans ~20k.

Both streams are continuous under normal load (audit ≈ hundreds of lines per 5 minutes, AuditEvent ≈ tens of thousands per 15 minutes), so a full 15-minute silence means an outage, not quiet traffic.

Impact

These pipelines are the only audit paths: platform diagnostics, kube-audit-admin, KeyVault/storage audit events, Defender for Cloud assessments, and Entra sign-in/audit records all flow exclusively through:

Azure Event Hubs (corti-diagnostics-*, corti-audit-logs)
  → alloy-azure-diagnostics-source / alloy-azure-audit-source (prd-par01, ONE replica each)
    → loki-logs Kafka topic (mimir-ingest cluster)
      → alloy-loki-consumer
        → Loki (1-year tiered retention)

The outage is silent by construction: a stopped consumer produces no error logs anywhere else, pods stay Running (the historic alloy-wedge class — see 05-observability/01-alloy-operator-wedge.md), and Azure keeps accepting events into the hubs. Recovery windows on the hub partition differ by pipeline: the corti-diagnostics-* hubs retain messages for 7 days, the corti-audit-logs hub for only 1 day — an audit-pipeline outage over a weekend loses the data that was still buffered at the point the offset resumed.

This alert is the only detection.

Diagnosis

Work top-down: Loki → consumer → Kafka → Azure. Each step has a cheap discriminator.

1. Which pipeline is silent — the alert label already says (azure-audit vs azure-diagnostics/AuditEvent). For diagnostics, also confirm whether the whole pipeline or just the probe is quiet:

absent_over_time({service_name="azure-diagnostics"} |= "AuditEvent" [15m])
sum(count_over_time({service_name="azure-diagnostics"}[5m]))

2. Consumer pods on prd-par01 (monitoring-system, one replica each, by design — see Why it happens):

kubectl -n monitoring-system get pods -l app.kubernetes.io/name=alloy-azure-diagnostics-source -o wide
kubectl -n monitoring-system get pods -l app.kubernetes.io/name=alloy-azure-audit-source -o wide
kubectl -n monitoring-system logs deploy/synapse-server-alloy-azure-diagnostics-source --tail=50
kubectl -n monitoring-system logs deploy/synapse-server-alloy-azure-audit-source --tail=50

Pod Running + 0 restarts + no errors, but no data → suspect a frozen config (operator wedge) or a wedged internal pipeline: check the AlloyOperatorReconcileErrors alert and 05-observability/01-alloy-operator-wedge.md first.
Restart counts climbing → read the crash reason (OOM and MESSAGE_TOO_LARGE are the known classes; the batch caps added in corticph/charts#3302 prevent the latter — verify the running chart still carries them).

3. Kafka in the middle — if the sources produce but Loki is empty, the loki-logs topic consumer is the victim. The synapse Kafka is not covered by kminion; check lag from a broker pod of the mimir-ingest Strimzi cluster:

kubectl -n monitoring-system exec mimir-ingest-kafka-0 -- \
  bin/kafka-consumer-groups.sh --bootstrap-server mimir-ingest-kafka-bootstrap:9092 --list | grep -i loki_logs
# then describe the listed group (the loki-consumer does not pin a group id in config):
kubectl -n monitoring-system exec mimir-ingest-kafka-0 -- \
  bin/kafka-consumer-groups.sh --bootstrap-server mimir-ingest-kafka-bootstrap:9092 --describe --group <group-from-list>

Lag growing → alloy-loki-consumer is down/wedged (that is the affected component, not the Azure sources).

4. The Azure side — distinguish "nothing was ever sent" from "sent but not ingested":

SUBID=2849fdec-c143-421e-81e2-5a4b637a3343
for ns in corti-diagnostics-northeurope corti-diagnostics-westeurope corti-diagnostics-eastus \
          corti-diagnostics-eastus2 corti-diagnostics-westus3 corti-diagnostics-australiaeast \
          corti-diagnostics-swedencentral corti-diagnostics-switzerlandnorth corti-audit-logs; do
  echo -n "$ns: "
  az monitor metrics list --subscription "Infrastructure" \
    --resource "/subscriptions/$SUBID/resourceGroups/$ns/providers/Microsoft.EventHub/namespaces/$ns" \
    --metric IncomingMessages --offset 1h --interval PT5M \
    --query "sum([].timeseries[].data[].total)" -o tsv 2>/dev/null
done

IncomingMessages ≈ 0 across all namespaces → Azure-side problem (diagnostic settings / Defender export stopped — see PLA-6271 runbooks for the repoint). Incoming normal + Loki empty → the outage is in the synapse pipeline (steps 2–3).

Remediation

Restart the affected consumer deployment (a restart resumes from the committed consumer-group offset — messages still in the hub partition are picked up from where the group stalled):

kubectl -n monitoring-system rollout restart deploy/synapse-server-alloy-azure-diagnostics-source
kubectl -n monitoring-system rollout restart deploy/synapse-server-alloy-azure-audit-source

If the operator wedge is confirmed (step 2/AlloyOperatorReconcileErrors), run the re-adopt repair in 05-observability/01-alloy-operator-wedge.md — a plain pod restart does not unfreeze a wedged config.
If alloy-loki-consumer is the victim:

kubectl -n monitoring-system rollout restart deploy/synapse-server-alloy-loki-consumer

and confirm the loki-logs lag (step 3) drains. 4. After recovery, verify the offset did not skip: compare {service_name=~"azure-(diagnostics|audit)"} volume before/after the gap in Loki and confirm the hub's IncomingMessages over the outage window roughly matches what arrived late. For the audit pipeline, remember the 1-day hub retention bounds what a resumed offset can still recover.

Why it happens / prevention

Single replica per consumer is a deliberate design, not an oversight: the regional Event Hubs use one shared consumer group (alloy-azure-diagnostics-source / alloy-azure-audit-source), and a second replica would partition-split the stream between instances (the PLA-6271 consumer-split incident). The cost is that one wedged pod stops the pipeline — this alert is the compensating control.
Known failure classes to date: the alloy-operator Helm-secret wedge (frozen config, 2026-06-30 — runbook 01), Kafka MESSAGE_TOO_LARGE data loss (fixed by the chart's batch caps, corticph/charts#3302), and admission-webhook reconcile noise (runbook 01, Variant B). This alert catches the ingestion symptom of any of them within ~25 minutes of silence.
The alert window (15m absence + 10m for) is sized against the lowest-volume legitimate streams — Entra AuditLogs/SignInLogs and KeyVault AuditEvent arrive continuously, so normal operations never approach the threshold.
Known blind spot, accepted for now: the diagnostics check probes overall (or AuditEvent) volume, so a single regional hub stopping (say corti-diagnostics-eastus) is invisible while the other seven keep the stream alive — Loki carries no region label. A per-region alert (Azure IncomingMessages per namespace vs Loki volume) is the follow-up if regional coverage matters; raise it in PLA-6271 follow-ups before relying on this alert for regional outages.