Fires when a cluster accumulates a large number of policy-denied packet drops in 15 minutes. Cilium drops a packet when no NetworkPolicy or CiliumNetworkPolicy rule allows it; a sustained drop rate means a workload repeatedly attempts a connection that policy forbids.

This is the volume signal of the alert pair shipped under PLA-7600. Its companion, 05-observability/18-scrape-targets-failing.md, fires on the scrape side of the same blocked decision — when both fire, check the companion first: it names the failing job directly via its job label, and its diagnosis covers the same blocked-scrape ports this runbook's hubble observe step confirms from the drop side.

Scope: only reason="POLICY_DENIED" is counted. The UNSUPPORTED_L3_PROTOCOL (ICMPv6 multicast/router noise) drops visible in hubble are excluded from the alert on purpose — they are a benign L3 baseline, not a blocked-connection class.

Meaning

hubble_drop_total{reason="POLICY_DENIED"} counts packets the Cilium agents dropped because a NetworkPolicy/CiliumNetworkPolicy verdict denied them (metric collected by the cilium-hubble PodMonitor, synapse-client >= 0.13.0). A burst means a connection loop: a source pod wants a destination that no policy allows. The metric does not carry source/destination labels — hubble identify them via hubble, then fix the policy or silence the offender. Known classes:

An app phoning home to an internet destination we do not allow — Loki's built-in anonymous usage reporting posted to stats.grafana.org:443 continuously until synapse-server 0.24.4 disabled it (charts#3327); every retry was one rejected SYN — 5,343 drops in 7 minutes across 7 agents when measured. The egress policy worked as designed; the right fix was to stop the analytics, not to open the firewall.
A workload blocked from something it actually needs — the incident class behind PLA-7364 (operator egress blocked kube-apiserver), PLA-7531 (NetworkPolicy regression blocked Deployment writes), PLA-7271 (JWKS egress blocked, every token rejected), PLA-7366 (alloy blocked scraping varehus metrics — silently for ~12 days).
A blocked polling/scaling loop — 2026-10-23: KEDA's prometheus-server on dev-par02 polls mlservices/asr-aed-mock-predictor:80, whose ingress policy does not allow the keda namespace — ~66k denied packets/hour.

The threshold (5,000 packets / 15m cluster-wide) sits ~2.5× above the worst known-benign background (~2,000/15m — still-active librechat scrape annotations pending PLA-7629, plus a leftover redis-test retry loop) and well below the smallest real incident measured to date (~11,000/15m for the blocked loki usage-stats loop); the KEDA and prd-par02 loops run at 24,000–64,000/15m, quiet clusters at ~300/15m.

If a cluster's benign baseline approaches the threshold, lower the baseline by removing the offender (that is how PLA-7629 removes the librechat annotation noise, the largest single benign source) — do not raise the threshold. A threshold that tolerates known noise also tolerates small real incidents.

Impact

Blocked-dependency incidents page fast: the PLA-7364 class blocks deployments or token validation in seconds — and at enablement time this alert is expected to fire on prd-par01/prd-par02/stg-par02, which are currently at 28k–326k drops/hour untriaged.
Tolerated retry loops cost more than they look: 50–200 rejected SYNs/sec per pod (loki usage reporter, KEDA poller) burns Cilium CPU, fills the Hubble ring buffer (hubble_lost_events_total rises), and buries real signals in noise. Do not leave this alert as wallpaper — triage the same day; hubble makes identification a 5-minute job.

Diagnosis

The fired alert carries only k8s_cluster_name and environment — these two steps narrow it down entirely from Grafana before touching the cluster.

Fast triage — Mimir queries only, no cluster access (substitute CLUSTER with the alert's $labels.k8s_cluster_name):

# INGRESS vs EGRESS: EGRESS-dominant = a local pod is being blocked from
# reaching something (phoning home, missing egress rule); INGRESS-dominant =
# something is being blocked from reaching a local pod (missing ingress rule).
topk(6, sum by (direction, reason) (increase(cilium_drop_count_total{reason="Policy denied", k8s_cluster_name="CLUSTER"}[15m])))
# busiest agents (each agent pod = its node) for the hubble step
topk(5, sum by (pod) (increase(hubble_drop_total{reason="POLICY_DENIED", k8s_cluster_name="CLUSTER"}[15m])))
# failing scrape jobs on the same cluster — a blocked alloy scrape-port appears here as the failing job's name
count by (job) (avg_over_time(up{k8s_cluster_name="CLUSTER"}[6h]) < 0.05)

Direction split from the triage that motivated this runbook (dev-par02, 2026-10-23, the KEDA case): INGRESS=21,121 / EGRESS=1,650 per 15m — INGRESS-dominant, resolving the ambiguity of which side's policy blocked the flow. If a failing scrape job (varehus-class up == 0) correlates with the burst, see sibling 05-observability/18-scrape-targets-failing.md first — both signals are firing on the same blocked decision.

App-log fingerprints — Loki, still no cluster access — blocked dials usually appear in the app's own logs, which carry both the source pod identity and the blocked destination:
```
{k8s_cluster_name="CLUSTER"} |~ "(?i)(connection refused|no route to host|context deadline exceeded|dial tcp)" | json | keep level, message | level=~`(?i).*(warn|error).*`
```
Proven example (PLA-7593 round-1): failed to send usage report lines were what identified Loki itself as the stats.grafana.org:443 phoner — its own error logs named the blocked destination.
hubble observe — the definitive signal, source/destination identity from the flow records. Running it on production clusters requires Azure PIM elevation; on dev clusters the standard kubeconfigs work.
```
export KUBECONFIG=~/.kube/CLUSTER.yaml
AGENT=$(kubectl -n kube-system get pods -l k8s-app=cilium -o name | head -1 | cut -d/ -f2)
kubectl -n kube-system exec $AGENT -- hubble observe --verdict DROPPED --last 50
```
Each drop line shows namespace/pod:port (ID:x) -> destination — the offending source/destination pair repeats near-always as Policy denied DROPPED (TCP Flags: SYN).

Tally by source (if no single burst jumps out, run on 2-3 agents):

kubectl -n kube-system exec $AGENT -- sh -c \
  "hubble observe --verdict DROPPED --last 100 | grep -oE '[a-z0-9-]+/[a-z0-9-]+:[0-9]+' | sort | uniq -c | sort -rn | head"

Classify the workloads in the pair — the source pod's chart owner and the destination's role determine the intent:

Pattern	Class	Fix location
any app → internet:443 (stats.* / telemetry endpoints)	analytics phoning home	disable the analytics in that app's chart (Loki: `analytics.reporting_enabled: false`; others: vendor opt-out flag)
app → kube-apiserver / DB / API it depends on	missing allow-rule	the app's NetworkPolicy chart
`synapse-client-alloy-metrics-* / cilium PodMonitor` → a workload port	scrape-port allowlist gap (PLA-7366 class)	`networkpolicy.alloy-metrics-egress.yaml` entry in charts/synapse-client
workload → kube-dns / cluster service	broken policy selector	the CNP/NetworkPolicy's namespace/label selector in the deployments values

Related panels: https://synapse.corti.app/d/cilium-network-monitoring — "Hubble Drop Rate by Reason" shows the burst; "Top Nodes" localizes the agent.

Mitigation

Connection is intended but blocked — add the missing policy rule:
- Workload workload rules: owner app's NetworkPolicy/CNP in the deployments repo.
- Alloy scrape rules: shared synapse-client chart allow-list in charts/cortex-nav/…/networkpolicy.alloy-metrics-egress.yaml (see PLA-7366 precedent) — chart PR + version bump; the orca flow auto-bumps config.json.
Connection should not exist (analytics/telemetry) — disable the reporting in the app's chart values; most vendors ship an opt-out. Loki is already fixed (charts#3327); if another app phones home, apply the same pattern: config or CLI flag, not a firewall hole.
Stale/broken workload (dead scaler targeting a decommissioned endpoint, leftover test pods like redis-test on dev-par01) — delete or stop the workload; that clears the drops immediately and the fix belongs to the workload owner.

Verify

# rates back under threshold (Grafana / any Prometheus query):
topk(10, sum by (k8s_cluster_name) (increase(hubble_drop_total{reason="POLICY_DENIED"}[15m])))
# busiest agents (each agent pod = its node) for the hubble step:
topk(10, sum by (k8s_cluster_name, pod) (increase(hubble_drop_total{reason="POLICY_DENIED"}[15m])))
# no new denied SYNs from the fixed source:
kubectl -n kube-system exec $AGENT -- hubble observe --verdict DROPPED --since 5m

The alert clears 15 minutes after the last matching drop. The dashboard trend should stay below ~2,000/15m cluster-wide.

Notes

Companion alert — 05-observability/18-scrape-targets-failing.md fires on the same blocked-scrape ports from the up == 0 side (any rate, including a single failing target). A blocked scraper retrying once a minute never reaches any volume threshold — the companion covers that, this runbook the sustained blocked-connection loops above it (PLA-7366 vs the loki/KEDA loops that motivated the 5,000 threshold).
k8s_cluster_name not cluster — the alert and all sum by/topk by clauses use k8s_cluster_name, the single cluster label shared with Loki so dashboards and cross-source queries align. The scrape-target companion groups by k8s_cluster_name, environment, job.
Cluster-level aggregation is deliberate — per-agent alert instances would fragment one incident across many series and lose discrimination: benign annotation noise and real blocked loops produce similar per-agent rates (~200–1,600/15m per node); only the cluster sum separates noise (~2,000/15m) from incidents (≥11,000/15m). The Verify section carries the per-agent topk query that localizes the busiest node during triage.
Threshold discipline — if a cluster's benign baseline approaches 5,000/15m, lower the baseline by removing the offender (PLA-7629 will remove the librechat-annotation noise, the largest known benign source), never raise the threshold. A threshold that tolerates known noise also tolerates small real incidents.