Fires per failing scrape job when its targets failed ≥95% of their scrapes across the last 6 hours (avg_over_time(up[6h]) < 0.05, grouped by (k8s_cluster_name, environment, job), per-job failing-target count > 0). This is the misconfiguration detector: even a single failing target is a real signal — the memcached-exporter monitors on prd-par01 sat at up == 0 for two months while we assumed the caches were being watched (found and fixed in #3332), and the librechat annotation scrapes fail every minute with no other alert able to see them.

The job label in the fired alert names the misconfigured scrape directly — triage starts from it, one alert per failing job instead of a shapeless cluster-wide count.

This is the rate-independent companion to 07-kubernetes/25-cilium-policy-drops.md. That runbook fires on volume-tiered POLICY_DENIED drop bursts (≥5,000/15m cluster-wide); this one catches the same blocked-scrape decision at any rate — a blocked scraper retrying once a minute never registers on any drop-volume threshold, but up == 0 sees it.

Meaning

Every scrape target — annotation-based (k8s.grafana.com/scrape), PodMonitor/ServiceMonitor-based, and the built-in integrations — produces an up series in the central Mimir. A target whose up averaged below 0.05 across six hours has been failing essentially every scrape for a working day's worth of samples.

The 6-hour window is the deploy-blip guard: a target that fails for a few minutes during rollouts keeps its average high; the comparison then keeps only the sustained-failure series, so a single pod/agent restart never fires this alert.

The alert is deliberately sensitive (> 0, not a count threshold): a misconfigured scrape of one target is still a misconfiguration. When it fires on many targets at once, it lists real infrastructure debt — incident class proven in the wild three times in the last two months:

Allow-listed-port blocked scrapes: the alloy-metrics egress NetworkPolicy only opens known scrape ports; each new metrics port needs a one-line entry. Cases: varehus (PLA-7366), the cilium hubble/agent endpoints (9965/9962), the memcached exporters (9150, PR #3332), librechat annotations (3080 — the NP drops them, and even unblocked the annotation path cannot send the Bearer token the endpoint requires).
Dead-but-discovered workloads: long-gone pods on legacy cluster setups whose annotations/monitors were never removed (cockroachdb, frontend, search-service, template-* on on-prem/customer clusters — failing chronically since monitoring started; plus a systematic descheduler target failing on every cluster on port 10258 despite it being allow-listed, pointing to an app-side cause on that port).
App-side failures on a scrape port that is open: varehus targets on 9090/9091 — the scrapes reach the pod but fail application-side; descheduler on 10258 (allow-listed) fails on every cluster.

Impact

Missing series are blind spots: empty dashboards, alerts built on absent metrics not evaluating, and datapoints silently absent (varehus metrics were out for ~12 days before anyone noticed — PLA-7366).
Priority 4 (info) — not a page. Treat it as a worklist; at enablement time it carries the backlog accumulated across the fleet (~30 clusters have long-failing targets). Each entry stays visible until fixed — that is the point: a misconfiguration hidden between dashboards looks like silence, and silence becomes an outage later.

Diagnosis

The alert already carries the failing job — its count is the number of failing targets for that job. List the failing instances of that job (also useful for jobs with several failing targets):
```
up{job="<job-from-the-alert>"}
avg_over_time(up{job="<job-from-the-alert>"}[6h])
```
Fleet-wide context (optional; which jobs are failing across clusters — the alert is per cluster+job, this shows the pattern):
```
count by (k8s_cluster_name, job) (avg_over_time(up[6h]) < 0.05)
```

Classify the failure — pick the target and check the Cilium drops on a suspect node; then test the pod by hand. Running these on production clusters requires Azure PIM elevation; on dev clusters the standard kubeconfigs work:

export KUBECONFIG=~/.kube/CLUSTER.yaml
# cluster runs Cilium (gen3 Scaleway/Kommodity)? find an agent and watch drops:
kubectl -n kube-system get pods -l k8s-app=cilium -o name | head -1   # take one agent
kubectl -n kube-system exec <agent-pod> -- hubble observe --verdict DROPPED --last 60
# then probe the target directly from inside the target's own pod:
kubectl exec <pod-from-the-target-labels> -- sh -c 'wget -qO- http://localhost:<port>/metrics'

Read the outcome per runbook 07-kubernetes/25-cilium-policy-drops.md (which covers the same signal from the drops side).

Assign the class:

Signal	Class	Fix
hubble shows `Policy denied DROPPED (TCP SYN)` from alloy → target pod:port	scrape port not allow-listed	one-line entry in `networkpolicy.alloy-metrics-egress.yaml` (charts/synapse-client) + chart bump — see PLA-7366, #3330, #3332
target answers 401/403/auth error	metrics endpoint needs Bearer auth the annotation path cannot send	PodMonitor/ServiceMonitor with `bearerTokenSecret` (common-chart support = PLA-7629) — then the annotation source should be removed
target pod gone / no endpoints / connection refused, no drops	dead workload / stale monitor	delete the annotation or monitor config; owner-app chart values in the deployments repo
monitor's service/selector matches nothing while the workload still runs (`up` target flaps as discovery retries)	broken monitor selector	fix the ServiceMonitor/PodMonitor label selector in the deployments values
alive pod, allow-listed port, but probe fails (app-side broken metrics handler)	broken metrics handler	investigate the app — e.g. `descheduler` fails on 10258 on every cluster despite the port being open, app-side

Fix it when it is yours; file a ticket when it is another team's — but either way, do not let it sit at up == 0 for the next two months.

Mitigation

Allow-list gap → chart PR to charts/.../synapse-client/templates/networkpolicy.alloy-metrics-egress.yaml (one port entry + a test guard, chart version bump — orca auto-bumps the deployments tag; recent examples: #3330 for cilium, #3332 for memcached-exporter). The alert clears once the alloy pod picks up the change and 6h/avg_over_time window re-populates with successes — honor the alert's keep_firing_for by re-checking after the sync.
Auth-required endpoint (the PLA-7627/LibreChat pattern): give the app's chart a PodMonitor/ServiceMonitor carrying the token (bearerTokenSecret) and remove the now-misleading k8s.grafana.com/* annotations so the failing annotation target disappears — while the metric still needs the correct config to flow.
Dead workloads/stale monitors: remove the annotation from the deployments values or scale the release back; the target disappears from discovery on the next alloy refresh.
Systematic multi-cluster cases (descheduler 10258, legacy-cluster monitors): file one ticket for the pattern (they cluster by job name), not one per cluster; monitor resolution of the ticket in that runbook.

Verify

count by (k8s_cluster_name) (avg_over_time(up[6h]) < 0.05)
# and per fixed job:
avg_over_time(up{job="<fixed-job>"}[1h])   # back to 1

The count drops by the number of fixed targets per cluster; the alert for the cluster fully clears once no target stays below the 0.05 average for 6 hours (with keep_firing_for: 1h smoothing evaluation flaps).

Notes

Companion alert — 07-kubernetes/25-cilium-policy-drops.md fires on the same blocked-scrape ports from the drops side (cluster-wide volume tier). See its hubble observe diagnosis step to confirm allow-list gaps from the drop side; the present runbook's per-instance probe confirms the up == 0 side.
PLA-7366 — the 12-day silent-scrape outage — the original incident defining this failure class: varehus metrics silently blocked until someone noticed dashboards. This alert exists precisely so that class can never go unnoticed again.
PLA-7629 — the common-chart PodMonitor support — removes the librechat annotation failures, the main known offender at this alert's enablement (18–20 targets on dev-par01).
#3332 — the memcached-exporter :9150 allow-list entry — the blind-spot case this alert is designed to catch: prd-par01's cache monitors dark for ~12 weeks before being found by hand.
k8s_cluster_name not cluster — all by clauses group on k8s_cluster_name, the single cluster label shared with Loki, so dashboards and cross-source queries align.