Runbook for a kyverno-admission-controller replica that cannot reach the Kubernetes API server, triggered by the frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts (LogQL → Loki ruler → Mimir Alertmanager → incident.io).

Meaning

Counts Kyverno admission-controller log lines containing dial tcp 10.0.0.1:443 (the kubernetes.default.svc ClusterIP) in platform-system over a 10m window. More than 100 occurrences sustained for 15m fires the alert.

The failing log line looks like:

{"level":"error","error":"Get \"https://10.0.0.1:443/apis/status.gatekeeper.sh/v1alpha1\": dial tcp 10.0.0.1:443: i/o timeout","logger":"dynamic-client","gv":"status.gatekeeper.sh/v1alpha1","time":"...","message":"failed to retrieve API group"}

Kyverno's admission webhook validates/patches almost every write in the cluster (kyverno-resource-validating-webhook-cfg, failurePolicy=Fail, timeoutSeconds=10). When a replica's network path to the API server breaks, webhook requests that the API server load-balances onto it hang past the 10s deadline; the API server then rejects the write with failed calling webhook "validate.kyverno.svc-fail": ... context deadline exceeded. Controllers that reconcile frequently (the alloy-operator patches its collector DaemonSet every ~1 min) are the first visible victims — see 05-observability/01-alloy-operator-wedge.md, Variant B.

Why the pod stays "healthy"

Kyverno's liveness/readiness probes are served locally by the pod, so a replica with dead API-server connectivity still passes them, stays in the Service endpoints, and keeps receiving its share of admission traffic. Nothing else flags it — this alert is the direct signal.

Why `> 100` per 10m

A wedged replica emits thousands of dial errors per hour (~600–2000 per 10m in the observed incident; a healthy cluster emits zero across the fleet). The 100-threshold rides out a brief API-server blip (a few transient errors across all replicas) while firing well within an hour of a real wedge. A sustained cluster-wide API-server outage also trips this alert — correctly, since failurePolicy=Fail means that outage is rejecting writes cluster-wide.

Why `for: 15m`

Matches the grafana-provisioning alert's noise discipline: a single 10m window of dial errors during an API-server restart should not page; two consecutive windows (10m window + 15m sustain) means a persistent wedge.

Impact

Rejected writes cluster-wide, intermittently. Roughly 1/N of admission calls (N = healthy kyverno admission replicas + 1) fail, where the API server returns failed calling webhook ... context deadline exceeded. Deployments, Helm installs, and controller reconciles fail with that error and retry.
Controllers alert first. High-frequency reconcilers surface the problem before humans notice anything — e.g. the alloy-operator's AlloyOperatorReconcileErrors alert firing with a misleading "frozen on stale config" framing (2026-10-29, prd-ric02).
Config is NOT frozen — retries land on healthy replicas and eventually succeed, so this is degraded-but-functional, not a wedge. Sustained errors still page because every rejected write is a real (if retried) API failure.

Diagnosis

Set the kubectl context / KUBECONFIG for the cluster named in the alert's k8s_cluster_name label, then:

1. Confirm which replica is wedged

{k8s_cluster_name="<cluster>", k8s_namespace_name="platform-system", service_name="kyverno-admission-controller"}
  |= "dial tcp 10.0.0.1:443" | keep level, logger, message

Then map to pods (the log stream does not index pod names):

export KUBECONFIG=~/.kube/<cluster>.yaml
for p in $(kubectl -n platform-system get pods -o name \
  | grep kyverno-admission-controller | cut -d/ -f2); do
  echo "== $p =="
  kubectl -n platform-system logs $p --since=1h | grep -c 'dial tcp 10.0.0.1:443'
done
# The wedged replica counts hundreds/thousands; healthy ones count 0.

2. Rule out a cluster-wide API-server problem

If all replicas log dial errors, the API server itself (or the node networking common to all Kyverno pods) is down — that is a bigger incident; check the API server health and Azure status before recycling pods.

3. Note the node

kubectl -n platform-system get pod <wedged-pod> -o jsonpath='{.spec.nodeName}{"\n"}'

If other system pods on the same node restarted around the same time (as on 2026-10-29), suspect node-local networking (conntrack/SNAT, azure-ip-masq-agent) rather than the pod itself.

Mitigation

Recycle the wedged webhook backend pod — no other action is needed:

kubectl -n platform-system delete pod <wedged-pod>

Do not touch the workloads whose reconcile errors brought you here (e.g. the alloy-operator or its collectors) — they are healthy victims, and their retry loops recover on their own once the webhook answers again.

If the replacement lands on the same node and immediately starts timing out, the problem is the node, not the pod: investigate that node's networking (conntrack/SNAT, azure-ip-masq-agent), and consider draining it.

Verify

# Replacement pod: no dial errors.
kubectl -n platform-system logs <new-pod> --since=5m | grep -c 'dial tcp 10.0.0.1:443'   # 0

# Downstream victims recover: e.g. alloy-operator errors stop.
kubectl -n monitoring-system logs deploy/synapse-client-alloy-operator --since=10m \
  | grep -c '"level":"error"'   # 0, with successful "Reconciled release" lines

The alert clears once the 10m error count drops below threshold for the for: 15m window (no_data → resolved).

Notes

Live example (2026-10-29, prd-ric02): pod kyverno-admission-controller-669bd6b548-2wtr5 (node aks-default-19265985-vmss000001) lost API-server connectivity after a routine Kyverno pod churn the day before. It flooded ~48,000 dial tcp 10.0.0.1:443 errors (~3,500–13,000/hour) while the other two replicas logged zero. The alloy-operator accumulated 141 reconcile errors in 24h (all failed calling webhook), and the AlloyOperatorReconcileErrors alert fired with the Helm-wedge framing. Deleting the pod fixed everything; the replacement (scheduled on a different node) was clean, and the operator's errors stopped within one reconcile cycle.

Backtest (Oct 15–29, all clusters): this alert would have fired on prd-ric02 only, for two wedged periods — Oct 20 10:00 → Oct 21 19:40 (~34h, 204–12,700/10m) and Oct 24 12:00 → Oct 29 11:10 (~5 days, 120–8,600/10m). The 5-day episode is the one that surfaced via the alloy-operator alert — 4 days late, with the wrong cause. Notably, the Oct 27 Kyverno rollout did not clear it: the replacement pod re-wedged within the hour (both wedged replicas sat on the same node — treat node-level networking as the suspect when a recycle does not stick). All other clusters logged zero dial errors in the window — no false positives at this threshold. The alloy-operator was erroring during both earlier episodes too (2–50/hr, bursts to 616/hr); its alert auto-resolved on the intermittent drip, which is the gap this alert closes.

Kyverno Admission API Server Unreachable