k8s-monitoring 4.x (shipped by the synapse-client chart) deploys its collectors — alloy-metrics, alloy-singleton, alloy-logs — as Alloy custom resources reconciled by the alloy-operator. The operator does not manage a StatefulSet directly: it runs helm install/helm upgrade for a per-collector Helm sub-release (e.g. synapse-client-alloy-metrics) and records its state in a Helm release Secret (sh.helm.release.v1.synapse-client-alloy-metrics.vN).

If that release Secret is lost, the operator sees no existing release and falls back to helm install. The install then fails because the collector's resources (ServiceAccount, Service, StatefulSet, ClusterRole, ClusterRoleBinding) still exist but no longer carry the meta.helm.sh/release-name / meta.helm.sh/release-namespace ownership annotations Helm requires to adopt them:

failed to install release: Unable to continue with install: ServiceAccount
"synapse-client-alloy-metrics" ... cannot be imported into the current release:
invalid ownership metadata; annotation validation error: missing key
"meta.helm.sh/release-name"

The operator then reconciles with result="error" indefinitely.

A second, unrelated failure mode produces the same alert with a completely different root cause: the operator's patch/install requests are rejected because an admission webhook (Kyverno validate.kyverno.svc-fail / vpol.validate.kyverno.svc-fail, failurePolicy=Fail, timeoutSeconds=10) intermittently times out. When the webhook's backend loses connectivity to the API server, admission calls that land on it exceed the 10s deadline and the API server rejects the operator's patch — reconcile error. The operator retries every ~1 minute and usually succeeds, so config is not frozen; the alert is noisy rather than a true wedge. See Variant B: admission-webhook timeouts (Kyverno).

Impact

Cause A (Helm release secret lost): the affected collector's StatefulSet is frozen on its last-applied spec. Any change made to the Alloy CR (memory limit, GOMEMLIMIT, scrape config) is accepted into the CR but never reaches the running pods.
This is silent. The collector pods keep Running — no crashloop, no failing liveness probe — so nothing else alerts. The 2026-06-30 occurrence went unnoticed for a week; memory bumps merged to stop OOMs sat unapplied while the pods stayed on stale 2Gi config.
Other collectors managed by the same operator (e.g. alloy-singleton) can be perfectly healthy at the same time — the wedge is per-release.
Cause B (admission-webhook timeouts): config changes do eventually apply via the operator's retry loop, but reconciles fail continuously (the alert's "for: 30m" is easily met by a 2–16 errors/hour drip). The 2026-10-29 occurrence on prd-ric02 looked "wedged" but the GOMEMLIMIT/memory bump was already live in both the StatefulSet and the pod — no repair to the collectors was needed.

Diagnosis

Set the kubectl context / KUBECONFIG for the cluster named in the alert's cluster label, then:

Step 0 — identify the error signature first; it decides which cause you have.

NS=monitoring-system
kubectl logs -n $NS deploy/synapse-client-alloy-operator --since=1h \
  | grep '"level":"error"' | tail -5

failed to install release: ... cannot be imported ... missing key "meta.helm.sh/release-name" → Cause A (Helm-secret wedge) — continue with steps 1–4 below.
patch error: ... failed calling webhook "validate.kyverno.svc-fail" (or "vpol.validate.kyverno.svc-fail") ... context deadline exceeded → Cause B (admission-webhook timeout) — jump to Variant B.

Steps 1–4 (Cause A):

NS=monitoring-system

# 1. Which collector release is wedged? A healthy release shows Deployed=InstallSuccessful.
kubectl get alloy -n $NS -o custom-columns=\
'NAME:.metadata.name,DEPLOYED:.status.conditions[?(@.type=="Deployed")].reason'

# 2. Confirm the missing Helm release Secret (present for healthy collectors, gone for wedged ones).
kubectl get secret -n $NS | grep 'helm.release.*alloy'

# 3. Confirm the config is stuck: CR spec vs live StatefulSet.
kubectl get alloy synapse-client-alloy-metrics -n $NS \
  -o jsonpath='CR={.spec.alloy.resources.limits.memory}{"\n"}'
kubectl get statefulset synapse-client-alloy-metrics -n $NS \
  -o jsonpath='STS={.spec.template.spec.containers[0].resources.limits.memory}{"\n"}'
# CR != STS  →  wedged.

# 4. Operator log shows the adoption failure on a loop.
kubectl logs -n $NS -l app.kubernetes.io/name=alloy-operator --tail=50 \
  | grep -i "cannot be imported"

Remediation (Cause A: re-adopt the Helm release)

Re-attach Helm ownership annotations to the orphaned resources so the operator's next helm install adopts them (no downtime — the pods keep running throughout). Substitute the wedged release name (synapse-client-alloy-metrics, -alloy-singleton, or -alloy-logs).

NS=monitoring-system
REL=synapse-client-alloy-metrics

# Namespaced resources.
for kind in serviceaccount service configmap statefulset deployment; do
  kubectl annotate $kind -n $NS -l app.kubernetes.io/instance=$REL \
    meta.helm.sh/release-name=$REL \
    meta.helm.sh/release-namespace=$NS --overwrite
done

# Cluster-scoped resources (needs cluster-scoped RBAC — elevate if Forbidden).
for kind in clusterrole clusterrolebinding; do
  kubectl annotate $kind -l app.kubernetes.io/instance=$REL \
    meta.helm.sh/release-name=$REL \
    meta.helm.sh/release-namespace=$NS --overwrite
done

# Force an immediate reconcile.
kubectl rollout restart deployment/synapse-client-alloy-operator -n $NS

Within ~1 minute, verify recovery:

kubectl get secret -n $NS | grep 'helm.release.*'"$REL"          # release Secret recreated (v1)
kubectl get alloy $REL -n $NS \
  -o jsonpath='{.status.conditions[?(@.type=="Deployed")].reason}{"\n"}'   # InstallSuccessful
kubectl get statefulset $REL -n $NS \
  -o jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}{"\n"}'  # now matches the CR

All cluster-scoped kubectl annotate steps require cluster-admin RBAC. On the Scaleway/kommodity clusters (stg-par01, the *-par* fleet) a normal user is namespaced-only and the ClusterRole/ClusterRoleBinding annotate returns Forbidden — the operator stays wedged on the ClusterRole adoption until those two annotations are applied.

Workaround when you lack cluster-scoped RBAC but hold impersonate serviceaccounts: apply the cluster-scoped annotations as the alloy-operator's own ServiceAccount, which already owns these resources. Use the full literal --as string (a shell-built system:serviceaccount:$NS:... can truncate and be rejected as a user impersonation):
for REL in synapse-client-alloy-metrics synapse-client-alloy-singleton; do
  kubectl annotate clusterrole,clusterrolebinding $REL \
    --as='system:serviceaccount:monitoring-system:synapse-client-alloy-operator' \
    meta.helm.sh/release-name=$REL meta.helm.sh/release-namespace=monitoring-system --overwrite
done

Variant B: admission-webhook timeouts (Kyverno)

Observed on prd-ric02 on 2026-10-29: the alert fired with the "frozen on stale config" framing, but the Helm release state was fully healthy and the memory/GOMEMLIMIT bump was already live in the StatefulSet and pod. The operator's errors were all:

patch error: Internal error occurred: failed calling webhook
"validate.kyverno.svc-fail": failed to call webhook: Post
"https://kyverno-svc.platform-system.svc:443/validate/fail?timeout=10s":
context deadline exceeded

Root cause: one kyverno-admission-controller replica had lost connectivity to the API server (dial tcp 10.0.0.1:443: i/o timeout), so admission calls landing on it exceeded the 10s webhook deadline; with failurePolicy=Fail the API server rejected the operator's patch. The operator's ~1-minute retry loop usually lands on a healthy replica, so reconciles mostly succeed and config is not actually frozen.

Diagnosis, remediation, and prevention for this variant live in 11-platform/02-kyverno-admission-api-server-unreachable.md (the Kyverno Admission API Server Unreachable alert now detects the wedged replica directly). In short: confirm config is not frozen (CR == StatefulSet), find and delete the Kyverno admission pod flooding dial tcp 10.0.0.1:443 errors, and do not run the Cause-A annotation repair below against a healthy release.

Why it happens / prevention (Cause A)

The observed occurrence coincided with the k8s-monitoring 4.0.1 → 4.1.6 upgrade on 2026-06-30 (which also bumped alloy-operator 0.5.3 → 0.5.10). The exact step that deleted the release Secret is not proven, but two tempting explanations were investigated and ruled out:

Not a direct-render → operator-model transition — the operator model predates both versions (k8s-monitoring 2.1.0); 4.0.1 and 4.1.6 are both operator-based.
Not an ArgoCD prune — the operator's child resources (StatefulSet, Service, release Secret) carry no argocd.argoproj.io/instance label and never appear in the synapse-client Application resource tree (ArgoCD tracks only the Alloy CRs), so it cannot prune them. Verified on an untouched healthy cluster.

Only the remote-write (client-only) clusters wedged — those with remoteWrite.enabled, whose alloy-metrics CR carries a projected ServiceAccount-token volume (mounts.extra: sa-token) for OIDC auth to the central synapse-server gateway. Clusters running their own synapse-server (dev-neu, prd-par01) scrape into the local gateway, have no remoteWrite and no projected-token mount, and were untouched. The discriminating variable is the remote-write collector shape, not the presence of synapse-server per se; why that CR variant's sub-release lost its Helm Secret during the upgrade is the unproven part.

Recurrence: under routine operation this will not re-trigger — ArgoCD tracks nothing prunable here and the release Secrets are stable. The real risk window is a future k8s-monitoring chart upgrade (upstream reports below show the class recurs on installs/upgrades/reconciles). Bump on a canary and watch the AlloyOperatorReconcileErrors alert before rolling the fleet — it catches a recurrence in ~30 minutes rather than weeks.

Upstream: the operator's inability to adopt leftover artifacts (e.g. a ClusterRole) from an incomplete uninstall is tracked in grafana/k8s-monitoring-helm#2140 — open and unfixed as of k8s-monitoring 4.2.0. Related reports: #1856 (Helm releases not found) and #1572 (mandatory operator model — no opt-out to a plain StatefulSet). Until an upstream fix lands, the annotate-adopt repair above is the remediation.

Why it happens / prevention (Cause B)

The 2026-10-29 prd-ric02 occurrence: a Kyverno admission-controller replica (node aks-default-19265985-vmss000001) silently lost connectivity to kubernetes.default.svc after a routine Kyverno pod churn the day before. The pod kept passing liveness probes — Kyverno serves its health endpoint locally, so an API-server-less replica looks healthy to the kubelet and keeps receiving its share of admission traffic.

Detection now exists: the Kyverno Admission API Server Unreachable alert (see 11-platform/02-kyverno-admission-api-server-unreachable.md) fires on sustained dial tcp 10.0.0.1:443 errors in the admission-controller logs, catching a wedged replica directly instead of via a downstream victim like this alert.