Runbook for trivy vulnerability metrics that have gone stale — the trivy-operator is no longer producing trivy_image_vulnerabilities samples. This is an absence alert — it fires when metrics that should be there are not. A scrape-target up alert would not catch this: the trivy-operator pod can stay Running with up=1 while producing zero vulnerability metrics — it cannot download its DB but still serves an empty /metrics endpoint.

Meaning

Trivy-operator (in trivy-system) scans all container images running in the cluster for known vulnerabilities (CVEs). The trivy_image_vulnerabilities metric is a gauge — count of vulnerabilities per image, grouped by severity. It is scraped hourly via a ServiceMonitor (interval: 1h) and remote-written to Mimir.

The alert uses the same self-maintaining unless pattern as 08-mimir-self-monitoring-scrape-stalled.md and 09-synapse-cluster-metrics-silent.md:

Left side — count by (k8s_cluster_name, environment) (count_over_time(trivy_image_vulnerabilities[6h])): the number of distinct trivy series seen in the last 6h. A cluster that has trivy-operator deployed and working produces hundreds of series per scrape.
Right side — count by (k8s_cluster_name, environment) (last_over_time(trivy_image_vulnerabilities[65m])): the number of distinct trivy series with a sample in the last 65m. The 65m window is 5m beyond the 1h scrape interval, so it covers one full scrape cycle plus a margin — without it, a bare instant vector would be empty between scrapes (the 5m staleness window is shorter than the 1h interval) and the alert would flap hourly.
unless fires when the left side has data (trivy was working in the last 6h) but the right side is empty (no scrape in the last 65m).

This is self-maintaining and decommission-safe: a cluster that never had trivy, or one that was decommissioned, clears itself after 6h of silence. There is no hardcoded cluster list to keep in sync.

for: 30m rides out the 1h scrape interval — the alert fires ~95m after the last successful scrape, meaning at least one full scrape cycle was missed by 35m.

Impact

All trivy vulnerability alerts are blind — 07-kubernetes/13-trivy-critical-vulnerability-1w through 18-trivy-critical-vulnerability-immediate silently stop firing. They query last_over_time(trivy_vulnerability_id[1h]), which returns an empty vector when no scrape arrives, so they stop evaluating rather than firing on the gap. New critical CVEs will not page anyone.
The Security Scan and Security Scanv2 Grafana dashboards are blank — their variable queries (label_values(trivy_image_vulnerabilities, ...)) return empty when no current series exist, so all panels render nothing.
The cluster itself is fine — this is a monitoring blind spot, not a service outage. Images are still running; they are just not being scanned for new vulnerabilities.

Diagnosis

The alert label carries k8s_cluster_name and environment. From the alert, note the cluster.

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prod-weu.yaml

1. Confirm the absence

Confirm via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

# Should return 0 if stale, > 0 if healthy
QUERY='count by (k8s_cluster_name, environment) (last_over_time(trivy_image_vulnerabilities{k8s_cluster_name="<cluster>"}[65m]))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[]? | "\(.metric.k8s_cluster_name): \(.value[1]) series"'

# Confirm it was recently seen (within 6h)
QUERY2='count by (k8s_cluster_name, environment) (count_over_time(trivy_image_vulnerabilities{k8s_cluster_name="<cluster>"}[6h]))'

2. Check the trivy-operator pod

kubectl -n trivy-system get pods -l app.kubernetes.io/name=trivy-operator -o wide
kubectl -n trivy-system get pods -l app.kubernetes.io/name=trivy-operator -o jsonpath='{range .items[*]}{.metadata.name} restarts={.status.containerStatuses[0].restartCount} phase={.status.phase}{"\n"}{end}'

If the pod is CrashLoopBackOff or OOMKilled, see 07-kubernetes/02-pod-crashloop.md or 07-kubernetes/11-container-oomkilled.md.

3. Check scan job errors (most common cause: DB download failure)

Trivy-operator creates scan Jobs in trivy-system. If the vulnerability DB cannot be downloaded, the scan Jobs fail:

# Recent scan jobs
kubectl -n trivy-system get jobs --sort-by=.metadata.creationTimestamp | tail -10

# Check scan job logs for DB download errors
kubectl -n trivy-system logs -l app.kubernetes.io/managed-by=trivy-operator --tail=50 | grep -iE 'error|denied|unauthorized|failed|timeout'

Common DB download failures:

DENIED — the DB registry requires authentication (e.g. ghcr.io/aquasec/trivy-db returns DENIED for anonymous pull). The chart must use docker.io (Docker Hub) which serves trivy OCI artifacts with anonymous pull. See chart trivy values: trivy-operator.trivy.dbRegistry / javaDbRegistry / policiesBundle.registry.
BLOB_UNKNOWN — the DB registry does not serve the OCI artifact (e.g. mirror.gcr.io returns BLOB_UNKNOWN for trivy-checks). Same fix: use docker.io.
UNAUTHORIZED — the scan Job cannot pull images from the container registry (e.g. ACR credentials missing on gen3/Talos clusters). The chart mounts the containerd socket so trivy can inspect already-pulled images locally; if the socket is not mounted, trivy falls back to remote pull and fails.

4. Check the trivy-operator logs

kubectl -n trivy-system logs -l app.kubernetes.io/name=trivy-operator --tail=100 | grep -iE 'error|warn|failed|reconcile'

Look for:

failed to download vulnerability DB — DB registry issue (step 3)
failed to scan image — image pull or containerd socket issue
no vulnerabilities found (repeated for every image) — the DB is empty or corrupt; the operator downloaded a stub but has no actual vulnerability data

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the DB registry (if DB download is failing)

If the scan Jobs show DENIED or BLOB_UNKNOWN, the DB registry is wrong. The chart should use docker.io for dbRegistry, javaDbRegistry, and policiesBundle.registry. Check the chart version:

kubectl -n trivy-system get deployment trivy-operator -o jsonpath='{.spec.template.spec.containers[0].image}'

Check the chart values for the affected cluster (in the deployments repo, config/services/trivy/ and environments/<env>/<cluster>/infra/apps/trivy/values.yaml). The trivy-operator.trivy.dbRegistry must be docker.io, not ghcr.io or mirror.gcr.io.

If the chart needs a fix, open a PR against charts/charts/trivy/ and update the config.json tag in the deployments repo once merged.

Step 3 — Restart the trivy-operator (if pod is stuck)

If the operator pod is in a bad state (not crash-looping but not producing metrics):

kubectl -n trivy-system delete pod -l app.kubernetes.io/name=trivy-operator

The Deployment recreates the pod. Within ~1h (the scrape interval), trivy_image_vulnerabilities should reappear if the DB downloads successfully.

Step 4 — Fix containerd socket mount (if image scanning fails)

On gen3/Talos clusters, trivy needs the containerd socket to inspect already-pulled images. If the socket is not mounted, trivy falls back to remote pull and fails with UNAUTHORIZED. The chart mounts the socket via trivy-operator.trivyOperator.scanJobCustomVolumes — verify the volume mount is present in the scan Jobs.

Verify

The alert clears once count by (k8s_cluster_name, environment) (last_over_time(trivy_image_vulnerabilities{k8s_cluster_name="<cluster>"}[65m])) returns > 0, sustained for the for: 30m window.
The Security Scan / Security Scanv2 Grafana dashboards repopulate.
The trivy vulnerability alerts (07-kubernetes/13-18) resume evaluating.