Trivy Critical Vulnerability One Week

Runbook for a critical vulnerability that has persisted in an external-facing image for more than 1 week. This is the primary Trivy runbook — the other 3 tiers (14-trivy-critical-vulnerability-4w, 15-trivy-high-vulnerability-1w, 16-trivy-high-vulnerability-6w) share the same diagnosis and mitigation and reference this runbook.

Meaning

Trivy-operator (in trivy-system) scans all container images running in the cluster for known vulnerabilities (CVEs). The trivy_vulnerability_id metric is a gauge — 1 when a vulnerability is present for a given image_repository/image_tag/vuln_id combination, absent when not. It is scraped hourly via a ServiceMonitor and remote-written to Mimir.

The DataDog alerts these replace used calendar_shift to check "vuln present now AND present N time ago". In PromQL this is an and on (...) between the current vector and the same vector with an offset, wrapped in count by to produce one alert per image per tenant (not one per CVE):

count by (image_repository, image_tag, k8s_cluster_name, environment, exported_namespace) (
  max by (image_repository, image_tag, vuln_id, k8s_cluster_name, environment, exported_namespace) (
    last_over_time(trivy_vulnerability_id{...}[1h])
    and on (image_repository, image_tag, vuln_id, k8s_cluster_name, environment, exported_namespace)
    last_over_time(trivy_vulnerability_id{...}[1h] offset 1w)
  )
)

The max by (... vuln_id ...) deduplicates across containers (a vuln in both the main and init container of the same image is counted once). The count by collapses the per-vuln series into a single alert per image per tenant, with {{ $value }} = the number of distinct vulnerabilities. exported_namespace is the tenant namespace where the scanned workload runs (distinct from namespace=trivy-system where the operator scrapes).

last_over_time(...[1h]) is used instead of a bare instant vector because the trivy ServiceMonitor scrapes hourly — a 5m staleness window would be empty at almost every ruler eval. The [1h] window matches the scrape interval; the offset 1w/4w/6w reaches into historical Mimir blocks.

The 4 alert tiers

Scope: external-facing images only

The image_repository filter restricts the alert to external-facing services — the images that receive traffic from outside the cluster:

platform/(medical-coding|code-api|copilot-api|platform-api|copilot-api-v2|code-frontend|copilot-frontend|copilot-audio-bridge|voice-vault)
|speech-recognition/audio-bridge
|ai-agents/agent-api
|traefik
|grafana/grafana

Internal-only images (GPU operators, redis, cert-manager, etc.) are excluded — they are not exposed to external traffic and are tracked separately. Production-only (environment="production").

Impact

A known-critical CVE is exploitable in an external-facing service. If the service is internet-reachable, this is an active security exposure. The 1-week window means the vuln has survived at least one weekly remediation cycle — it is not being caught by routine image updates.

The vuln_id label is the CVE identifier (e.g. CVE-2026-33815). Look it up in the NVD to assess exploitability and CVSS score.

Diagnosis

The alert labels carry image_repository, image_tag, cluster, environment, exported_namespace. The alert value ({{ $value }}) is the count of distinct vulnerabilities — use the query in step 1 to list the individual CVEs.

1. Identify the vulnerability and the image

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par02.yaml

From the alert, note image_repository, image_tag, and exported_namespace. Find the running pods using that image:

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name} {.spec.containers[*].image}{"\n"}{end}' | grep "<image_repository>:<image_tag>"

To list the individual vulnerability IDs (the alert reports a count, not individual CVEs), query Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

QUERY='count by (vuln_id) (last_over_time(trivy_vulnerability_id{image_repository="<image_repository>",image_tag="<image_tag>",exported_namespace="<exported_namespace>",severity=~"(?i)critical",environment="production"}[1h]) and on (image_repository, image_tag, vuln_id, k8s_cluster_name, environment, exported_namespace) last_over_time(trivy_vulnerability_id{image_repository="<image_repository>",image_tag="<image_tag>",exported_namespace="<exported_namespace>",severity=~"(?i)critical",environment="production"}[1h] offset 1w))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[].metric.vuln_id'

2. Look up the CVE

Check the CVE details — CVSS score, exploitability, affected versions, fixed versions:

• Search the NVD: https://nvd.nist.gov/vuln/detail/<vuln_id>
• Check if a fix is available in a newer image tag: look at the image's registry (ACR corti.azurecr.io) for newer tags.

3. Check the trivy report in the cluster

Trivy-operator stores vulnerability reports as ConfigMaps (or CRDs, depending on the trivy-operator version):

kubectl -n trivy-system get configmaps -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -i <image_repository>
# Or via the vulnerabilityreports CRD:
kubectl get vulnerabilityreports.aquasecurity.github.io -A | grep <image_repository>

The report gives the full list of vulnerabilities for the image, including fixed versions.

4. Check how long the vulns have been present

Confirm the age via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti). For each vuln_id from step 1:

QUERY='last_over_time(trivy_vulnerability_id{image_repository="<image_repository>",image_tag="<image_tag>",exported_namespace="<exported_namespace>",vuln_id="<vuln_id>"}[1h] offset 1w)'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

If this returns 1, the vuln was present 1 week ago — confirming the alert.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Remediate the vulnerability

The fix depends on the CVE and the image:

• Fixed version available — update the image tag in the service's Helm values (a GitOps change to the deployments repo). The image is rebuilt/pulled from ACR with the fixed version. This is the durable fix.
• No fixed version yet — assess whether the vuln is exploitable in our deployment (network exposure, config, whether the vulnerable code path is reached). If not exploitable, document the risk and monitor for a fix. If exploitable, consider a workaround (config change, WAF rule, network policy) until a fixed image is available.
• Base image vuln (OS package, e.g. openssl, glibc) — rebuild the image with an updated base image. For Corti platform images, this is a change to the service's Dockerfile base image tag.

Step 3 — Deploy and verify

After the image update is merged and deployed (ArgoCD sync), trivy-operator re-scans the new image within the hour. The trivy_vulnerability_id series for the old image_tag disappears (the old pods are gone); the new image_tag has no series for that vuln_id (if the fix worked).

Step 4 — Verify the alert clears

The alert clears once the and on (...) no longer matches — i.e. the vulnerabilities are no longer present now (the last_over_time(...[1h]) returns empty for those vuln_ids) or were not present at the offset time. With the for: 2h window, the alert clears within 2h of the fix being deployed and trivy re-scanning. When all vulns for an image are resolved, the count by returns no series and the alert clears.

Verify

• kubectl get pods -A -o jsonpath='{...}' | grep <image_repository> shows the new image tag running.
• The Mimir query count by (vuln_id) (last_over_time(trivy_vulnerability_id{image_repository="<repo>",image_tag="<new-tag>",exported_namespace="<ns>",severity=~"(?i)critical",environment="production"}[1h])) returns no data (no vulns present in the new image).
• The alert clears within 2h of the trivy re-scan.

Notes

• Scope — environment="production" only; image_repository restricted to external-facing services (see Meaning). Internal images are excluded.
• Hourly scrape + [1h] window — the trivy ServiceMonitor scrapes hourly; last_over_time(...[1h]) ensures the ruler sees the last scrape at every 1m eval. A bare instant vector would be empty at most evals.
• offset requires historical blocks — the offset 1w/4w/6w queries reach into Mimir's store-gateway (historical TSDB blocks). These evaluations require the ruler→store-gateway NetworkPolicy (port 9095) to allow gRPC traffic; without it they fail with err-mimir-store-consistency-check-failed (fixed in PLA-5955).
• Related — 07-kubernetes/09-certificate-expiring-soon.md for the cert-manager expiry alert (same kubernetes.security group).