GPUs Underutilized For 48h

Runbook for GPUs that have been underutilized for 48h, triggered by the GpusUnderutilized Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). This mirrors the Datadog [platform/finops] GPUs underutilized for 48h alert. There is no matching Mimir-ruler alert.

This is a finops / cost signal, not an availability signal — hence priority: 4 (P4, low). A GPU sitting idle is expensive; the alert catches it so it can be re-assigned or the node released. The alert reports a count of idle GPUs per cluster ({{ $value }}), not individual GPUs — one alert per cluster, not one per GPU.

Meaning

DCGM_FI_DEV_GPU_UTIL is the GPU SM (streaming multiprocessor) utilization percentage (0–100), scraped from the DCGM-exporter (nvidia-dcgm-exporter) running on GPU nodes, remote-written to synapse-server Mimir. The metric carries device (GPU identifier, e.g. nvidia0), gpu (GPU index), Hostname (the node name), modelName (e.g. Tesla T4), k8s_cluster_name, environment, and exported_namespace/exported_pod/exported_container (the workload allocated to the GPU, exported from the pod's cgroup labels).

The alert fires when the 48h-average DCGM_FI_DEV_GPU_UTIL per k8s_cluster_name / device / Hostname is < 1 — i.e. the GPU averaged under 1% utilization over the last 48h. The count by (k8s_cluster_name, environment) collapses the per-GPU results into one alert per cluster, with {{ $value }} = the number of idle GPUs. 1% is deliberately low: a GPU doing real inference or training sits well above 1% while working; under 1% for 48h means it is effectively idle. The 48h window is long enough to ride out a workload gap (no jobs queued overnight) but short enough to catch a persistently-idle GPU.

Impact

• Cost — GPUs are the most expensive per-node resource (an NVIDIA A100/H100 node costs orders of magnitude more than a CPU node). A GPU sitting idle is pure waste. 48h of idle on an H100 is meaningful spend.
• Not an availability issue — the workload assigned to the GPU is either not running, not using the GPU (running on CPU by mistake), or genuinely idle (no work to do).
• The finops/impact signal is "this GPU could be doing useful work elsewhere, or the node could be released." It is not urgent (the system is not broken), but it is a recurring cost leak.

Diagnosis

The alert labels carry k8s_cluster_name and environment. The alert value ({{ $value }}) is the count of idle GPUs — use the query in step 1 to list the individual nodes and devices.

1. List the idle GPUs on the cluster

The alert reports a count, not individual GPUs. Query Mimir for the per-node/device breakdown (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

QUERY='avg_over_time(avg by (k8s_cluster_name, environment, device, Hostname) (DCGM_FI_DEV_GPU_UTIL{k8s_cluster_name="<k8s_cluster_name>"})[48h:]) < 1'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[] | "\(.metric.Hostname) \(.metric.device)"'

From the output, note each Hostname (node name) and device (GPU identifier, e.g. nvidia0).

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-ric02.yaml
NODE=<Hostname>
GPU=<device>
kubectl get node $NODE -o wide
kubectl describe node $NODE | sed -n '/^Allocated resources:/,$p'   # GPU capacity vs allocated

Confirm the GPU is allocated (a pod is using it) vs free (no pod). Allocated resources shows nvidia.com/gpu requests — if 0, the GPU is not allocated to any pod (and the alert is just "the GPU has no workload, consider releasing the node"). If allocated, the alert's exported_pod/exported_namespace labels already name the workload; confirm on the node:

kubectl get pods -A --field-selector spec.nodeName=$NODE -o wide | grep -i gpu
# or, directly from the alert labels:
kubectl -n <exported_namespace> get pod <exported_pod> -o wide

2. Is the assigned workload supposed to use the GPU?

If a pod is allocated the GPU but it is idle, the workload is not using the GPU it asked for. The alert's exported_pod/exported_container labels identify it:

kubectl -n <exported_namespace> describe pod <exported_pod> | grep -A3 -iE "gpu|nvidia"
kubectl -n <exported_namespace> logs <exported_pod> -c <exported_container> --tail=50 | grep -iE "gpu|cuda|nvidia|device"

Common causes:

• Wrong runtime — the container requests nvidia.com/gpu but the app does not call CUDA (running on CPU by mistake, a missing CUDA_VISIBLE_DEVICES, a fallback path). The pod is scheduled on the GPU but never uses it.
• Idle model — an inference server loaded a model but has no traffic (no requests → no GPU work). The GPU is "allocated" but idle. Either route traffic to it or scale it down.
• No work queued — a training/inference job pod is up but the queue is empty. Either feed it work or scale it down.
• Wrong node pool — the workload was scheduled on a GPU node pool by mistake (a node selector / toleration that admits it to GPU nodes when it does not need a GPU). Move it to a CPU node pool.

3. Check the GPU is healthy (not failed-idle)

A GPU that has crashed/hung reports low util too. Confirm the GPU is actually usable:

kubectl -n <dcgm-exporter-namespace> exec <dcgm-exporter-pod> -- nvml-query   # or via the DCGM exporter metrics
# on the node (if you have node access):
# nvidia-smi   # shows GPU status, processes, errors

If the GPU is in an error state (ECC errors, hung), it is idle because it cannot do work — that is a hardware failure, not a finops issue. See the GPU-operator / DCGM alerts for ECC/health.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Release or re-assign the GPU

Based on Diagnosis:

• No workload allocated — the GPU node is idle. Scale the node pool down (if autoscaled) or cordon/drain and release the node (if fixed) so the GPU is not being paid for unused.
• Wrong workload allocated — the pod does not need a GPU. Remove the nvidia.com/gpu request / GPU toleration from the chart values in a PR and let it reschedule to a CPU node pool. This frees the GPU node for a real GPU workload.
• Idle model — the inference server has no traffic. Scale it down, or route traffic to it. If it is a standby (warm pool), decide whether the standby is worth the idle GPU cost.
• No work queued — the job pod is up but idle. Either feed it work or scale it down between jobs.

Step 3 — Fix the node pool / scheduling (if mis-scheduled)

If GPU workloads keep landing on a GPU node pool without using the GPU, the scheduling is wrong — fix the chart values (node selector / tolerations / resource requests) so only workloads that actually need a GPU request one. This is a chart-values PR; ArgoCD self-heal reverts manual changes.

Verify

• The alert clears once the 48h-average DCGM_FI_DEV_GPU_UTIL >= 1 for the GPU. Given the 48h lookback, this can lag the fix by up to 48h (the low-util samples age out slowly) — the alert may not clear immediately even after the GPU is doing work; recheck over a day.
• For a node release, the node is gone from kubectl get nodes and the GPU series disappears from Mimir.
• For a workload moved off the GPU, kubectl describe node $NODE shows no nvidia.com/gpu allocation, and the pod reschedules to a CPU node (kubectl get pod <pod> -o wide).
• For a workload that started using the GPU, DCGM_FI_DEV_GPU_UTIL for that GPU rises above 1% in Mimir.