Pod Pending (GPU) Over 60 Minutes

Runbook for a GPU-requesting pod that has been stuck in phase Pending for more than 60 minutes, triggered by the PodPendingGPU Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). There is no matching Mimir-ruler alert.

This alert is the GPU-specific companion to 23-pod-pending.md (the catch-all PodPendingOver30Minutes alert). GPU pods are excluded from that alert via an unless clause on kube_pod_container_resource_requests{resource="nvidia_com_gpu"} and routed here instead, because:

• GPU node provisioning is slow. The cluster-autoscaler has a ~15m timeout for new GPU VMs to register. Azure H100 VMs can take 10-20m to provision even when quota is available. A 30m for window fires during normal provisioning.
• KEDA self-corrects. Inference services are KEDA-autoscaled on request metrics. A traffic spike scales replicas beyond GPU capacity, then scales back down as traffic drops. The pending pods are deleted by the HPA downscale — typically within 10-15m.
• 60m covers both cycles. Autoscaler provisioning (15m) + KEDA downscale (15m) + scrape/eval margins = ~35m worst case. 60m gives comfortable headroom before paging.

The discriminator is kube_pod_container_resource_requests{resource="nvidia_com_gpu"} — a core kube-state-metrics metric that is always available (no label allowlist needed). The resource label uses nvidia_com_gpu (underscores) because Prometheus sanitises dots in label values. This correctly identifies any pod requesting a GPU, regardless of team, namespace, or workload type.

Pending is the API phase a pod holds after it is admitted but before any container starts — for GPU pods this almost always means FailedScheduling with Insufficient nvidia.com/gpu: the scheduler cannot find a node with a free GPU. See 23-pod-pending.md for the general phase semantics and the container-level sibling alerts (01/02) that fire first on init-container problems.

Meaning

GPU predictor pods on inference clusters (prd-par02 / prd-ric02) have a distinct Pending shape: the FailedScheduling event names Insufficient nvidia.com/gpu as the constraint, and the pod's nodeSelector / affinity pins it to a specific GPU node pool (e.g. gpu-type=h100, nvidia.com/gpu.product: NVIDIA-H100-PCIe). Each H100 node exposes exactly 1 GPU, so the pool's effective capacity is one predictor replica per node — a KEDA HPA scale-up that requests more replicas than free GPUs produces Pending pods by construction.

This is expected transient behaviour, not a misconfiguration: the cluster-autoscaler is supposed to add nodes when GPU capacity runs out, and KEDA scales the InferenceService back down when traffic drops. The oncall question is whether the autoscaler is actually provisioning nodes, or whether it is blocked (Azure quota, region stockout, scale-up timeout).

Impact

• The pending pod is doing no work, but existing replicas are typically still serving — the service is degraded-but-up, not down.
• If all replicas are pending (no running pods), the InferenceService has no serving capacity — this is caught earlier by DeploymentNoRunningReplicas at P1.
• Prolonged Pending (hours) with no autoscaler action means the GPU node pool is at capacity and Azure cannot or will not provision more — this is an infrastructure bottleneck that needs escalation.

Diagnosis

1. Confirm the scheduling constraint is GPU

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par02.yaml
NS=<namespace>
POD=<pod>
kubectl -n $NS get pod $POD -o wide
kubectl -n $NS describe pod $POD | sed -n '/^Events:/,$p'

Look for FailedScheduling with Insufficient nvidia.com/gpu in the message. The full message lists all constraints that failed; if GPU is the only one, the node pool is full but the pod would fit if a GPU were free. Cross-check with the node selector and tolerations:

kubectl -n $NS get pod $POD -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.tolerations}{"\n"}'

KServe InferenceService pods carry nodeSelector: {gpu-type: h100, workload: gpu-h100} and a toleration for workload=gpu-h100:NoSchedule; the affinity may also pin nvidia.com/gpu.product to a specific GPU model.

2. Check GPU allocation across the node pool

# List all H100 nodes and their GPU capacity vs allocated
kubectl get nodes -l gpu-type=h100 -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\t"}{.status.capacity.nvidia\.com/gpu}{"\n"}{end}'

Then check what is actually consuming the GPUs — each row shows a running predictor pod and how many GPUs it holds:

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.spec.nodeName}{"\t"}{range .spec.containers[*]}{.resources.requests.nvidia\.com/gpu}{end}{"\n"}{end}' | grep -E '\t[0-9]' | grep 'h100-gpu-pool'

If every node shows nvidia.com/gpu 1 1 in Allocated resources (i.e. 1 requested / 1 capacity), the pool is saturated. The pending pods will stay Pending until either a node is added or a running pod is terminated (HPA scale-down).

3. Check whether the cluster-autoscaler tried to scale up

kubectl get events -A --sort-by='.lastTimestamp' | grep -E 'TriggeredScaleUp|ScaledUpGroup|ScaleUpTimedOut|cluster-autoscaler' | tail -10

Key events to look for:

• TriggeredScaleUp — the autoscaler detected the pending pod and requested more nodes (e.g. MachineDeployment/.../h100-gpu-pool 6->9 (max: 20)). This means the system is working as designed.
• ScaledUpGroup — the node pool size was actually increased.
• ScaleUpTimedOut — the new VMs failed to register within the timeout (typically 15m). This usually means Azure cannot fulfill the GPU VM request (quota, regional stockout, or capacity constraints). This is an infrastructure bottleneck, not a Kubernetes problem — escalate to the platform/infra team to check Azure GPU quota and availability in the region.
• NotTriggerScaleUp — the autoscaler decided not to scale (e.g. max node count reached, or the pod's constraints don't match any scalable node group).

4. Check the HPA / KEDA scaling trajectory

kubectl -n $NS get hpa -l 'serving.kserve.io/inferenceservice' -o wide
kubectl -n $NS get events --sort-by='.lastTimestamp' | grep -E 'ScalingReplicaSet|SuccessfulRescale.*hp' | tail -10

If the HPA is actively scaling down (New size: N; reason: All metrics below target), the pending pods will be deleted as the replica count drops — no action needed, the system is self-correcting. If it is scaling up or holding, the pending pods will remain until new nodes arrive or a manual intervention reduces the replica count.

5. Determine if intervention is needed

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the cause the events named

• Pool saturated, autoscaler provisioning — wait. The autoscaler is adding nodes; pods will schedule once they join. No action needed unless the ScaleUpTimedOut event appears.
• ScaleUpTimedOut — escalate to the platform/infra team. Azure cannot fulfill the GPU VM request — check Azure GPU quota (az vm list-usage --location <region>) and regional availability. A quota increase request or a capacity reservation may be needed. This is an infrastructure bottleneck, not a Kubernetes problem.
• Node pool at max (NotTriggerScaleUp) — raise the node pool max size (IaC/cluster PR) or request an Azure quota increase to allow a higher max. Alternatively, reduce the InferenceService maxReplicas to match the available GPU capacity (chart-values PR) so KEDA does not overshoot.
• HPA scaling down — no action. The pending pods will be deleted by the HPA downscale. This is the expected self-correction path.

Step 3 — Reduce KEDA pressure (emergency)

If the pending pods are blocking other workloads or the alert is recurring:

# Check current replica count vs desired
kubectl -n $NS get deploy <predictor-deployment> -o jsonpath='replicas={.spec.replicas}{"\n"}'
# Temporarily reduce maxReplicas via KEDA ScaledObject (will be reverted by ArgoCD)
kubectl -n $NS patch scaledobject <keda-scaledobject-name> --type=merge -p '{"spec":{"maxReplicaCount":<n>}}'

This is a temporary hold — ArgoCD self-heal reverts the change in ~15s. For durability, reduce maxReplicas in the chart-values PR so KEDA cannot scale beyond the available GPU pool size.

Verify

• The alert clears once the pod leaves Pending: the phase="Pending" gauge drops to 0 (the pod is Running, or deleted) and the series stops matching == 1, sustained past the for: 60m window.
• kubectl -n $NS get pod $POD shows Running (and Ready once probes pass).
• kubectl -n $NS describe pod $POD shows Scheduled (successfully assigned to a node) and no new FailedScheduling events.
• If the cause was HPA downscale, the pod is simply deleted — kubectl -n $NS get pod $POD returns "NotFound".
• Check that the autoscaler successfully added nodes: kubectl get nodes -l gpu-type=h100 should show the new nodes as Ready.

Background: why GPU pods are split into a separate alert

The general PodPendingOver30Minutes alert (see 23-pod-pending.md) fires at 30m — too aggressive for GPU pods, where the autoscaler's 15m provisioning timeout and KEDA's 10-15m HPA downscale cycle routinely produce Pending pods that self-resolve within 30-45m. Paging oncall at 30m for a transient, self-correcting condition is noise.

The split uses kube_pod_container_resource_requests{resource="nvidia_com_gpu"} as the discriminator — a core kube-state-metrics metric that is always available (no label allowlist needed). The resource label uses nvidia_com_gpu (underscores) because Prometheus sanitises dots in label values. The and / unless operators join on (k8s_cluster_name, environment, namespace, pod):

• This alert (PodPendingGPU): ... and on (...) count by (...) (kube_pod_container_resource_requests{resource="nvidia_com_gpu"} > 0) — fires only for pods that request a GPU, at 60m, severity warning.
• General alert (PodPendingOver30Minutes): ... unless on (...) count by (...) (kube_pod_container_resource_requests{resource="nvidia_com_gpu"} > 0) — excludes GPU pods, fires at 30m, severity critical.

This approach directly identifies the characteristic that matters: the pod requests a GPU.

Live examples (2026-10-21)

• prd-par02 / mlservices, corti-s1-tiny-20260707-predictor-74c86d676c-6vwln Pending 46m. KEDA scaled the InferenceService from 2 to 5-6 replicas (traffic spike), but all 6 H100 GPUs were occupied. The cluster-autoscaler triggered a scale-up (6→9→10) but the new VMs failed to register (ScaleUpTimedOut after 15m). The HPA then scaled back down (6→5→4→3) and the pending pods were deleted. No action was needed — the system self-corrected. This would have paged at 30m under the old catch-all alert; with the split, the 60m window would have covered the entire cycle without paging.