🖥️ Managing NVIDIA GPUs

⚙️ GPU Operator

The GPU operator is a Kubernetes operator that automates the management of NVIDIA GPUs in a cluster. It handles driver installation, monitoring, and enabling features like MIG mode.

🔀 MIG Mode

MIG (Multi-Instance GPU) mode partitions a single physical GPU into multiple isolated instances, each with dedicated resources. This improves resource utilization when running multiple workloads on the same GPU.

📋 Supported GPU Models and MIG Profiles

• Supported GPU models
• Supported MIG profiles per model

Custom MIG configurations can also be added. See more in the Custom MIG configurations on Scaleway section below.

🚀 Enabling MIG Mode

Prerequisites

Before enabling MIG mode, deploy the Node-feature-discovery (NFD) helm chart to the cluster. NFD is a Kubernetes add-on that detects hardware features and system configuration. See the upstream repository.

☁️ AKS GPU Nodes

Deploy the GPU operator helm chart to the cluster.

GPU model-specific configuration may be required:

H100

No extra configuration needed — H100 GPUs support enabling MIG without a reboot.

A100

A100 GPUs require a reboot to enable MIG mode. Configure the GPU operator to allow rebooting the node:

gpu-operator:
  migManager:
    env:
      - name: WITH_REBOOT
        value: "true"

Since the node will reboot, take the following steps beforehand to prevent scheduling issues:

1. Add a taint to prevent new workloads from being scheduled:

   "nvidia.com/gpu:NoSchedule"
   

2. Add the corresponding toleration to both the GPU operator and NFD charts, so they can still run on the tainted node:

   - key: nvidia.com/gpu
     operator: Exists
     effect: NoSchedule
   

The GPU operator will remove the taint automatically once MIG is enabled and configured.

🔧 Talos GPU Nodes

NVIDIA drivers

In Talos nodes, GPU drivers are pre-installed using the following system extensions (use -production for latest driver versions):

• ghcr.io/siderolabs/nvidia-container-toolkit-production:<version>
• ghcr.io/siderolabs/nvidia-open-gpu-kernel-modules-production:<version>

See the Talos Linux Image Factory to find the correct extension versions for a specific Talos version.

As for all Talos nodes, our custom extensions are also needed:

• ghcr.io/kommodity-io/kommodity-attestation-extension:v1.1.1 (see releases)
• ghcr.io/kommodity-io/kommodity-autobootstrap-extension:v1.6.0 (see releases)

Build the Talos image with the required extensions using this pipeline.

vLLM hosted models

The vLLM runtime needs to be configured with the correct CUDA driver version and library paths to work with the pre-installed NVIDIA drivers on Talos nodes.

As a rule of thumb, use latest driver versions with latest vLLM runtime version.

# Scaleway/Talos GPU cluster overrides
runtime:
  # Required for CUDA 13.x / driver 595.58.03 on Scaleway H100 nodes
  version: "v0.20.1"
  env:
    # Host glibc path must come first to load host CUDA driver
    # before container compat libs to avoid driver mismatch errors
    # There is a known discrepancy of driver locations between nodes on AKS and how we configure them on Talos
    - name: LD_LIBRARY_PATH
      value: "/usr/local/glibc/usr/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu"

GPU Operator and MIG

In Talos, GPU drivers are pre-installed at a non-standard path, so the GPU operator cannot manage MIG on these nodes.

Instead, deploy the following helm charts:

deviceplugin:
  migManager:
    enabled: true

🛠️ Configuring MIG on a GPU node

Apply the nvidia.com/mig.config label to a node to select a MIG configuration. The GPU operator or mig-manager will detect the label and apply it automatically.

# Creates 7 partitions of 1g.10gb each
nvidia.com/mig.config: 1g.10gb

Custom MIG configurations on Scaleway

On Scaleway we use a custom mig-manager (see mig-daemonset.yaml) that supports MIG configurations not natively available on the GPU nodes.

To add a custom MIG configuration:

1. Add the configuration to customConfigs in the nvidia-device-plugin values file:

customConfigs: |-
  # Splits into 2 partitions: one 3g.40gb and one 4g.40gb
  3g.40gb-4g.40gb:
    devices:
      - 3g.40gb: 1
      - 4g.40gb: 1

2. Label the GPU node. For asymmetric configurations (unequal partition sizes), also add nvidia.com/device-plugin.config: mig-mixed:

labels:
  nvidia.com/mig.config: 3g.40gb-4g.40gb
  nvidia.com/device-plugin.config: mig-mixed

🎛️ Scheduling GPU Workloads

Use a combination of taints/tolerations and labels/node selectors (or node affinity) to control GPU workload scheduling:

• Taints + tolerations — restrict non-GPU workloads from landing on GPU nodes.
• Labels + node selectors/affinity — force GPU workloads to land on GPU nodes.

Both are needed: tolerations alone permit scheduling on GPU nodes but don't guarantee it.

Naming conventions

Taints and labels encode the GPU model and MIG configuration using the following templates:

Examples:

🔒 Taints and Tolerations

Taint the GPU node:

workload=gpu-h100-mig:NoSchedule

Add the corresponding toleration to the workload:

tolerations:
  - key: "workload"
    operator: "Equal"
    value: "gpu-h100-mig"
    effect: "NoSchedule"

🏷️ Labels and Node Selectors

Label the GPU node:

workload: "gpu-h100-mig"
gpu-type: "h100mig"

Add the corresponding nodeSelector to the workload:

nodeSelector:
  workload: "gpu-h100-mig"
  gpu-type: "h100mig"

🎯 Node Affinity

Node affinity is more flexible than node selectors and is required for dynamically-applied labels like nvidia.com/gpu.product, which encodes the GPU model and MIG state:

# H100 with MIG disabled
nvidia.com/gpu.product: NVIDIA-H100-PCIe

# H100 with MIG enabled, 7 partitions of 1g.10gb
nvidia.com/gpu.product: NVIDIA-H100-PCIe-MIG-1g.10gb

Example node affinity targeting a specific GPU product:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: nvidia.com/gpu.product
                operator: In
                values:
                  - NVIDIA-H100-PCIe

📚 References

• MIG User Guide
• GPU Operator Documentation
• Kubernetes:
  • Taints and Tolerations
  • Labels and Selectors
  • Node Affinity