Deploying a New LLM Model

This manual walks through adding a new self-hosted LLM to an inference (GPU) cluster — e.g. dev-par02, stg-par02, prd-par02. Models are served with vLLM via the inference-service chart (a KServe InferenceService), weights are streamed from an Azure storage account (and optionally pre-cached on the node), and traffic is routed through the LLM gateway.

The worked example throughout is corti-llm-kimi-k26-nvfp4 (Kimi K2.6, NVFP4) on Scaleway B300 nodes.

Model tiers

Self-hosted LLMs are categorised into tiers by parameter count. The tier determines which virtual model alias a model is exposed under (e.g. corti-s1, corti-s1-mini, corti-s1-tiny) and helps API consumers pick the right size class.

Rules:

• Active params is the primary sort key for s1-tiny: any MoE model with <5B active parameters is s1-tiny, regardless of total parameter count.
• Total params is the primary sort key for the remaining tiers:
  • s1-ultra: 1T+ total.
  • s1: 50B–999B total with >5B active.
  • s1-mini: 20B–50B total, dense or >5B active.
• -instant variants (thinking/reasoning disabled) inherit the same tier as their base model.
• s1-ultra is reserved for 1T+ models. GLM-5.2 (~355B) currently serves as a placeholder in the s1-ultra virtual model until enough GPUs are available to deploy both GLM-5.2 and a 1T+ model simultaneously.

Naming convention

Base configs in config/services/<release-name>/ follow the pattern:

corti-<tier>-<date>

Routing keys in the AI gateway follow the pattern:

<provider>-corti-<tier>-<date>[-instant]

The <date> is the model's initial launch date in YYYYMMDD format. The -instant suffix denotes the same model with reasoning/thinking disabled via chat_template_kwargs (self-hosted) or reasoning_effort (Azure).

Examples:

How it fits together

flowchart LR
  client["client"] -->|"x-corti-model: <model.name>"| gw["LLM gateway (shared ns)"]
  gw -->|HTTPRoute| isvc["InferenceService (mlservices ns)"]
  isvc -->|"corti-models://"| azure[("Azure storage account<br/>cortipipelineartifacts")]
  azure -.->|"pre-download Job"| cache[("node-local cache<br/>/var/models")]
  isvc --> pod["vLLM predictor pod<br/>on GPU node"]
  cache -.->|"bind-mount (if cached)"| pod

• Chart: inference-service resolves a GPU profile (CPU/memory/GPU count + scheduling labels) from gpu.product, renders the KServe InferenceService, a NetworkPolicy, the gateway HTTPRoute(s), and a SecurityPolicy.
• Weights: model.source: storageAccount + model.repository: <container>/<folder> becomes a corti-models://… storage URI, pulled from the cortipipelineartifacts account (Internal subscription).
• Local cache (optional but recommended on big models): with localModelCache.enabled, KServe pre-downloads the weights once to node-local /var/models and the predictor pod bind-mounts them instead of re-streaming from Azure on every pod start — see Local model caching.
• Routing: clients select the model with the x-corti-model (or x-ai-eg-model) header whose value is model.name — see the AI Gateway guide.

Prerequisites

1. Access to the model on Hugging Face, the orca CLI installed, and an az login session. Step 1 uses orca tools model upload, which drives the Hugging Face download and the azcopy upload for you. Getting the weights into the ml-model-llm-base container is Step 1 below.
2. A GPU node pool exists in the target cluster and is Ready, with the device plugin advertising nvidia.com/gpu on it. If the node is brand new, confirm Node-feature-discovery, the device plugin and GFD actually schedule on it — see Managing NVIDIA GPUs.
3. A chart GPU profile for that node's nvidia.com/gpu.product exists in inference-service values (see Adding a GPU profile if not).
4. A vLLM build (runtime.version) that supports the model's architecture, quantization, and the node's GPU compute capability.
5. (Only if you'll use the node-local cache) the cluster's KServe localmodel controller is enabled and the localmodelnode-agent is wired up (hostPath /var/models, workload identity for the download Jobs). This is a one-time per-cluster setup — already done on dev-par02. See Local model caching.

1. Get the weights into Azure

Use orca tools model upload to get the weights into the cortipipelineartifacts/ml-model-llm-base container. It downloads the snapshot from Hugging Face first (resumable, retried until the snapshot is complete), then azcopys the files flat into ml-model-llm-base/<dest>/ and verifies the upload — it lists the container afterwards and fails if any local file is missing or size-mismatched.

It authenticates to Azure via your az login session (azcopy AZCLI mode), so log in first. For gated or rate-limited Hugging Face models, also authenticate to HF (HF_TOKEN or hf auth login).

az login   # or: az account set --subscription Internal

# Download from HF (retrying until complete) AND upload in one go. The
# destination folder defaults to the lowercased repo name (here:
# kimi-k2.6-nvfp4). Always --dry-run first to confirm the file list:
orca tools model upload --repo nvidia/Kimi-K2.6-NVFP4 --dry-run

orca tools model upload --repo nvidia/Kimi-K2.6-NVFP4

# Override the destination folder with --dest:
orca tools model upload --repo nvidia/Kimi-K2.6-NVFP4 --dest kimi-k2.6-nvfp4

# Or upload a snapshot already in the HF cache with --source (e.g. after a
# manual `hf download nvidia/Kimi-K2.6-NVFP4`):
orca tools model upload \
  --source ~/.cache/huggingface/hub/models--nvidia--Kimi-K2.6-NVFP4 \
  --dest kimi-k2.6-nvfp4

2. Inspect the model

Read the model's own config before choosing runtime args — it tells you the architecture, context length, and quantization. Use the az CLI (set the Internal subscription first):

az account set --subscription Internal

az storage blob list --account-name cortipipelineartifacts \
  --container-name ml-model-llm-base --prefix "<model-folder>/" \
  --auth-mode login --query "[].name" -o tsv

az storage blob download --account-name cortipipelineartifacts \
  --container-name ml-model-llm-base --name "<model-folder>/config.json" \
  --file /tmp/config.json --auth-mode login -o none

What to pull out of config.json / hf_quant_config.json:

3. Create the service config

Lives under config/services/<release-name>/. Put reusable settings in base/ and dev-only settings in development/. The directory name must equal the releaseName (see value precedence).

config/services/corti-llm-kimi-k26-nvfp4/base/values.yaml:

# yaml-language-server: $schema=../../../schemas/service.schema.json
corti:
  team: text-generation
  repo: corticph/charts
  description: |
    Kimi K2.6 with NVFP4 quantization. Optimized for 4x B300 (288GB) GPUs.
    Reference: https://huggingface.co/nvidia/Kimi-K2.6-NVFP4

team: text-generation

model:
  type: llm
  name: corti-llm-kimi-k2.6-nvfp4   # API id sent via x-corti-model (dots OK here)
  source: storageAccount
  format: "vllm"

runtime:
  name: "corti-vllm"
  args:
    - "--tensor-parallel-size=4"     # MUST match the GPU profile's gpu count
    - "--enable-expert-parallel"     # MoE model
    - "--kv-cache-dtype=fp8"
    - "--max-model-len=262144"
    - "--max-num-seqs=40"
    - "--max-num-batched-tokens=16384"
    - "--enable-chunked-prefill"
    - "--enable-prefix-caching"
    - "--gpu-memory-utilization=0.92"
    - "--trust-remote-code"
    - "--reasoning-parser=kimi_k2"
    - "--tool-call-parser=kimi_k2"
    - "--enable-auto-tool-choice"

gpu:
  product: "NVIDIA-B300-SXM-8-288G"   # must equal the node's nvidia.com/gpu.product

replicas:
  min: 2                              # half-node profile → 2 instances pack per B300 node
  max: 2

access:
  enabled: true

serviceInfra:
  workloadIdentity:
    enabled: false

serviceAccount:
  create: false

config/services/corti-llm-kimi-k26-nvfp4/development/values.yaml:

---
model:
  # <container>/<folder> in the storage account
  repository: ml-model-llm-base/kimi-k2.6-nvfp4

access:
  employees:
    enabled: true
  clusters:
    dev-weu:
      enabled: true

Choosing gpu.product

gpu.product does double duty in the chart: it is both the profile lookup key and the required nvidia.com/gpu.product node-affinity value. So it must simultaneously match a key in the chart's gpu.profiles map and the node's nvidia.com/gpu.product label — otherwise the pod never schedules (label mismatch) or the chart fails to render (no profile). See Adding a GPU profile.

Runtime args worth knowing

4. Create the environment app

This is what ArgoCD's ApplicationSet watches. environments/<env>/<cluster>/mlservices/apps/<release-name>/config.json:

{
  "appName": "mlservices-corti-llm-kimi-k26-nvfp4",
  "releaseName": "corti-llm-kimi-k26-nvfp4",
  "chart": {
    "repository": "corti.azurecr.io",
    "name": "inference-service",
    "tag": "1.13.0"
  },
  "destination": {
    "namespace": "mlservices"
  }
}

Add an env overlay for cluster-specific runtime settings. On Talos/Scaleway nodes vLLM needs the host driver libraries on LD_LIBRARY_PATH and a CUDA-matched runtime version (see vLLM hosted models). This is also where you enable the node-local cache. …/apps/corti-llm-kimi-k26-nvfp4/values.yaml:

---
# Scaleway/Talos B300 (Blackwell) GPU cluster overrides
runtime:
  # Required for CUDA 13.x / driver 595.71.05 on Scaleway B300 nodes
  version: "v0.20.1"
  env:
    # Host glibc path must come first to load the host CUDA driver before
    # container compat libs, to avoid driver mismatch errors. Driver locations
    # differ between AKS and how we configure Scaleway nodes.
    - name: LD_LIBRARY_PATH
      value: "/usr/local/glibc/usr/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu"
localModelCache:
  enabled: true
  modelSize: 500Gi   # ≥ the model's on-disk size; sizes the cache PV/PVC

Local model caching

Large models take a long time to stream from Azure on every pod start, which makes scale-ups and restarts slow and hammers the storage account. KServe's LocalModelCache pre-downloads the weights once to node-local /var/models (via a download Job), and the predictor pod bind-mounts them instead of re-streaming. Both LocalModelCache and LocalModelNodeGroup are cluster-scoped resources rendered by the chart.

Enable it per-model in the env overlay (Step 4):

localModelCache:
  enabled: true
  modelSize: 500Gi   # LocalModelCache.spec.modelSize — set ≥ the real on-disk size

5. Render & verify locally

Use drdep to render the merged manifests before committing:

orca tools drdep render \
  --app corti-llm-kimi-k26-nvfp4 --cluster dev-par02 --tenant mlservices

Check the rendered InferenceService matches the target node on all three scheduling gates, plus resources and storage:

• nodeSelector → gpu-type + workload from the GPU profile
• tolerations → the node's workload=gpu-<model> taint
• affinity.nodeAffinity → nvidia.com/gpu.product = your gpu.product
• resources → expected cpu / memory / nvidia.com/gpu (4 on B300)
• storageUri: corti-models://<container>/<folder>
• if caching: a LocalModelCache + LocalModelNodeGroup are rendered

To preview against the live cluster: orca tools drdep diff ….

6. Commit

Follow the repo commit conventions: feat/fix only, scope = cluster dir (or development), append the Linear ticket.

feat(development): add scaleway kimi k2.6 model [PLA-5985]

7. Smoke-test once it's live

After ArgoCD syncs and the predictor pod is Running, benchmark it with orca tools vllm-bench to confirm it actually serves and to get a throughput baseline:

orca tools vllm-bench run \
  --selector serving.kserve.io/inferenceservice=corti-llm-kimi-k26-nvfp4 \
  --namespace mlservices \
  --model corti-llm-kimi-k26-nvfp4

It port-forwards to the pod and reports requests, errors, and generation tok/s over a fixed window. errors=0 and steady gen tok/s means it's serving; then confirm gateway routing by sending x-corti-model: <model.name> through the LLM gateway.

Adding a GPU profile (chart side)

If gpu.product has no matching key in the chart, add a profile to inference-service values. The profile key must equal the node's nvidia.com/gpu.product label exactly.

The B300 profile is half-node so two tensor-parallel=4 instances pack onto one 8-GPU node:

gpu:
  profiles:
    NVIDIA-B300-SXM-8-288G:        # == node's nvidia.com/gpu.product
      cpu: 110                     # 2 × 110 = 220 of 224 vCPUs; ~4 left for daemonsets
      memory: "1850Gi"            # 2 × 1850 = 3700 of ~3779Gi allocatable; ~79Gi headroom
      gpu: nvidia.com/gpu=4        # 4 of 8 GPUs per pod; pair with --tensor-parallel-size=4
      labels:
        workload: "gpu-b300"       # must match the node taint/label
        type: "b300"

See Scheduling GPU Workloads for the taint/label/affinity naming conventions.

Troubleshooting

References

• Managing NVIDIA GPUs
• AI Gateway development guide
• inference-service chart
• vLLM engine arguments