AI Gateway – Development Guide

Public LLM Gateway

External clients (opencode, copilot-api, CLI tools) connect to the public LLM gateway. Each environment exposes its own hostname:

Environment	Endpoint
dev-weu	https://ai.dev-weu.corti.app/v1
staging-eu	https://ai.staging-eu.corti.app/v1
prod-weu	https://ai.eu.corti.app/v1
prod-us	https://ai.us.corti.app/v1
prd-par01	https://ai.fr.corti.app/v1

The public gateway authenticates requests via an API key validated by the authorization-service (Keycloak JWT exchange). Use the model field in the request body — the AI Gateway ExtProc rewrites it to the x-ai-eg-model header before routing.

curl https://ai.dev-weu.corti.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -s \
  -d '{
        "model": "corti-s1-beta",
        "messages": [
          { "role": "user", "content": "count r in strawberry and then tell me what the number is" }
        ]
      }' | jq

Available Virtual Models

Virtual models are stable aliases that route to the underlying model backends. Use these in your application config — platform teams can adjust routing, add fallbacks, or swap providers without requiring changes to your service.

Virtual model	Backend	Description
corti-s1	scaleway-corti-s1-20261026	GLM-5.3-NVFP4, Inferact checkpoint (stable)
corti-s1-alpha	scaleway-corti-s1-20261026	GLM-5.3-NVFP4, Inferact checkpoint (alpha)
corti-s1-beta	scaleway-corti-s1-20261026	GLM-5.3-NVFP4, Inferact checkpoint (beta)
corti-s1-instant	scaleway-corti-s1-20261026-instant	GLM-5.3-NVFP4, Inferact checkpoint, low-effort reasoning
corti-s1-instant-alpha	scaleway-corti-s1-20261026-instant	GLM-5.3-NVFP4, Inferact checkpoint, low-effort reasoning (alpha)
corti-s1-instant-beta	scaleway-corti-s1-20261026-instant	GLM-5.3-NVFP4, Inferact checkpoint, low-effort reasoning (beta)
corti-s1-mini	corti-s1-mini-20260831	Qwen3.8-27B-FP8 (stable)
corti-s1-mini-beta	corti-s1-mini-20260831	Qwen3.8-27B-FP8 (beta)
corti-s1-mini-instant	corti-s1-mini-20260831-instant	Qwen3.8-27B-FP8, thinking off
corti-s1-mini-instant-beta	corti-s1-mini-20260831-instant	Qwen3.8-27B-FP8, thinking off (beta)
corti-s1-tiny	corti-s1-tiny-20260707	Tiny model
corti-s1-embedding	scaleway-corti-embedding-qwen3-4b	Embedding model

Check Orca for the full, up-to-date list of models and their backends per environment.

Getting a list of available models through the Gateway

curl https://ai.dev-weu.corti.app/v1/models \
  -s \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  | jq -r ".data | .[].id"

Getting the Full Internal Model List through the Private Proxy

The public gateway exposes the virtual models enabled for each environment; query its /v1/models endpoint or check Orca for the current list. The private proxy (proxy.<env>.corti.app, authenticated with your Azure AD access token) additionally exposes the full internal catalog: every directly-addressable backend model across providers (Corti self-hosted, Azure OpenAI, Anthropic, Nebius, Google Vertex AI), plus internal ml-service models — many more than the public list.

curl -s https://proxy.dev-weu.corti.app/v1/models \
  -H "Authorization: Bearer $(az account get-access-token --query accessToken -o tsv)" \
  | jq -r ".data | .[].id"

Entries with a live backend in the target environment can also be called directly — the proxy is a standard OpenAI-compatible surface:

curl -s https://proxy.dev-weu.corti.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $(az account get-access-token --query accessToken -o tsv)" \
  -d '{
        "model": "corti-s1",
        "messages": [{ "role": "user", "content": "hello" }]
      }' | jq -r '.choices[0].message.content'

Reaching the Anthropic API Through the Gateway

The gateway proxies the Anthropic Messages API in addition to the OpenAI-compatible endpoint. This is useful for clients that speak the Anthropic protocol (e.g. the Claude SDK, anthropic Python package) rather than the OpenAI schema. The Anthropic proxy is exposed on the proxy.<cluster> hostname and authenticated with your Azure AD access token rather than an API key.

# Get auth token (Azure AD)
TOKEN=$(az account get-access-token --query accessToken -o tsv)

curl -s https://proxy.dev-weu.corti.app/anthropic/v1/messages \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -H "x-ai-eg-model: anthropic-claude-fable-5-1" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "anthropic-claude-fable-5-1",
    "max_tokens": 1024,
    "messages": [
      {"role": "user", "content": "What is the medical coding code for a routine office visit?"}
    ]
  }'

Querying the Inference Cluster Directly

You can also query the inference cluster directly using your Azure token, the `x-corti-model` header, and the `model` field in your request:

curl https://llm.stg-par02.corti.app/v1/chat/completions -X POST \
  -H "Content-Type: application/json" \
  -H "x-corti-model: corti-s1-20261026" \
  -H "Authorization: Bearer $(az account get-access-token | jq -r .accessToken)" \
  -d '{
    "model": "corti-s1-20261026",
    "messages": [
      {
        "role": "user",
        "content": "Say Hi!"
      }
    ]
  }' -s | jq -r '.choices[0].message.content'

---

Model Info (Orca)

https://orca.dev-weu.corti.app

Orca is an internal tool for looking up what models are currently deployed across environments (among other things). For each model it surfaces:

Column	Description
Name	The model identifier used in requests
Type	The type of model
Provider	The underlying provider (e.g. Corti, Azure, Nebius, etc)
Mode	Whether it's a direct or virtual model
Backends	The concrete backend(s) the model resolves to per environment (e.g. single, fallback, weighted)

---

Setting Up Your Application

Service Endpoint

There are two access paths depending on where your application runs:

In-cluster services (ml-services, platform services in the same cluster as the inference pods) point at the internal Envoy AI Gateway service. In common-chart it will be automatically injected as VLLM_API_BASE and LLM_API_BASE when models.llms.enabled: true. If not using common-chart, set the base URL in your environment variables or configuration:

http://envoy-gateway-internal-ai.shared:80/v1

External clients (opencode, CLI tools, copilot-api) use the public LLM gateway endpoint for their environment (see the table above), authenticated with an API key.

This is the standard OpenAI-compatible endpoint, so any OpenAI SDK or HTTP client can be used with minimal configuration — just swap the base URL and model name.

Specifying a Model

Include the model field in your request body. This can be either:

A direct model — routes straight to a specific provider backend:

{
  "model": "scaleway-corti-s1-20261026",
  "messages": [{ "role": "user", "content": "Hello!" }]
}

A virtual model — an alias that may include fallbacks or weighted routing across multiple backends. Use these when you want resilience or environment-agnostic behavior:

{
  "model": "corti-s1-beta",
  "messages": [{ "role": "user", "content": "Hello!" }]
}

Recommendation: Prefer virtual models in your app config. You or the platform teams can adjust routing, add fallbacks, or swap providers without requiring changes to your service environment variables or code.