AI Gateway – Merged Deployment Guide

Single-cluster topology: workload + inference live together. No public LB, no cross-cluster JWT. Self-hosted models only (no Azure / Vertex / Anthropic / Nebius).

Request flow

---
config:
  look: neo
  theme: redux
  layout: elk
---
flowchart LR
  subgraph cluster["Merged Cluster (workload + inference)"]

    subgraph workload_ns["Workload Namespaces (base, agents, ...)"]
      ml_transcription["ml-service-transcription-*"]
      ml_summarisation["ml-service-summarisation"]
      ml_llm_reasoning["ml-service-llm-reasoning"]
      ml_other["other ml-services"]
    end

    subgraph shared_ns["Namespace: shared (envoy-gateway chart)"]
      subgraph envoy_internal_gateway["Envoy Internal Gateway (ASR)"]
        ig_listener["Listener :80\nallowedRoutes: All"]
      end

      subgraph envoy_internal_ai_gateway["Envoy Internal AI Gateway (LLM)"]
        aig_listener["Listener :80\nallowedRoutes: All"]
        extproc["ExtProc Sidecar\nbody validation\nbody.model -> x-ai-eg-model"]
      end
    end

    subgraph mlservices_ns["Namespace: mlservices (one inference-service release per model)"]
      subgraph asr_inference["ASR inference-service releases"]
        asr_routes["HTTPRoute (per model)\nparentRef: envoy-gateway-internal\nmatch: x-corti-model = MODEL\nbackendRef: RELEASE-predictor"]
        asr_predictors["KServe Predictor Services\nasr-tdt-*, asr-aed-*,\nfaster-whisper-*, diarizer-*"]
      end

      subgraph llm_inference["LLM (ai-gateway-route chart + inference-service)"]
        llm_routes["AIGatewayRoute (per model)\nparentRef: envoy-ai-gateway-internal\nmatch: x-ai-eg-model = MODEL\nbackendRef: AIServiceBackend"]
        llm_aisb["AIServiceBackend + Backend\nfqdn: RELEASE-predictor.mlservices.svc.cluster.local:80"]
        llm_predictors["KServe Predictor Services\n(deployed by inference-service)\ncorti-s1-*, corti-s1-mini-*,\ncorti-s1-tiny-*, corti-llm-gen1, ..."]
      end
    end

    ml_transcription -->|http\nHeader x-corti-model = asr_model| ig_listener
    ml_other -->|http| ig_listener
    ig_listener -.->|attached via\nparentRef| asr_routes
    asr_routes -->|backendRefs Service\nsame namespace| asr_predictors

    ml_summarisation -->|OpenAI POST /v1/chat/completions\nbody.model = MODEL| aig_listener
    ml_llm_reasoning -->|OpenAI POST /v1/chat/completions\nbody.model = MODEL| aig_listener
    aig_listener --> extproc
    extproc -.->|injects header\nx-ai-eg-model = MODEL| llm_routes
    llm_routes --> llm_aisb
    llm_aisb -->|fqdn| llm_predictors

    asr_predictors@{shape: procs}
    llm_predictors@{shape: procs}

    style envoy_internal_gateway fill:#A594F9
    style envoy_internal_ai_gateway fill:#A594F9
    style ig_listener fill:#A594F9
    style aig_listener fill:#A594F9
    style extproc fill:#CBE2B5
    style asr_routes fill:#A594F9
    style llm_routes fill:#A594F9
    style llm_aisb fill:#CBE2B5
    style asr_predictors fill:#B7E0FF
    style llm_predictors fill:#B7E0FF
  end

  subgraph legend["Legend"]
    l_envoy["Envoy Gateway resource"]
    l_ai["Envoy AI Gateway resource (ExtProc)"]
    l_svc["Service / App / Predictor"]
    style l_envoy fill:#A594F9
    style l_ai fill:#CBE2B5
    style l_svc fill:#B7E0FF
  end

ASR:

1. ml-service hits envoy-gateway-internal.shared:80 with x-corti-model header set by client.
2. Plain HTTPRoute (from inference-service) matches header → forwards to <release>-predictor Service in mlservices.

LLM:

1. ml-service POSTs OpenAI body to envoy-gateway-internal-ai.shared:80/v1/chat/completions, no header.
2. AI Gateway Controller injects ExtProc sidecar (because AIGatewayRoute exists). ExtProc reads body.model, sets x-ai-eg-model header.
3. AIGatewayRoute (from ai-gateway-route chart) matches header → AIServiceBackend → Backend (fqdn = in-cluster Service DNS).
4. Forwards to <release>-predictor Service in mlservices.

No public DNS, no TLS, no JWT, no ReferenceGrant (route + Service co-located).

Chart values

envoy-gateway chart

team: platform
dnsZone: corti.app

cm:
  common:
    CLUSTER_NAME: <cluster-name> # required

platform:
  type: aks # or scaleway

# --- Disable external + public gateways ---
gateway:
  enabled: false # inference cluster external LB
publicLLMGateway:
  enabled: false # Keycloak-gated public ingress
aiGatewayProxy:
  enabled: false # JWKS-gated public proxy

# --- Enable internal gateways (ASR + LLM) ---
internalGateway:
  enabled: true
  name: envoy-gateway-internal
  service:
    type: ClusterIP
  inferenceClusterAuth:
    enabled: false # skips SA + ExternalSecret JWT plumbing

internalAIGateway:
  enabled: true
  name: envoy-ai-gateway-internal
  service:
    type: ClusterIP
  inferenceClusterAuth:
    enabled: false # skips cross-cluster JWT (ESO stack preserved)
  # No upstream provider creds — self-hosted only
  apiKeys:
    azure: { enabled: false }
    nebius: { enabled: false }
    googleVertexAI: { enabled: false }
    anthropic: { enabled: false }

# --- Controllers still required ---
ai-gateway-crds-helm:
  enabled: true
ai-gateway-helm:
  enabled: true
gateway-crds-helm:
  enabled: true
gateway-helm:
  enabled: true

# --- No ACME (external gateway dead) ---
issuers:
  letsEncryptProduction: { enabled: false }
  letsEncryptStaging: { enabled: false }

networkPolicy:
  enabled: true
  acmeHttpChallenge:
    enabled: false
  allowInternalToExternalGateway:
    enabled: false

inference-service chart (per model release, ASR + LLM)

defaults:
  gateway:
    internal:
      enabled: true # parent on envoy-(ai-)gateway-internal, skip hostname/sectionName
access:
  enabled: false # no JWT — drops SecurityPolicy + HTTPS redirect

For LLM models, the HTTPRoute rendered here is functionally superseded by the AIGatewayRoute from ai-gateway-route (header match is the same; AIGatewayRoute additionally activates ExtProc body→header rewrite). Leave both or delete the LLM HTTPRoute — keeping both is harmless.

ai-gateway-route chart (cluster-wide, LLM only)

Required for LLM path: deploys AIGatewayRoute so AI Gateway Controller wires ExtProc on envoy-ai-gateway-internal. Uses chart ≥ 3.6.0 for in-cluster mode toggles (tls.enabled: false, auth.enabled: false, model fqdn override).

asr: {} # ASR handled by inference-service

llm:
  defaults:
    auth: { enabled: false } # no BackendSecurityPolicy
    tls: { enabled: false } # no BackendTLSPolicy
    port: 80
  providers:
    cortiSelfHosted:
      quotas: -1 # no rate limit
      defaultModelSchema: { name: OpenAI, version: "v1" }
  models:
    corti-s1-tiny-20260707:
      enabled: true
      provider: cortiSelfHosted
      fqdn: corti-s1-tiny-20260707-predictor.mlservices.svc.cluster.local
      timeouts:
        request: 120s
        backendRequest: 120s
      bodyMutation:
        set:
          enabled: true
          chat_template_kwargs:
            enabled: true
            value: '{"enable_thinking": false}'
          # Add other body mutation rules like temperature, top_p, top_k, etc... if needed
            
  # Configure virtualModels to point to the model(s)
  virtualModels:
    ml-service-summarisation:
      enabled: true
      models:
        - corti-s1-tiny-20260707
    ml-service-llm-reasoning:
      enabled: true
      models:
        - corti-s1-tiny-20260707
    copilot-api-v2:
      enabled: true
      models:
        - corti-s1-tiny-20260707
    ml-service-formatting:
      enabled: true
      models:
        - corti-s1-tiny-20260707
    ml-service-voice-command:
      enabled: true
      models:
        - corti-s1-tiny-20260707
    ml-service-facts:
      enabled: true
      models:
        - corti-s1-tiny-20260707

Add one models.<name> entry per LLM, pointing at the in-cluster predictor DNS. No upstream provider credentials, no public FQDN.

What does not change

• ml-service env vars: still point at envoy-gateway-internal.shared:80 (ASR) and http://envoy-gateway-internal-ai.shared:80/v1 (LLM). No image rebuild.
• Header contract: x-corti-model (ASR, set by client). body.model (LLM, rewritten by ExtProc).
• OpenAI request shape unchanged — ExtProc still rewrites body→header.
• Predictor Services themselves (KServe <release>-predictor).

Caveats

• All workloads + data in one cluster = one jurisdiction. Not suitable when data sovereignty requires inference in a separate region.
• Inference GPU pool not shareable across multiple workload clusters.
• Verify no ml-service has hardcoded *.<cluster>.corti.app URL. Grep deployment values before cutover.