AI Gateway – Split Deployment Guide

Two-cluster topology: workload cluster (ml-services) calls a separate inference cluster (model pods) over public DNS, authenticated by a workload SA JWT validated by the inference cluster against the workload's OIDC discovery proxy. Use when data sovereignty requires inference in a different jurisdiction, or when one GPU pool serves multiple workload clusters.

Bird's-eye view

---
config:
  look: neo
  theme: redux
  layout: elk
---
flowchart LR
    subgraph inference_cluster["Inference Cluster"]
      envoy_external_gateway
      asr_models["ASR Models"]
      llm_models["LLM Models"]
    end
    subgraph envoy_external_gateway_details["Envoy External Gateway"]
      header_routing["Header Routing"]
      token_validation["Token Validation"]
    end
    subgraph cluster["Cluster"]
      envoy_internal_gateway["Envoy Internal Gateway"]
      envoy_internal_ai_gateway["Envoy Internal AI Gateway"]
      ml_service_summarization["Summarization Service"]
      ml_service_transcription["Transcription Service"]
      sa_secret["Service Account Secret"]
      azure_secret["Azure Secret"]
      oidc_discovery_proxy[OIDC Discovery Proxy]
    end

    subgraph azure["Azure"]
      azure_llm_models["LLM models"]
    end

    subgraph azure_request["Request"]
      api_token["API Token"]
    end

    subgraph envoy_external_request["Request"]
      header["Header"]
      jwt_token["JWT Token"]
    end

    envoy_internal_ai_gateway --> azure_request
    envoy_internal_ai_gateway -.- azure_secret
    azure_secret -.-> |inject| azure_request
    azure_request --> azure_llm_models

    envoy_internal_ai_gateway --> envoy_external_request
    envoy_internal_ai_gateway -.- sa_secret

    sa_secret -.-> |inject| envoy_external_request
    envoy_external_request --> envoy_external_gateway

    envoy_internal_gateway --> envoy_external_request
    envoy_internal_gateway -.- sa_secret

    envoy_external_gateway --> |routes| asr_models
    envoy_external_gateway --> |routes| llm_models

    ml_service_summarization --> |call| envoy_internal_ai_gateway
    ml_service_transcription --> |call| envoy_internal_gateway

    envoy_external_gateway -.-> |validate identity| oidc_discovery_proxy

    asr_models@{shape: procs}
    llm_models@{shape: procs}

Workload cluster = identity source (SA tokens signed by workload kube-apiserver). Inference cluster = JWT validator (fetches JWKS from workload oidc-discovery-proxy over public DNS). Azure auth orthogonal: static API key, never leaves the workload AI gateway.

ASR path – k8s resources

flowchart LR
    subgraph cluster["Cluster"]
      services["Services"]
      envoy_internal_gateway["Internal Gateway"]
      http_route["HTTP route"]
      backend["ASR Backend"]
      route_filter["Route Filter"]
      sa_token["SA Token"]
      inference_cluster_secret["Inference Cluster Secret"]
      es_secret["External Secret"]
    end
    subgraph remote["Remote"]
      inference_cluster["Inference Cluster"]
    end

    services --> |call| envoy_internal_gateway
    envoy_internal_gateway --> |resolves| http_route
    sa_token -.-> |uses| es_secret
    es_secret -.-> |creates| inference_cluster_secret
    http_route --> |route| backend
    http_route -.-> |call| route_filter
    route_filter -.-> |add auth| http_route
    inference_cluster_secret -.-> |use| route_filter

    backend -.-> |point to| inference_cluster
    envoy_internal_gateway --> |forwards request| inference_cluster

• Gateway envoy-gateway-internal (ClusterIP) — entry point for ASR clients.
• HTTPRoute matches x-corti-model=MODEL → Backend{fqdn: asr.<inferenceCluster>.corti.app:443}.
• HTTPRouteFilter (credentialInjection) attaches Authorization: Bearer <JWT>.
• ExternalSecret reads workload SA token, materializes inference-cluster-secret.
• BackendTLSPolicy validates inference cluster cert against system CAs.

LLM path – k8s resources

flowchart LR
    subgraph cluster["Cluster"]
      services["Services"]
      envoy_ai_gateway_controller["AI Gateway Controller"]
      subgraph envoy_internal_gateway["Internal AI Gateway"]
        external_processor["External Processor Side Car"]
        body_validation["Body Validation"]
        creds_injection["Creds Injection"]
        body_param_to_header["Body param to header"]
      end
      ai_route["AI Route"]
      http_route["HTTP route"]
      ai_service_backend["AI Service Backend"]
      backend["LLM Backend"]
      backend_security_policy["Backend Security Policy"]
      sa_token["SA Token"]
      inference_cluster_secret["Inference Cluster Secret"]
      es_secret["External Secret"]
      azure_api_secret["Azure API secret"]
    end
    subgraph remote["Remote"]
      inference_cluster["Inference Cluster"]
      azure["Azure"]
    end

    external_processor -.-> body_validation
    external_processor -.-> creds_injection
    external_processor -.-> body_param_to_header

    envoy_ai_gateway_controller -.-> |inject side car| envoy_internal_gateway
    envoy_ai_gateway_controller -.- |manage| ai_route
    envoy_ai_gateway_controller -.- |manage| ai_service_backend
    envoy_ai_gateway_controller -.- |manage| backend_security_policy

    ai_route -.-> |defines| http_route
    ai_service_backend -.-> |defines| backend

    services --> |call| envoy_internal_gateway
    envoy_internal_gateway -.-> |resolves| http_route
    sa_token -.-> |uses| es_secret
    es_secret -.-> |creates| inference_cluster_secret
    http_route -.-> |route| backend

    backend_security_policy -.-> |add auth| http_route
    inference_cluster_secret -.-> |use| backend_security_policy
    azure_api_secret -.-> |use| backend_security_policy

    backend -.-> |point to| inference_cluster
    backend -.-> |point to| azure
    envoy_internal_gateway --> |forwards request| inference_cluster
    envoy_internal_gateway --> |forwards request| azure

    style envoy_ai_gateway_controller fill:#CBE2B5
    style ai_route fill:#CBE2B5
    style backend_security_policy fill:#CBE2B5
    style ai_service_backend fill:#CBE2B5
    style external_processor fill:#CBE2B5
    style body_validation fill:#CBE2B5
    style creds_injection fill:#CBE2B5
    style body_param_to_header fill:#CBE2B5
    style envoy_internal_gateway fill:#A594F9
    style http_route fill:#A594F9
    style backend fill:#A594F9
    style sa_token fill:#B7E0FF
    style es_secret fill:#B7E0FF
    style azure_api_secret fill:#B7E0FF
    style inference_cluster_secret fill:#B7E0FF
    style services fill:#B7E0FF

• Gateway envoy-ai-gateway-internal (ClusterIP) with AI Gateway Controller–injected ExtProc sidecar.
• ExtProc does body validation, creds injection, and body.model → x-ai-eg-model header rewrite.
• AIGatewayRoute → HTTPRoute; AIServiceBackend → Backend.
• BackendSecurityPolicy injects inference-cluster-secret (self-hosted) or azure-api-secret (Azure).
• Backend FQDN = llm.<inferenceCluster>.corti.app or Azure endpoint.

Trust model

• Per-cluster SA token = scoped, revocable credential (delete SA or token Secret to cut access).
• Inference cluster trusts JWKS endpoints of explicitly listed workload clusters — no mTLS, no peering.
• Token rotation handled by ESO (refreshInterval: 5m).

Caveats

• Cross-cluster latency: 1 internet RTT + 2 TLS handshakes per request.
• Inference cluster outage = no models for any workload cluster.
• DNS, TLS certs, ACME challenges, public LB billing for every inference cluster hostname.
• Each new workload cluster must be added to access.clusters in every inference-service release.
• Hardcoded URLs in ml-services break sovereignty guarantees — keep callers behind envoy-(ai-)gateway-internal.