On-prem / external customer deploy — API-key remote-write

On-prem hospitals, GMR sites, and external customer clusters push metrics and logs to a central synapse-server using server-minted static bearer tokens ("API keys"). Unlike corti-managed clusters they have no corti-controlled DNS and keep their Kubernetes OIDC discovery behind a firewall, so they cannot use the OIDC path (envoyGateway.remoteClusters) where the server fetches the client's JWKS. Path 2 inverts the trust anchor: the credential is minted on the server cluster and validated against the server's own JWKS — which its Envoy can always reach.

This document covers the on-prem deploy (a docker-compose skeleton running Grafana Alloy) and the operator handoff (mint → distribute → verify). For the full design rationale, alternatives, and security analysis see PLAN-apikey-auth.md (PLA-6392). For the server-side chart values (cluster.apiKeys, envoyGateway.apiKeyJwks) see README.md and values.yaml.

How it works

A server-minted kubernetes.io/service-account-token is a non-expiring JWT signed by the server cluster's SA signing key, whose public half is served at the cluster's /openid/v1/jwks. It validates with the same Envoy SecurityPolicy JWT machinery as the OIDC path — only the JWKS source differs (the server's own issuer instead of the client's). One token per customer covers both metrics and logs: the customer mounts the same delivered token in their alloy-metrics collector (OTLP /v1/metrics → Mimir) and alloy-logs collector (OTLP /v1/logs → loki-logs Kafka → alloy-loki-consumer → Loki). The apiKeys HTTPRoute and SecurityPolicy are path-agnostic, so both flow through one sub allow rule per customer.

Every metric and log is labelled with customer, host, and environment="onprem" (set in .env, applied by an otelcol.processor.transform in each River config). These mirror the DataDog customer/host tags so on-prem telemetry is filterable the same way, and environment separates on-prem from development/staging/production in cross-environment queries. See the docker-compose skeleton for the exact labels.

HTTP routes

flowchart LR
  subgraph onprem["On-prem customer — docker-compose"]
    NE["node-exporter<br/>:9100"]
    AM["alloy-metrics<br/>scrape → OTLP/HTTP"]
    AL["alloy-logs<br/>filelog → OTLP/HTTP"]
    NE -- "/metrics" --> AM
  end

  subgraph server["Synapse server cluster (e.g. prd-par01)"]
    direction TB
    EG["Envoy Gateway<br/>https://otel.synapse.corti.app :443"]
    subgraph gw["Gateway API — shared hostname"]
      HRK["HTTPRoute otlp-apikeys<br/>match: x-corti-cluster per apiKey<br/>(pinned to HTTPS listener)"]
      HRR["HTTPRoute otlp-1..N<br/>(remoteClusters — OIDC path)"]
    end
    SPK["SecurityPolicy otlp-apikeys-auth<br/>1 provider: apikey-issuer<br/>defaultAction: Deny<br/>N rules: allow one sub per key"]
    JWKS[("Server's own JWKS<br/>oidc.SERVER.corti.app<br/>/openid/v1/jwks")]
    AG["alloy-gateway :4318<br/>OTLP receiver"]
    M[("Mimir — metrics")]
    LK[("loki-logs Kafka topic")]
    L[("Loki — logs")]
  end

  AM -- "POST /v1/metrics<br/>Authorization: Bearer &lt;JWT&gt;<br/>x-corti-cluster: onprem-gmr" --> EG
  AL -- "POST /v1/logs<br/>same bearer + header" --> EG
  EG --> HRK
  HRK --> SPK
  SPK -. "validate JWT signature" .-> JWKS
  SPK -- "sub matches an allow rule" --> AG
  AG --> M
  AG -- "otlp kafka exporter" --> LK --> L
  SPK -. "no / invalid token → 403" .-> AM

The request path:

1. The customer's alloy sends POST https://otel.synapse.corti.app/v1/metrics (or /v1/logs) with Authorization: Bearer <JWT> and x-corti-cluster: <name>.
2. The x-corti-cluster header selects the apiKeys HTTPRoute (one rule per key, pinned to the HTTPS listener so the bearer is never accepted on port 80).
3. The apiKeys SecurityPolicy (defaultAction: Deny) validates the JWT against the server's own JWKS and allows the request only if its sub equals system:serviceaccount:<ns>:synapse-server-<name> — one allow rule per key.
4. alloy-gateway receives OTLP on :4318 and fans metrics → Mimir (remote_write) and logs → loki-logs Kafka → alloy-loki-consumer → Loki.
5. No token, an invalid signature, or a revoked sub → 403 at the gateway; the request never reaches alloy-gateway.

The apiKeys HTTPRoute + SecurityPolicy are a dedicated pair (one apikey-issuer provider for all keys, N allow rules) so they do not consume the ≤4-provider sharding slots used by remoteClusters.

1. Mint the key (server side, via GitOps)

A Corti operator adds the customer to the server overlay and merges the PR:

# deployments/environments/<env>/<server-cluster>/<ns>/apps/synapse-server/values.yaml
envoyGateway:
  enabled: true
  hostname: "otel.synapse.corti.app"
  apiKeyJwks:                         # this server's OWN issuer; REQUIRED
    host: oidc.<server-alias>.corti.app
    path: openid/v1/jwks

cluster:
  apiKeys:
    onprem-gmr: true                   # mint SA + token Secret; authorize its sub

ArgoCD syncs; the chart creates — in the server cluster's namespace — a ServiceAccount and a Secret, both named synapse-server-<name>. Kubernetes' token controller signs a long-lived JWT into Secret.data.token. Nothing secret is in git — only the line onprem-gmr: true. Validation fails fast at render time if envoyGateway.enabled is false or apiKeyJwks.host is empty.

2. Read the token out of the cluster (once, by an operator)

Point kubectl at the server cluster and decode the token:

# server cluster, e.g. prd-par01
kubectl -n monitoring-system get secret synapse-server-onprem-gmr \
  -o jsonpath='{.data.token}' | base64 -d

This prints the raw JWT (eyJ...). base64 -d does not append a trailing newline, so the output is safe to redirect into the token file (see Store and deliver out-of-band for the printf / no-newline rule). It is a bearer token — anyone holding it can push metrics and logs as onprem-gmr, so treat it like a password. Sanity check (optional): cut -d. -f2 <<<$TOKEN | base64 -d and confirm "sub": "system:serviceaccount:monitoring-system:synapse-server-onprem-gmr" — that sub is exactly the string the SecurityPolicy allow rule matches.

3. Store and deliver out-of-band

Put the token in 1Password (customer-specific item) and deliver it to the customer over a secure channel. Never over email/Slack/Linear in plaintext. The token never goes back into git or any Corti-side ConfigMap.

Writing the token file — no trailing newline. The token is a raw JWT (eyJ...) that goes verbatim into the Authorization: Bearer <token> header. A trailing newline (\n) makes the header invalid: Go's net/http rejects it client-side with invalid header field value and no request is ever sent. The River config reads the file as-is (no trimming), so the file must be clean.

• Do use printf (no trailing newline): printf '%s' "$TOKEN" > secrets/token
• Do NOT use echo (appends \n): echo "$TOKEN" > secrets/token
• Pasting into an editor is fine if the editor doesn't auto-add a final newline (most terminal editors do not, but vim/nano in some modes do — verify with xxd secrets/token | tail -1; the last byte should be a } or alnum, not 0a).

Verify: wc -c secrets/token — the byte count should match the JWT length exactly (no extra byte for \n).

4. Customer deploy — docker-compose skeleton

The on-prem site is not Kubernetes, so we do not use the synapse-client Helm chart. Instead run two Grafana Alloy containers — alloy-metrics (scrapes a node-exporter sidecar, pushes OTLP metrics) and alloy-logs (tails host log files, pushes OTLP logs) — both presenting the same delivered bearer token.

Drop these four files into a directory on the on-prem host:

onprem-synapse/
├── .env                 # endpoint + apiKey name (from .env.example)
├── docker-compose.yml
├── alloy-metrics.river
├── alloy-logs.river
└── secrets/token        # the delivered JWT, chmod 0600 (gitignored)

.env.example

# Public Synapse OTLP endpoint (the server's envoyGateway.hostname).
SYNAPSE_ENDPOINT=https://otel.synapse.corti.app

# The apiKey name you were onboarded under — your x-corti-cluster header value.
# Also used as the `cluster` label on every metric and log.
SYNAPSE_CLUSTER=onprem-gmr

# ── Labels (applied to every metric AND log) ──────────────────────────────────
# These mirror the DataDog `customer` / `host` tags so on-prem telemetry is
# filterable the same way. Set them once here; both alloy containers read them.

# Customer / site identity, e.g. nursenav-onprem, scottishas-onprem.
SYNAPSE_CUSTOMER=onprem-gmr

# Specific host within the site, e.g. ddccorti01, scottishas-onprem-avaya.
SYNAPSE_HOST=ddccorti01

# Environment. Fixed to "onprem" for this path so on-prem is distinguishable
# from development/staging/production in cross-environment queries.
SYNAPSE_ENVIRONMENT=onprem

Copy to .env and fill in. The token is not in .env — it lives in secrets/token (mounted read-only into each alloy container), so .env can be committed and secrets/ kept out of git.

docker-compose.yml

# On-prem metrics + logs shipper to Corti Synapse (Path 2 — API keys).
# Requires a delivered bearer JWT in ./secrets/token (chmod 0600; never commit).
# Configure endpoint + cluster name in .env (copy from .env.example).
#
# Images: grafana/alloy matches the server's alloy version (chart appVersion).
# Mirror to the customer's registry if they have no Docker Hub access.

services:
  node-exporter:
    image: prom/node-exporter:v1.8.2   # pin to a recent stable release; bump as needed
    # Not network_mode: host — alloy is on the default bridge network and
    # reaches node-exporter via compose service DNS (node-exporter:9100).
    # host mode is only needed for host-network scraping on Linux; the proc/sys
    # bind mounts below expose the host's /proc and /sys regardless of network
    # mode. Publish 9100 in case you want to scrape it from the host too.
    ports:
      - "127.0.0.1:9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - --path.procfs=/host/proc
      - --path.sysfs=/host/sys
      - --path.rootfs=/rootfs
    restart: unless-stopped

  alloy-metrics:
    image: grafana/alloy:v1.17.0
    command:
      - run
      - /etc/alloy/config.river
      - --server.http.listen-addr=0.0.0.0:12345
      - --stability.level=generally-available
    # Map host.docker.internal → host-gateway so alloy (bridge network) can
    # reach host-local scrape endpoints (e.g. Avaya actuator on localhost:8080).
    # Required on Linux (Docker Desktop resolves this automatically).
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./alloy-metrics.river:/etc/alloy/config.river:ro
      - ./secrets/token:/etc/alloy/token:ro
    env_file: .env
    depends_on: [node-exporter]
    restart: unless-stopped
    # Publish the Alloy UI to the host for local debugging (scrape targets,
    # exporter send/error counters). Uncomment to access http://localhost:12345.
    # ports:
    #   - "127.0.0.1:12345:12345"

  alloy-logs:
    image: grafana/alloy:v1.17.0
    command:
      - run
      - /etc/alloy/config.river
      - --server.http.listen-addr=0.0.0.0:12346
      # otelcol.receiver.filelog is "public-preview", so the logs collector
      # must run at public-preview (the metrics collector stays GA — it uses
      # only GA components).
      - --stability.level=public-preview
    volumes:
      - ./alloy-logs.river:/etc/alloy/config.river:ro
      - ./secrets/token:/etc/alloy/token:ro
      - /var/log:/var/log:ro            # host logs — adjust to your syslog paths
      - /etc/machine-id:/etc/machine-id:ro
    env_file: .env
    restart: unless-stopped
    # Publish the Alloy UI to the host for local debugging (filelog targets,
    # exporter send/error counters). Uncomment to access http://localhost:12346.
    # ports:
    #   - "127.0.0.1:12346:12346"

alloy-metrics.river

// Host metrics → Synapse (OTLP/HTTP, bearer API key).
// SYNAPSE_ENDPOINT / SYNAPSE_CLUSTER come from .env (docker-compose env_file).
// The bearer token is read from /etc/alloy/token (mounted from secrets/token).

local.file "synapse_token" {
  filename = "/etc/alloy/token"
}

otelcol.auth.bearer "synapse" {
  token = local.file.synapse_token.content
}

// ── Labels: customer / host / environment ────────────────────────────────────
// These three resource attributes become the queryable labels you filter on in
// Mimir (metrics):
//   customer    — the on-prem customer/site identity, e.g. "nursenav-onprem",
//                 "scottishas-onprem". Mirrors the DataDog `customer` tag.
//   host        — the specific host within the site, e.g. "ddccorti01",
//                 "scottishas-onprem-avaya". Mirrors the DataDog `host` tag.
//   environment — always "onprem" for this path, so on-prem telemetry is
//                 distinguishable from development/staging/production in
//                 cross-environment queries.
//
// All three are read from .env (docker-compose env_file) so the operator sets
// them in one place without editing the River config.
//
// Why a transform processor and not scrape relabeling: these are resource
// attributes, not scrape labels, so they apply to every signal uniformly and
// survive the server's OTLP→Prometheus conversion as Prom labels. The server's
// alloy-gateway already upserts `cluster` from the x-corti-cluster header;
// this processor adds the customer/host/environment granularity on top. The
// same three labels are applied to logs in alloy-logs.river so the names match
// across signals.
//
// Set on BOTH the resource (so target_info carries them) AND the datapoint
// (so node_* and other scraped series carry them as Prom labels) — the
// server's otelcol.exporter.prometheus runs with
// resource_to_telemetry_conversion=false (the OTel default), so resource
// attrs do NOT become Prom labels on datapoint metrics; only datapoint attrs
// do. Without the datapoint context, only target_info shows in Mimir under
// these labels.
otelcol.processor.transform "labels" {
  error_mode = "ignore"

  metric_statements {
    context = "resource"
    statements = [
      string.format("set(attributes[\"customer\"], %q)", sys.env("SYNAPSE_CUSTOMER")),
      string.format("set(attributes[\"host\"], %q)", sys.env("SYNAPSE_HOST")),
      string.format("set(attributes[\"environment\"], %q)", sys.env("SYNAPSE_ENVIRONMENT")),
    ]
  }

  metric_statements {
    context = "datapoint"
    statements = [
      string.format("set(attributes[\"customer\"], %q)", sys.env("SYNAPSE_CUSTOMER")),
      string.format("set(attributes[\"host\"], %q)", sys.env("SYNAPSE_HOST")),
      string.format("set(attributes[\"environment\"], %q)", sys.env("SYNAPSE_ENVIRONMENT")),
    ]
  }

  output {
    metrics = [otelcol.processor.batch.synapse.input]
  }
}

otelcol.processor.batch "synapse" {
  // Cap batch size to avoid HTTP 413 (payload too large) at the receiver.
  // The OTLP exporter defaults send_batch_max_size to 0 (no limit); a single
  // node-exporter scrape or a large log burst would then build one oversized
  // batch and be rejected. send_batch_size flushes early at 4096 items,
  // send_batch_max_size is the hard cap; timeout=2s bounds latency. These
  // match the synapse-client chart's batch processor (size: 4096,
  // maxSize: 8192, timeout: 2s) so on-prem and in-cluster clients behave
  // the same at the gateway.
  timeout              = "2s"
  send_batch_size      = 4096
  send_batch_max_size  = 8192
  output {
    metrics = [otelcol.exporter.otlphttp.synapse.input]
  }
}

otelcol.exporter.otlphttp "synapse" {
  client {
    endpoint = sys.env("SYNAPSE_ENDPOINT")
    auth     = otelcol.auth.bearer.synapse.handler
    headers  = {
      "x-corti-cluster" = sys.env("SYNAPSE_CLUSTER"),
    }
  }
  retry_on_failure {
    enabled          = true
    initial_interval = "5s"
    max_interval     = "30s"
    max_elapsed_time = "5m"
  }
  sending_queue {
    enabled       = true
    queue_size    = 1000
    num_consumers = 10
  }
}

// Bridge Prometheus-style scrape output → OTLP metrics, then through the
// labels transform (customer/host/environment) before batching.
otelcol.receiver.prometheus "synapse" {
  output {
    metrics = [otelcol.processor.transform.labels.input]
  }
}

// Scrape the node-exporter sidecar for node_* metrics (CPU, disk, net, mem, fs).
// alloy is on the default bridge network — reach node-exporter via the compose
// service DNS name (node-exporter:9100), not 127.0.0.1.
prometheus.scrape "node_exporter" {
  targets = [
    {"__address__" = "node-exporter:9100"},
  ]
  scrape_interval = "60s"
  forward_to      = [otelcol.receiver.prometheus.synapse.receiver]
}

alloy-logs.river

// Host logs → Synapse (OTLP/HTTP, bearer API key).
// Tail files under /var/log (mounted read-only from the host). Adjust `include`
// to your syslog/journald paths. `start_at = "end"` collects only new lines;
// use "beginning" to backfill existing files.

local.file "synapse_token" {
  filename = "/etc/alloy/token"
}

otelcol.auth.bearer "synapse" {
  token = local.file.synapse_token.content
}

// ── Labels: customer / host / environment ────────────────────────────────────
// These three resource attributes become the stream labels you filter on in
// Loki (logs):
//   customer    — the on-prem customer/site identity, e.g. "nursenav-onprem",
//                 "scottishas-onprem". Mirrors the DataDog `customer` tag.
//   host        — the specific host within the site, e.g. "ddccorti01",
//                 "scottishas-onprem-avaya". Mirrors the DataDog `host` tag.
//   environment — always "onprem" for this path, so on-prem telemetry is
//                 distinguishable from development/staging/production in
//                 cross-environment queries. The server's Loki is already
//                 configured to index `environment` as a stream label.
//
// All three are read from .env (docker-compose env_file) so the operator sets
// them in one place without editing the River config.
//
// The same three labels are applied to metrics in alloy-metrics.river so the
// names match across signals. The server's alloy-gateway upserts `cluster`
// from the x-corti-cluster header; this processor adds the
// customer/host/environment granularity on top.
otelcol.processor.transform "labels" {
  error_mode = "ignore"

  log_statements {
    context = "resource"
    statements = [
      string.format("set(attributes[\"customer\"], %q)", sys.env("SYNAPSE_CUSTOMER")),
      string.format("set(attributes[\"host\"], %q)", sys.env("SYNAPSE_HOST")),
      string.format("set(attributes[\"environment\"], %q)", sys.env("SYNAPSE_ENVIRONMENT")),
    ]
  }

  output {
    logs = [otelcol.processor.batch.synapse.input]
  }
}

otelcol.processor.batch "synapse" {
  // Cap batch size to avoid HTTP 413 (payload too large) at the receiver.
  // The OTLP exporter defaults send_batch_max_size to 0 (no limit); a large
  // log burst would then build one oversized batch and be rejected.
  // send_batch_size flushes early at 4096 records, send_batch_max_size is
  // the hard cap; timeout=2s bounds latency. These match the synapse-client
  // chart's batch processor (size: 4096, maxSize: 8192, timeout: 2s) so
  // on-prem and in-cluster clients behave the same at the gateway.
  timeout              = "2s"
  send_batch_size      = 4096
  send_batch_max_size  = 8192
  output {
    logs = [otelcol.exporter.otlphttp.synapse.input]
  }
}

otelcol.exporter.otlphttp "synapse" {
  client {
    endpoint = sys.env("SYNAPSE_ENDPOINT")
    auth     = otelcol.auth.bearer.synapse.handler
    headers  = {
      "x-corti-cluster" = sys.env("SYNAPSE_CLUSTER"),
    }
  }
  retry_on_failure {
    enabled          = true
    initial_interval = "5s"
    max_interval     = "30s"
    max_elapsed_time = "5m"
  }
  sending_queue {
    enabled       = true
    queue_size    = 1000
    num_consumers = 10
  }
}

// OTel-native filelog receiver: tails files and emits OTLP log records, then
// through the labels transform (customer/host/environment) before batching.
otelcol.receiver.filelog "host_logs" {
  include          = ["/var/log/*.log", "/var/log/syslog", "/var/log/messages"]
  start_at         = "end"
  include_file_name = true
  include_file_path = true
  output {
    logs = [otelcol.processor.transform.labels.input]
  }
}

Start

mkdir -p secrets
printf '%s' "$TOKEN" > secrets/token   # write the JWT WITHOUT a trailing newline (see "Writing the token file" above)
chmod 0600 secrets/token
cp .env.example .env          # edit endpoint, cluster, customer, host, environment
docker compose up -d
docker compose logs -f alloy-metrics alloy-logs

The alloy UI listens on :12345 (metrics) / :12346 (logs) inside the container — useful for confirming scrape targets and the OTLP exporter's send success/error counters. It is not published to the host by default; uncomment the ports: lines in docker-compose.yml and docker compose up -d to browse http://localhost:12345 / http://localhost:12346. Without publishing, you can still reach the UI from the host with docker compose exec alloy-metrics curl -s localhost:12345.

Scraping application metrics (Prometheus / OpenMetrics endpoints)

The skeleton above scrapes only node-exporter (host metrics). Many on-prem apps also expose a Prometheus/OpenMetrics endpoint. To ship those metrics too, add a prometheus.scrape block to alloy-metrics.river for each endpoint — they flow through the same labels transform (customer/host/environment) and batch as the node-exporter scrape, so the dashboard can filter them by the same labels.

DataDog → Alloy mapping. The on-prem sites currently scrape these endpoints with the DataDog openmetrics check. The translation is straightforward:

Avaya Call Recorder

The Avaya recorder exposes Spring Boot Actuator on ports 8080-8083 (one per recorder instance). The instance label Prometheus sets from the scrape target becomes the endpoint dashboard variable:

// Avaya Call Recorder actuator endpoints — one scrape target per recorder
// instance. The recorders publish actuator on the host's localhost ports; alloy
// (bridge network) reaches them via host.docker.internal (requires
// extra_hosts: host.docker.internal:host-gateway in docker-compose.yml).
prometheus.scrape "avaya_recorder" {
  targets = [
    {"__address__" = "host.docker.internal:8080"},
    {"__address__" = "host.docker.internal:8081"},
    {"__address__" = "host.docker.internal:8082"},
    {"__address__" = "host.docker.internal:8083"},
  ]
  scrape_interval = "60s"
  metrics_path    = "/actuator/prometheus"
  forward_to      = [otelcol.receiver.prometheus.synapse.receiver]
}

The instance label will be host.docker.internal:8080 etc. The Grafana dashboard's endpoint variable filters on instance, so each recorder instance is selectable. Relabel to a cleaner instance if you prefer (e.g. ddccorti01:8080) via a relabel_config — not required for the dashboard to work.

CART recorder

The CART recorder exposes a /metrics endpoint on port 9876. Unlike Avaya it exports a mix of metric families (cart.*, drachtio.*, rtpengine.*, siprec_client.*, plus Go runtime metrics). The DataDog config filters to those families; in Alloy the simplest approach is to scrape everything and let the dashboard query what it needs:

// CART recorder — /metrics on port 9876. host.docker.internal bridges to the
// host's localhost (requires extra_hosts in docker-compose.yml).
prometheus.scrape "cart_recorder" {
  targets = [
    {"__address__" = "host.docker.internal:9876"},
  ]
  scrape_interval = "60s"
  metrics_path    = "/metrics"
  forward_to      = [otelcol.receiver.prometheus.synapse.receiver]
}

To filter at scrape time (equivalent to the DataDog metrics list), insert a prometheus.relabel component between the scrape and the OTel receiver. Alloy uses a separate component for this — not inline metric_relabel_config like Prometheus YAML:

// Optional: keep only the metric families the DataDog config collected.
// Prometheus replaces dots in metric names with underscores, so DataDog's
// "cart.+" becomes "cart_+" here.
prometheus.relabel "cart_filter" {
  forward_to = [otelcol.receiver.prometheus.synapse.receiver]

  rule {
    action        = "keep"
    source_labels = ["__name__"]
    regex         = "cart_.+|drachtio_.+|rtpengine_.+|siprec_client_.+|go_goroutines|go_threads|go_gc_duration_seconds|process_open_fds"
  }
}

prometheus.scrape "cart_recorder" {
  targets = [
    {"__address__" = "host.docker.internal:9876"},
  ]
  scrape_interval = "60s"
  metrics_path    = "/metrics"
  forward_to      = [prometheus.relabel.cart_filter.receiver]
}

Collecting Docker container logs

The skeleton's alloy-logs.river tails /var/log (syslog). On-prem apps running as Docker containers write logs to /var/lib/docker/containers/<id>/ as JSON files. To ship those (so the dashboard's log widgets — container restarts, "Read Timed Out", AES events — work), add a local.file_match + otelcol.receiver.filelog block:

// Docker container logs — JSON-file logging driver. The host's
// /var/lib/docker/containers is mounted read-only into the alloy container
// (add it to docker-compose.yml's alloy-logs volumes). Each container logs to
// <id>/<id>-json.log.
otelcol.receiver.filelog "docker" {
  include           = ["/var/lib/docker/containers/*/*-json.log"]
  start_at          = "end"
  include_file_name = false
  include_file_path = true

  // Parse the Docker JSON envelope; extract container_id from the file path.
  operators = [
    { type = "json" },
    {
      type  = "regex",
      id    = "extract_container",
      regex = "/var/lib/docker/containers/(?P<container_id>[^/]+)/.*-json\\.log$",
      on_error = "send",
      parse_from = "log.file.path",
      parse_to   = "attributes.container_id",
    },
  ]

  output {
    logs = [otelcol.processor.transform.labels.input]
  }
}

And add the mount to docker-compose.yml's alloy-logs service:

    volumes:
      - ./alloy-logs.river:/etc/alloy/config.river:ro
      - ./secrets/token:/etc/alloy/token:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro   # Docker logs
      - /etc/machine-id:/etc/machine-id:ro

The service label (for the dashboard's log queries like service:avaya-call-recorder) comes from the Docker container's log stream; alloy's filelog receiver preserves it as a log attribute. The labels transform adds customer/host/environment on top, so log queries filter by {customer="...",host="..."} |= "Read Timed Out".

Kubernetes on-prem? If a customer does run Kubernetes, they can use the existing synapse-client Helm chart unchanged: its OTLP destination already supports auth.type: bearerToken with bearerTokenFile and extraHeaders. No chart modification is needed — mount the delivered token as a Secret and set extraHeaders.x-corti-cluster: <name>. The docker-compose path above is for the non-K8s case.

5. Verify end-to-end (metrics AND logs)

From the on-prem host (or any curl with the token), confirm:

1. Without the token → server Envoy returns 403 (defaultAction: Deny).

   curl -i -H "x-corti-cluster: onprem-gmr" \
     https://otel.synapse.corti.app/v1/metrics -X POST -d '{}'
   # 403
   

2. With the token → 200; the customer's metrics appear in Mimir (filter by cluster="onprem-gmr", customer="onprem-gmr", host="ddccorti01", environment="onprem"), and their logs appear in Loki (filter by the same {customer=...}, {host=...}, {environment="onprem"} stream labels).

   curl -i -H "Authorization: Bearer $(cat secrets/token)" \
        -H "x-corti-cluster: onprem-gmr" \
        -H "Content-Type: application/json" \
        https://otel.synapse.corti.app/v1/metrics -X POST -d '{}'
   # 200
   

Lifecycle

Rotating a key (operational runbook)

Why "delete the Secret + ArgoCD recreates it" does NOT rotate

The natural instinct is: delete the synapse-server-<name> Secret on the server cluster, let ArgoCD self-heal recreate it, read the new token, hand it over. That mints a new JWT string, but it does not invalidate the old one at the gateway, so it is not a rotation:

• The token's sub is system:serviceaccount:<ns>:synapse-server-<name>. It is derived from the ServiceAccount name, not the Secret. Recreating the Secret under the same SA produces a JWT with the same sub.
• Both the old and new JWT are signed by the same cluster service-account signing key, whose public half is published at the server's JWKS.
• Envoy's SecurityPolicy validates the JWT by signature + sub claim only. It does not call the Kubernetes API to check whether the bound Secret still exists. (That bound-token revocation check only applies to Kubernetes API server auth in 1.24+ — Envoy is not the K8s API; it is pure JWT validation.)

So the old token still has a valid signature and a sub that is still in an allow rule → it is still 200, not 403. You have minted a duplicate, not rotated a credential. The only things that invalidate a long-lived apiKey token at the gateway are:

1. Remove its sub from the allow rules — i.e. remove the key name from cluster.apiKeys (or set it false). This is the revoke lever.
2. Rotate the cluster's service-account signing key — rare, disruptive (it rotates every projected token in the cluster too). Not a per-customer lever.

Rotation therefore means changing the name (which changes the sub), not re-rolling the token of the same name. Use overlapping keys for zero downtime.

Overlapping-keys rotation (zero downtime)

sequenceDiagram
  participant Op as Corti operator
  participant Git as Server GitOps
  participant K8s as Server cluster
  participant Cust as Customer (docker-compose)
  participant GW as Server Envoy

  Note over Op,K8s: Phase A — mint the successor
  Op->>Git: add onprem-gmr-v2: true (keep onprem-gmr: true)
  Git->>K8s: ArgoCD sync — create SA + Secret synapse-server-onprem-gmr-v2
  Op->>K8s: kubectl get secret synapse-server-onprem-gmr-v2 -o jsonpath='{.data.token}' | base64 -d
  Op->>Cust: deliver new token over secure channel

  Note over Cust,GW: Phase B — customer cuts over (overlap window)
  Cust->>Cust: replace secrets/token, set SYNAPSE_CLUSTER=onprem-gmr-v2, docker compose up -d
  Cust->>GW: POST /v1/metrics with v2 token, x-corti-cluster: onprem-gmr-v2
  GW-->>Cust: 200 (v2 sub is allowed)
  Note over Cust,GW: old token still 200 too — overlap is what gives zero downtime

  Note over Op,Git: Phase C — retire the predecessor
  Op->>Git: set onprem-gmr: false (or remove the line)
  Git->>K8s: ArgoCD sync — delete SA + Secret synapse-server-onprem-gmr, drop its allow rule
  Note over GW: onprem-gmr sub no longer in any rule
  Cust->>GW: any straggler request with the old token
  GW-->>Cust: 403 (old sub is now denied)

Concrete steps:

A. Mint the successor key (server GitOps). Add a new entry alongside the existing one — keep the old key enabled during the overlap:

cluster:
  apiKeys:
    onprem-gmr: true       # keep enabled during cutover
    onprem-gmr-v2: true    # successor — new SA, new sub, new token

ArgoCD syncs; a new ServiceAccount + Secret named synapse-server-onprem-gmr-v2 are created in the server namespace. Read the new token once:

kubectl -n monitoring-system get secret synapse-server-onprem-gmr-v2 \
  -o jsonpath='{.data.token}' | base64 -d

Deliver it to the customer over a secure channel. Sanity-check its sub is system:serviceaccount:monitoring-system:synapse-server-onprem-gmr-v2.

B. Customer cuts over (overlap window). The customer replaces secrets/token with the new JWT and updates .env:

SYNAPSE_CLUSTER=onprem-gmr-v2

Then restarts the collectors so they re-read the token file and pick up the new header value:

docker compose up -d   # re-creates containers with the new env + token

During this window both tokens are 200 at the gateway (both subs are in allow rules), so there is no ingest gap even if the cutover takes minutes or a collector restart is staggered.

C. Retire the predecessor (server GitOps). Once the customer confirms the onprem-gmr-v2 token is pushing (metrics + logs visible in Mimir/Loki under the new cluster label), remove or disable the old key:

cluster:
  apiKeys:
    onprem-gmr: false    # or drop the line entirely
    onprem-gmr-v2: true

ArgoCD syncs; Helm deletes the old ServiceAccount + Secret, and the old sub is dropped from the apiKeys SecurityPolicy allow rules. From that moment any request with the old token is 403 at the gateway — even though the old JWT's signature is still valid, its sub is no longer authorized. The old token is now effectively dead; the customer can delete their old copy.

Want to keep the same customer-facing name across rotations? You can't, for as long as tokens are long-lived (non-expiring) and authorized by sub: two tokens with the same sub are indistinguishable to the gateway, so the only way to invalidate one is to drop the sub — which drops both. The -v2/-v3 suffix IS the rotation. If a stable name with real per-token revocation becomes a hard requirement, the follow-up is short-lived projected tokens with an exp that Envoy enforces (noted in PLAN-apikey-auth.md alternatives) — out of scope here.

Security notes

• Least privilege — the minted SA has no RBAC; the token is only meaningful as an ingest identity at the gateway.
• x-corti-cluster is routing, not auth — spoofing the header cannot bypass the SecurityPolicy; authorization is on the JWT sub, and a request with the header but no valid token hits defaultAction: Deny.
• Per-customer blast radius — one token = one sub = one allow rule; compromise of one customer's token never grants another's identity, and revocation is a one-line values change.
• HTTPS only — the apiKeys HTTPRoute pins parentRefs.sectionName to the HTTPS listener, so the bearer is never accepted on the plaintext port-80 listener. The remoteJWKS.uri is always composed as https://….
• Token at rest on-prem — keep secrets/token chmod 0600, never commit it, and restrict the docker-compose host's access. The token is a live credential.