Documentation navigation
Synapse MCP Agent Skill
A ready-to-paste agent skill that teaches your AI coding harness how to query Corti's Mimir (metrics) and Loki (logs) through the Synapse MCP — the label namespaces, verified label values, and battle-tested query patterns. It exists so the agent doesn't waste tokens guessing label names or running empty selectors.
The same content powers the synapse-mcp skill installed locally; this page is
the canonical copy to paste into any harness that supports agent skills
(OpenCode, Claude Code, etc.).
Install in OpenCode
OpenCode auto-loads skills from ~/.config/opencode/skills/<name>/SKILL.md.
mkdir -p ~/.config/opencode/skills/synapse-mcp
Then save the block below as
~/.config/opencode/skills/synapse-mcp/SKILL.md. Restart OpenCode.
The MCP server itself must also be configured — see Configuring the Synapse MCP.
Install in other harnesses
For harnesses without a skills directory (e.g. Claude Code via CLAUDE.md),
paste the block below into your project or global rules file. The frontmatter is
harmless if your harness ignores it.
The skill
The block below is the complete skill, including frontmatter. Use the copy button (top-right of the block) to grab it whole.
---
name: synapse-mcp
description: Use when querying Corti observability data — Mimir (Prometheus metrics), Loki (logs), or Grafana dashboards — via the synapse MCP tools. Triggers on: metrics, logs, Mimir, Loki, Grafana, Synapse, PromQL, LogQL, p95/latency, error rate, pod/deployment/namespace, dashboards, alerts, or any "query/show me" request about production system behavior. Provides the label namespaces, verified label values, and battle-tested query patterns so you don't waste tokens on empty selectors.
---
# Synapse MCP — querying Corti Mimir & Loki
Two datasources behind the Grafana MCP, both read-only:
| Store | datasource UID | type | for |
|---|---|---|---|
| Mimir | `mimir` | prometheus | metrics (PromQL) |
| Loki | `loki` | loki | logs (LogQL) |
MCP tool families: `synapse_query_prometheus` / `synapse_query_prometheus_histogram` / `synapse_list_prometheus_*` for Mimir; `synapse_query_loki_logs` / `synapse_query_loki_stats` / `synapse_query_loki_patterns` / `synapse_list_loki_label_*` for Loki; `synapse_search_dashboards` / `synapse_get_dashboard_*` for Grafana.
## The #1 gotcha: Loki and Mimir use DIFFERENT label names for the same thing
This is the single most common reason a query returns nothing. The labels are NOT interchangeable.
| concept | Loki stream label | Mimir/PromQL label |
|---|---|---|
| cluster | `k8s_cluster_name` | `cluster` (also `k8s_cluster_name` exists but `cluster` is standard) |
| namespace | `k8s_namespace_name` | `namespace` |
| container | `k8s_container_name` | `container` |
| deployment | `k8s_deployment_name` | `deployment` |
| statefulset | `k8s_statefulset_name` | `statefulset` |
| daemonset | `k8s_daemonset_name` | `daemonset` |
| service | `service_name` | `job` (usually) or `service` |
| pod | (rarely indexed) | `pod` |
Rule of thumb: **Loki labels are `k8s_`-prefixed; Mimir labels are not.** If a Mimir selector returns empty, you probably used `k8s_cluster_name` instead of `cluster` (or vice versa for Loki).
## Always discover before querying
Don't guess metric or label names — the stack has ~4500+ series and 4500+ targets. Cheap discovery first:
1. **Mimir metric names** — `synapse_list_prometheus_metric_names` with a `regex` filter (e.g. `vllm.*` , `.*duration.*bucket`). Call this FIRST before writing PromQL; metric names are not obvious.
2. **Label names that exist** — `synapse_list_prometheus_label_names` / `synapse_list_loki_label_names`. Confirms a label is real before you filter on it.
3. **Label values** — `synapse_list_prometheus_label_values` / `synapse_list_loki_label_values`. Confirms the exact spelling of a cluster/namespace/service.
4. **Cheap log existence check** — `synapse_query_loki_stats` (index-level, no line scan) before `synapse_query_loki_logs`. If `entries` is 0, the stream is empty — don't run an expensive log query.
## Loki — indexed labels & verified values
Loki index labels (the ones inside `{...}` stream selectors). Verified live 2026-08-21.
### `environment`
`development`, `production`, `staging`
> NOTE: Loki only indexes 3 environment values. Mimir has 6 (adds `dev`, `prd`, `sandbox`). On-prem and sandbox clusters may have metrics in Mimir but no logs in Loki — if a Loki query on `environment="sandbox"` returns nothing, that's why. For on-prem/sandbox logs, filter by `k8s_cluster_name` directly instead.
### `k8s_cluster_name`
Cluster generations (see wiki for full context):
- **gen1 (on-prem/customer)**: `bellingham`, `bostonems`, `commonspirit`, `dssundhed`, `gmr`, `hamiltonctyoh`, `montgomerycty`, `pscdtucson`, `regionsjaelland`, `scottishas`, `sfdproduction`, `stlouiscounty`
- **gen2 (Azure AKS)**: `prod-weu`, `prod-us`, `dev-weu`, `staging-eu`
- **gen3 (Scaleway sovereign)**: `dev-par01`, `dev-par02`, `prd-par01`, `prd-par02`, `prd-ric02`, `stg-par01`, `stg-par02`
- **sandbox/misc**: `cortistaging`, `engagestaging`, `sbxtestgmr`, `try`
### `service_name` (OTel service label — primary service selector)
Regex-match the deployment family, e.g. `"ml-service-medical-coding-.*"`, `".*audio-bridge.*|.*transcription.*|.*formatting.*|.*voice-command.*"`.
Common values: `agent-api`, `agent-memory`, `agent-memory-mcp`, `assistant`, `authorization-service`, `code-api`, `copilot-api-v2`, `copilot-audio-bridge`, `envoy-gateway`, `keycloak`, `ml-service-routing`, `ml-service-transcription-online`, `ml-service-transcription-offline`, `ml-service-formatting`, `ml-service-voice-command`, `ml-service-medical-coding-*`, `ml-service-llm-reasoning`, `ml-service-facts`, `ml-service-speech-quality-assessment`, `ml-service-summarisation`, `ml-service-anonymization`, `ml-service-corrector`, `ml-service-evaluation`, `interaction-service`, `call-service`, `review-service`, `search-service`, `task-service`, `transcript-service`, `voice-vault`, `blob-watcher`, `platform-api`, `platform-service`, `orca`, `corti-s1-20260707-epp`, `corti-s1-tiny-20260707-predictor`, `synapse-mcp`, `traefik`, `kafka-kafka-0`, ...
### Other Loki index labels
- `k8s_namespace_name` — the K8s namespace (e.g. `monitoring-system`, `postgres`, `shared`, `kafka`, `base`, plus per-service namespaces)
- `k8s_container_name` — container within pod (e.g. `envoy`, `postgres`, `manager`, `redis`)
- `k8s_deployment_name`, `k8s_statefulset_name`, `k8s_daemonset_name`, `k8s_cronjob_name`, `k8s_job_name` — workload-scoped selectors
- `service_namespace` — OTel service namespace
- `__stream_shard__` — INTERNAL Loki sharding label, never use in queries
> Older dashboards also use unprefixed `namespace`/`app` labels (e.g. code-api, speech-quality-assessment). Prefer the `k8s_*` / `service_name` convention for new queries.
## Mimir — labels & verified values
### `environment`
`dev`, `development`, `prd`, `production`, `sandbox`, `staging` (superset of Loki — the short forms `dev`/`prd` and `sandbox` are Mimir-only)
### `cluster`
Same cluster names as Loki's `k8s_cluster_name`, plus a few Mimir-only: `dev-ams99`, `ghr-dev-ci`, `ghr-prd-platform`. This is the standard Mimir cluster label — use `cluster`, not `k8s_cluster_name`.
### `deployment` (primary workload selector)
`agent-api`, `agent-memory`, `agent-memory-mcp`, `ai-gateway-controller`, `argocd-server`, `argocd-repo-server`, `argocd-applicationset-controller`, `authorization-service`, `assistant`, `blob-watcher`, `call-service`, `code-api`, `copilot-api-v2`, `copilot-audio-bridge`, `envoy-gateway`, `interaction-service`, `keycloak`, `mcp-gateway`, `mcp-server-medical-coding`, `ml-service-facts`, `ml-service-formatting`, `ml-service-llm-reasoning`, `ml-service-medical-coding-*`, `ml-service-routing`, `ml-service-transcription-online/offline`, `ml-service-voice-command`, `orca`, `platform-api`, `platform-service`, `review-service`, `search-service`, `synapse-mcp`, `synapse-ui-grafana`, `task-service`, `traefik`, `transcript-service`, `voice-vault`, `corti-s1-20260707-epp`, `corti-s1-20260707-predictor`, `corti-s1-tiny-20260707-predictor`, `corti-llm-qwen36-27b-epp`, `corti-embedding-qwen3-4b-epp`, ...
### `statefulset`
`keycloak`, `cockroachdb`, `datawarehouse-cockroachdb`, `redis-master`, `redis-node`, `redis-replicas`, `postgresql`, `loki-backend`, `loki-write`, `loki-chunks-cache`, `loki-results-cache`, `synapse-server-mimir-*`, `synapse-server-loki-*`, `opik-redis-master`, `opik-zookeeper`, `argocd-application-controller`, `prompt-ui-openwebui`, ...
### `daemonset`
`cilium`, `calico-node`, `kube-proxy`, `nvidia-dcgm-exporter`, `nvidia-device-plugin-daemonset`, `gpu-feature-discovery`, `csi-azuredisk-node`, `csi-azurefile-node`, `csi-blob-node`, `aks-secrets-store-csi-driver`, `azure-ip-masq-agent`, `datadog`, `node-feature-discovery-worker`, ...
### Other Mimir labels
- `namespace` — K8s namespace
- `container` — container name (e.g. `query-frontend`, `redis`, `postgres`)
- `pod` — pod name (regex-match families: `"redis-node.*"`, `"$model_name-predictor.*"`)
- `job` — scrape job / service (e.g. `argocd-server-metrics`, `synapse-client-alloy-metrics`); roughly maps to Loki `service_name`
- `service`, `node`, `instance` — present on many metrics
- `exported_namespace`, `exported_pod`, `exported_service` — `exported_` prefix from kubelet/DCGM/trivy relabeling (use these on DCGM and trivy metrics, not the unprefixed ones)
- `envoy_cluster_name`, `envoy_response_code_class` (`"2"`/`"3"`/`"4"`/`"5"`), `envoy_http_conn_manager_prefix`
- `gen_ai_request_model`, `gen_ai_original_model`, `gen_ai_response_model`, `gen_ai_token_type` (`input`/`output`) — OTel GenAI semconv
- `model_name` — vLLM model (e.g. `corti-llm-.*`, `corti-s1-.*`)
- `code` — HTTP status code (`=~"5.."`, `!~"2.."`)
- `severity` — `Critical`/`High`/`Medium`/`Low` (trivy)
- `reason` — `CrashLoopBackOff`, `OOMKilled`, `ErrImagePull`, `ImagePullBackOff`
## Loki query patterns (battle-tested from synapse-dashboards)
### Error/warning log hunting — the canonical pattern
Parse JSON, filter on extracted `level` with case-insensitive regex:
```
{k8s_cluster_name=~"$cluster", k8s_namespace_name=~"$namespace", service_name=~"ml-service-medical-coding-.*"} | json | level=~`(?i).*error.*`
```
For warnings: `| level=~\`(?i).*warn.*\``
### `keep` operator — THE cardinality-reduction pattern (use this always)
`keep` drops all extracted labels except the ones named, BEFORE filtering/aggregating. Always use it after `| json` to avoid exploding label cardinality:
```
{k8s_cluster_name=~"$cluster", k8s_namespace_name=~"$namespace", service_name=~"ml-service-medical-coding-.*"} | json | keep level | level=~`(?i).*error.*`
```
Multi-field keep (then numeric filter on an extracted field):
```
... | json | keep route_name, response_code | response_code<200 or response_code>=500
```
Keep for a distinct count:
```
sum(count by (interaction_id) (count_over_time({service_name="copilot-audio-bridge", environment=~"$environment", k8s_cluster_name=~"$cluster", k8s_namespace_name=~"$namespace"} | json | keep interaction_id | interaction_id != "" [$__range])))
```
### Count errors over time (log-derived metric)
```
sum(count_over_time({k8s_cluster_name=~"$cluster", k8s_namespace_name=~"$namespace", service_name=~"ml-service-medical-coding-.*"} | json | keep level | level=~`(?i).*error.*` [$__interval]))
```
### Cheapest count — substring filter, no JSON parse
```
sum(count_over_time({namespace=~"$namespace", app=~".*code-api.*"} |= `error` [$__interval]))
```
### Latency percentiles FROM log fields (`unwrap` + `quantile_over_time`)
Extract a numeric field and compute p90 over time (divide by 1e9 for ns→s):
```
(quantile_over_time(0.9, {k8s_cluster_name=~"$cluster", service_name="traefik"} |= `/documents/` | json | keep RequestMethod, RequestPath, Duration | RequestMethod=`POST` | RequestPath=~`/v2/.+/.+/documents/` | unwrap Duration [$__interval]) by ()) / 1e9
```
### Extract a field with `regexp` + `line_format` (non-JSON logs)
```
sort_desc(topk(15, sum by (clientId, k8s_cluster_name) (count_over_time({service_name="keycloak", k8s_cluster_name=~"$cluster"} | json | environment=~"$environment" | level="WARN" | line_format `{{.message}}` | regexp `clientId="(?P<clientId>[^"]+)"` | keep clientId, k8s_cluster_name [$__range]))))
```
### Normalize a label before group-by (`label_format` + `regexReplaceAll`)
```
... | json | keep route_name, response_code | response_code<200 or response_code>=500 | label_format route_name=`{{ regexReplaceAll "^httproute/shared/ai-gateway-route-(.+?)/rule/.*$" .route_name "${1}" }}`
```
### Raw log volume / rate
```
sum(rate({k8s_cluster_name=~"$cluster"}[5m]))
sum by (k8s_cluster_name) (rate({k8s_cluster_name=~"$cluster"}[$__interval]))
```
### Week-over-week comparison (`offset 7d`)
```
sum(rate({k8s_cluster_name=~"$cluster"}[$__interval]))
sum(rate({k8s_cluster_name=~"$cluster"}[$__interval] offset 7d))
```
### Empty-result fallback
```
count(sum by (k8s_cluster_name) (count_over_time({...} |~ "(?i)(barman.*error|wal not found)" [1h])) > 0) or on() vector(0)
```
### Loki optimization rules
1. **Narrow with label matchers FIRST**, then line filters — `{k8s_cluster_name=..., k8s_namespace_name=..., service_name=...} |= "error"` is cheaper than `{k8s_cluster_name=...} |= "error"`.
2. **`keep` after `| json`** — always. Parsed JSON labels explode cardinality; `keep` limits to what you aggregate on.
3. **`|= "substr"` is cheaper than `|~ "regex"` is cheaper than `| json`** — use the cheapest filter that works. Substring `|=` for a known token; regex `|~` for patterns; `| json` only when you need extracted fields.
4. **`count_over_time` / `rate` for charts; raw query for browsing** — if you only need a count or rate, don't pull raw lines.
5. **`query_loki_stats` before `query_loki_logs`** — if stats say 0 entries, the stream is empty; don't scan it.
6. **Group by `k8s_cluster_name`** to control cardinality across many clusters.
### Log levels in production — many services run at WARN+ only
In `environment="production"`, a lot of services are configured to log at **WARN level and above** — INFO and DEBUG lines are never ingested into Loki. So an INFO-level query against a prod service often returns empty **even when the service is healthy and logging** — the logs simply aren't there.
This is **service-dependent, not universal** (verified 2026-08-21):
| service | INFO present in prod? | notes |
|---|---|---|
| `copilot-api-v2` | yes | logs at INFO (request logs, facts pipeline) |
| `platform-api` | yes | logs at INFO (HTTP access logs via logger_middleware) |
| `authorization-service` | no | WARN+ only (e.g. `level="WARN"` "Invalid JWT") |
| `ml-service-llm-reasoning` | no | no logs ingested under this `service_name` at all (0 entries) |
| `envoy-gateway` | no | 154k lines/hr but NOT JSON with a `level` field — envoy access logs are their own format; `| json | level=~` won't match |
**Implications for your query strategy:**
- **Don't assume empty = broken.** A production `| json | level=~\`(?i).*info.*\`` that returns nothing usually means the service logs at WARN+, not that it's down. Verify with `query_loki_stats` — if `entries > 0` for the stream but your level filter matches nothing, the service is logging (just not at INFO).
- **Start at WARN/ERROR for prod error hunting** — `| json | level=~\`(?i).*(warn|error|fatal).*\`` is the reliable production filter across services. INFO-only queries are only safe for the handful of services known to log at INFO (copilot-api-v2, platform-api, ...).
- **Not all services emit JSON with a `level` field.** Envoy and some infra components use their own log format. If `| json | level=~` matches nothing across ALL levels but the stream has data, fall back to a raw-line filter: `|= "error"` or `|~ "(?i)error|warn"`.
- **Some `service_name` values have zero logs in Loki** (e.g. certain `ml-service-*`). If `query_loki_stats` returns `entries: 0, streams: 0`, the service isn't ingesting to Loki — don't keep retrying with different filters; use Mimir metrics instead.
- **Non-prod (`development`, `staging`) services typically DO log at INFO/DEBUG** — use those environments if you need verbose trace-level logs to reproduce an issue.
## Mimir query patterns (battle-tested from synapse-dashboards)
### Histogram percentiles (117 usages — the most common pattern)
`le` MUST be in the `sum by`:
```
histogram_quantile(0.95, sum by (le) (rate(app_metrics_request_duration_seconds_bucket{cluster=~"$cluster", namespace=~"$namespace", method="POST"}[$__rate_interval])))
```
Grouped by extra dimensions (keep `le`):
```
histogram_quantile(0.95, sum by (cluster, namespace, job, le) (rate(medical_coding_request_duration_seconds_bucket{cluster=~"$cluster", environment=~"$environment", namespace=~"$namespace", job=~".*medical-coding.*"}[$__rate_interval])))
```
Use `synapse_query_prometheus_histogram` for the common case (it builds the `histogram_quantile` for you).
### Hide empty series (`> 0 or ... * 0`)
Keeps zero-traffic series in the legend at value 0 instead of dropping them (which would shift colors):
```
histogram_quantile(0.5, sum by (le, model_name) (rate(vllm_e2e_request_latency_seconds_bucket{...}[$__rate_interval]))) > 0
or sum by (model_name) (rate(vllm_e2e_request_latency_seconds_count{...}[$__rate_interval])) * 0
```
### Filter to top-N busiest (`and on(...) (topk ...)`)
Show latency ONLY for the busiest upstreams:
```
histogram_quantile(0.95, sum(rate(envoy_cluster_upstream_rq_time_bucket{...}[1h])) by (le, job, envoy_cluster_name))
and on (envoy_cluster_name) (topk(40, sum(rate(envoy_cluster_upstream_rq_total{...}[1h])) by (envoy_cluster_name)))
```
### Success/error rate ratios
Grouping labels MUST match on both sides of the division:
```
sum(rate(traefik_service_requests_total{cluster=~"$cluster", environment=~"$environment", code=~"5.."}[$__rate_interval]))
/ sum(rate(traefik_service_requests_total{cluster=~"$cluster", environment=~"$environment"}[$__rate_interval]))
* 100
```
### `increase` for counts over a window
```
sum by (cluster, gen_ai_response_model, gen_ai_original_model) (increase(gen_ai_client_token_usage_sum{cluster=~"$cluster", environment=~"$environment", gen_ai_token_type="output"}[$__rate_interval])) / 2
```
> The `/2` on `gen_ai_client_token_usage` is intentional — compensates for a known dual-reporting artifact. Don't "fix" it.
### Vector-matching joins (`* on(...) group_left(...)`)
Attach `node` to per-pod metrics:
```
(kube_pod_container_status_last_terminated_reason{reason="OOMKilled",environment=~"$environment",cluster=~"$cluster",namespace=~"$namespace"} == 1)
* on(namespace,pod,cluster,environment) group_left(node)
max by (namespace,pod,cluster,environment,node) (kube_pod_info{environment=~"$environment",cluster=~"$cluster",namespace=~"$namespace"})
```
### Crash/OOM classification (`idelta` + join)
```
(max by (cluster, pod, container, namespace, environment) (idelta(kube_pod_container_status_restarts_total{...}[$__rate_interval]))
* on(cluster, pod, container, namespace, environment) group_left()
max by (cluster, pod, container, namespace, environment) (kube_pod_container_status_last_terminated_reason{reason="OOMKilled",...})) > 0
```
### "Time since last X"
```
time() - max(process_start_time_seconds{job="argocd-server-metrics",namespace=~"$namespace",cluster=~"$cluster"})
```
### Resource utilization %
```
100 * (sum by (cluster, node) (container_memory_working_set_bytes{cluster=~"$cluster", environment=~"$environment", container!=""})
/ max by (cluster, node) (kube_node_status_capacity{cluster=~"$cluster", environment=~"$environment", resource="memory"}))
```
## Corti-specific metrics to know
- **vLLM**: `vllm_num_requests_running`, `vllm_num_requests_waiting`, `vllm_kv_cache_usage_perc`, `vllm_time_to_first_token_seconds_bucket`, `vllm_e2e_request_latency_seconds_bucket`, `vllm_prefix_cache_hits_total` / `vllm_prefix_cache_queries_total`, `vllm_spec_decode_num_accepted_tokens_total` / `vllm_spec_decode_num_draft_tokens_total`
- **EPP (inference scheduler)**: `inference_extension_scheduler_attempts_total` (label `status`, `target_model_name`, `pod_name`), `inference_extension_prefix_indexer_hit_ratio_sum`/`_count`, `inference_extension_scheduler_e2e_duration_seconds_bucket`, `inference_pool_per_pod_queue_size`, `inference_pool_ready_pods` (⚠ unverified metric — reads 0 on some EPP builds; see epp-inference-scheduler dashboard description), `inference_objective_request_error_total` (`error_code` incl. `ServiceUnavailable`), `inference_objective_request_duration_seconds_bucket`
- **Envoy**: `envoy_cluster_upstream_rq_time_bucket` (⚠ full E2E incl. generation — NOT a streaming-latency signal), `envoy_cluster_upstream_rq_total`, `envoy_cluster_upstream_rq_xx` (`envoy_response_code_class`)
- **OTel GenAI**: `gen_ai_server_time_to_first_token_seconds_bucket` (gateway TTFT), `gen_ai_client_token_usage_sum`/`_count` (`gen_ai_token_type`=`input`/`output`; three model labels track request→original→response for fallback analysis)
- **Traefik**: `traefik_service_requests_total` (`code`, `exported_service`), `traefik_service_request_duration_seconds_*`, `traefik_tls_certs_not_after`
- **K8s**: `kube_pod_container_status_restarts_total`, `kube_pod_container_status_last_terminated_reason`/`_exitcode`, `kube_deployment_spec_replicas`/`_status_replicas_updated`/`_status_replicas_unavailable`, `container_cpu_usage_seconds_total`, `container_memory_working_set_bytes`, `kube_node_status_allocatable`/`_capacity` (`resource`, `unit`), `kubelet_volume_stats_*`
- **Mimir self-monitor**: `cortex_*` (ingester active series, distributor samples, discarded samples, query-frontend latency by `route`)
- **Loki self-monitor**: `loki_*` (compactor, etc.)
- **GPU (DCGM)**: `DCGM_FI_DEV_GPU_UTIL`, `DCGM_FI_DEV_FB_USED` (labels are `exported_pod`/`exported_namespace`, not `pod`/`namespace`)
- **Kafka (kminion)**: `kminion_kafka_consumer_group_topic_lag` (`group_id`, `topic_name`), `kminion_kafka_topic_high_water_mark_sum`
- **Security (trivy)**: `trivy_image_vulnerabilities` / `trivy_vulnerability_id` (`severity`, `image_repository`, `image_tag`; use `last_over_time(...[65m])` to dedupe hourly scans)
## Monitoring a deployment you just shipped
When a developer asks "is my deploy healthy?" or "did the rollout succeed?", run this verification workflow. It is ordered cheapest-first so you bail early if something is obviously broken. Substitute the real `deployment`/`service_name` and `cluster` — discover them with `list_prometheus_label_values` if unsure.
> **Multi-tenant gotcha:** the same deployment is often rolled out across MANY tenant namespaces (e.g. `copilot-api-v2` exists in `aleris`, `base`, `bighand`, `medatixx`, ... on prod-weu). Always scope by `cluster` and consider `namespace` — a query without a namespace filter returns one series per tenant. Aggregate with `sum by (cluster, deployment)` for a single health verdict, or list per-namespace to find a single sick tenant.
### 1. Rollout complete? (Mimir, instant)
`spec_replicas - updated > 0` means pods still rolling out; `> 0` count = stuck rollouts:
```
kube_deployment_spec_replicas{deployment="copilot-api-v2", cluster="prod-weu"} - kube_deployment_status_replicas_updated{deployment="copilot-api-v2", cluster="prod-weu"}
```
Ready vs desired (should be equal when healthy):
```
kube_deployment_status_replicas_ready{deployment="copilot-api-v2", cluster="prod-weu"}
```
How many deployments are stuck across the cluster (single number):
```
count((kube_deployment_spec_replicas{cluster="prod-weu"} - kube_deployment_status_replicas_updated{cluster="prod-weu"}) > 0) or vector(0)
```
### 2. Pods crashing / restarting? (Mimir, instant)
Per-pod restart count (rising = crash loop):
```
kube_pod_container_status_restarts_total{deployment="copilot-api-v2", cluster="prod-weu", container!="", container!="POD"}
```
OOM-killed right now:
```
(kube_pod_container_status_last_terminated_reason{reason="OOMKilled", deployment="copilot-api-v2", cluster="prod-weu"} == 1)
```
CrashLoopBackOff / ImagePullBackOff counts:
```
count(kube_pod_container_status_waiting_reason{deployment="copilot-api-v2", cluster="prod-weu", reason="CrashLoopBackOff"} == 1) or vector(0)
count(kube_pod_container_status_waiting_reason{deployment="copilot-api-v2", cluster="prod-weu", reason=~"ErrImagePull|ImagePullBackOff"} == 1) or vector(0)
```
### 3. Error logs from the service? (Loki, range)
The canonical JSON-level error filter. `service_name` is the Loki selector that maps to the Mimir `deployment`:
```
{service_name="copilot-api-v2"} | json | level=~`(?i).*error.*`
```
Last 5m, newest first (the default), with `keep` if you only need the count:
```
sum(count_over_time({service_name="copilot-api-v2", k8s_cluster_name="prod-weu"} | json | keep level | level=~`(?i).*error.*` [5m]))
```
If the service logs in logfmt not JSON, use the raw-line regex instead:
```
{service_name="copilot-api-v2", k8s_cluster_name="prod-weu"} |~ "level=(warning|error)"
```
Read the actual error `message`/`error` fields from the parsed JSON in the result to diagnose — don't just count.
### 4. Error rate OK? (Mimir, range)
5xx ratio via traefik (the ingress sees every request). `exported_service` is the traefik label for the backend:
```
sum(rate(traefik_service_requests_total{cluster="prod-weu", exported_service=~".*copilot-api-v2.*", code=~"5.."}[$__rate_interval]))
/ sum(rate(traefik_service_requests_total{cluster="prod-weu", exported_service=~".*copilot-api-v2.*"}[$__rate_interval]))
* 100
```
Compare against the pre-deploy baseline (shift the time range back to before the rollout). A jump starting at deploy time = regression.
### 5. Latency OK? (Mimir, range)
App-native histogram (preferred if the app emits one) — use `query_prometheus_histogram` or:
```
histogram_quantile(0.95, sum by (le) (rate(<app>_request_duration_seconds_bucket{cluster="prod-weu", namespace="base"}[$__rate_interval])))
```
Or traefik upstream time:
```
histogram_quantile(0.95, sum by (le) (rate(traefik_service_request_duration_seconds_bucket{cluster="prod-weu", exported_service=~".*copilot-api-v2.*"}[$__rate_interval])))
```
### 6. Inference services only — EPP / vLLM checks
If the deployment is an `*-epp` or `*-predictor` (vLLM serving), also check:
- **EPP 503s** (picker couldn't find a pod — the client-visible failure mode):
```
sum by (cluster, error_code) (rate(inference_objective_request_error_total{cluster="prd-par01", pod=~"corti-s1-.*-epp.*"}[$__rate_interval]))
```
- **Routing skew** (one pod getting all traffic — winner-take-all):
```
sort_desc(sum by (cluster, pod_name) (increase(inference_extension_scheduler_attempts_total{cluster="prd-par01", status="success"}[15m])))
```
- **vLLM queue backing up**:
```
sum by (cluster, model_name) (vllm_num_requests_waiting{cluster="prd-par01", model_name=~"corti-s1-.*"})
```
See the `epp-inference-scheduler` dashboard (uid `epp-inference-scheduler`) for the full panel set.
### Putting it together
Report a verdict: rollout complete (updated==spec, ready==spec), no new restarts/OOM/CrashLoopBackOff, error rate within baseline, p95 within baseline. If any check fails, quote the offending log line / metric value and the timestamp so the developer can act. Always state the cluster and namespace — a "healthy" verdict is meaningless without them.
## Querying Kubernetes cluster events
Kubernetes cluster events (pod kills, failed scheduling, node pressure, image pull errors, ...) are collected on every cluster and shipped as OTLP logs by the `alloy-singleton` collector (enabled via `k8s-monitoring.clusterEvents`).
The events appear in Loki under the `integrations/kubernetes/eventhandler` service name:
```
{service_name="integrations/kubernetes/eventhandler", k8s_cluster_name="prd-par01"}
```
Useful patterns:
```logql
# Recent events in a cluster (use keep — always)
{service_name="integrations/kubernetes/eventhandler", k8s_cluster_name="prd-par01"}
| keep k8s_pod_name, k8s_namespace_name, k8s_event_reason, k8s_event_message
# Failed scheduling / pod kills in one namespace, last hour
{service_name="integrations/kubernetes/eventhandler", k8s_cluster_name="prd-par01"}
| k8s_namespace_name="copilot-api-v2"
| keep k8s_event_reason, k8s_event_message [1h]
```
When to reach for events: when a deployment's rollout is stuck, pods won't schedule, or you need the *why* behind CrashLoopBackOff — the event stream often has the reason (quota, affinity, node taint) before the app logs do. Always scope by `k8s_cluster_name`; events are low cardinality so a cluster-wide query is fine, but a namespace filter keeps the result readable.
## Common pitfalls
- **Wrong label namespace** — `k8s_cluster_name` in Mimir or `cluster` in Loki → empty results. See the gotcha table above.
- **`environment="sandbox"` in Loki** — returns empty; `sandbox` is Mimir-only. Use `k8s_cluster_name` to find sandbox/on-prem logs.
- **Production INFO/DEBUG queries return empty by design** — many prod services log at WARN+ only (e.g. `authorization-service`). Empty INFO results ≠ broken. Verify with `query_loki_stats`; if the stream has entries but your level filter matches nothing, the service is logging at a higher level. Start prod error-hunting at `level=~\`(?i).*(warn|error).*\``. See the "Log levels in production" section above.
- **Forgetting `le` in `histogram_quantile`** — `sum by (le, ...)` not `sum by (...)`. Without `le` the quantile is meaningless.
- **`envoy_cluster_upstream_rq_time` as streaming latency** — it includes full token generation. For TTFT use `gen_ai_server_time_to_first_token_seconds` (gateway) or `vllm_time_to_first_token_seconds` (engine).
- **Scanning an empty Loki stream** — always `query_loki_stats` first; `entries: 0` means don't bother with `query_loki_logs`.
- **`inference_pool_ready_pods`** — reads 0 on some EPP builds even when serving traffic; treat as unreliable until verified (see epp-inference-scheduler dashboard).
- **ArgoCD two-cluster confusion** — `cluster=~"$k8s_cluster"` is where ArgoCD runs; `dest_server=~"$cluster"` is what it deploys to. Don't conflate.
- **`gen_ai_client_token_usage` `/2`** — intentional dual-reporting fix, not a bug.
- **DCGM/trivy labels** — use `exported_pod`/`exported_namespace`/`exported_service`, not the unprefixed forms.
- **`kube_deployment_status_replicas` does not exist** — use `_ready` / `_updated` / `_unavailable` (and `kube_deployment_spec_replicas` for desired). Verify metric names with `list_prometheus_metric_names` before querying; a bare `count(kube_deployment_status_replicas)` returns empty and looks like "no data" rather than "wrong name".
- **Same deployment, many namespaces** — multi-tenant services (copilot-api-v2, authorization-service, ...) run one Deployment per tenant namespace. A Mimir query without a `namespace` filter returns one series per tenant; aggregate with `sum by (cluster, deployment)` for one verdict, or list per-namespace to find a single sick tenant.