Querying Loki (LogQL)

How to query Corti logs in Loki via the Synapse MCP or Grafana. This page covers the label conventions, the patterns that actually work in this stack, and the production log-level gotcha that catches people out.

The Mimir/PromQL companion is Querying Mimir (PromQL). For an agent-ready skill block bundling both, see Synapse MCP agent skill.

The #1 gotcha: Loki and Mimir use different label names

The same concept has a different label name in Loki vs Mimir. They are not interchangeable — using the wrong one returns empty results.

Rule of thumb: Loki labels are k8s_-prefixed; Mimir labels are not.

Loki indexed labels

Loki index labels — the ones inside {...} stream selectors.

environment

development, production, staging

NOTE: Loki only indexes 3 environment values. Mimir has 6 (adds dev, prd, sandbox). On-prem and sandbox clusters may have metrics in Mimir but no logs in Loki — if a Loki query on environment="sandbox" returns nothing, that's why. For on-prem/sandbox logs, filter by k8s_cluster_name directly.

k8s_cluster_name

Cluster generations (see the platform wiki for full context):

• gen1 (on-prem/customer): bellingham, bostonems, commonspirit, dssundhed, gmr, hamiltonctyoh, montgomerycty, pscdtucson, regionsjaelland, scottishas, sfdproduction, stlouiscounty
• gen2 (Azure AKS): prod-weu, prod-us, dev-weu, staging-eu
• gen3 (Scaleway sovereign): dev-par01, dev-par02, prd-par01, prd-par02, prd-ric02, stg-par01, stg-par02
• sandbox/misc: cortistaging, engagestaging, sbxtestgmr, try

service_name (OTel service label — primary service selector)

Regex-match the deployment family, e.g. "ml-service-medical-coding-.*", ".*audio-bridge.*|.*transcription.*". Common values: agent-api, agent-memory, agent-memory-mcp, assistant, authorization-service, code-api, copilot-api-v2, copilot-audio-bridge, envoy-gateway, keycloak, ml-service-routing, ml-service-transcription-online, ml-service-formatting, ml-service-voice-command, ml-service-medical-coding-*, ml-service-llm-reasoning, ml-service-facts, interaction-service, call-service, review-service, task-service, transcript-service, voice-vault, blob-watcher, platform-api, platform-service, orca, corti-s1-20260707-epp, synapse-mcp, traefik, kafka-kafka-0, ...

Other Loki index labels

• k8s_namespace_name — the K8s namespace (e.g. monitoring-system, postgres, shared, kafka, base, plus per-service/per-tenant namespaces)
• k8s_container_name — container within pod (e.g. envoy, postgres, manager, redis)
• k8s_deployment_name, k8s_statefulset_name, k8s_daemonset_name, k8s_cronjob_name, k8s_job_name — workload-scoped selectors
• service_namespace — OTel service namespace
• __stream_shard__ — INTERNAL Loki sharding label, never use in queries

Older dashboards also use unprefixed namespace/app labels (e.g. code-api, speech-quality-assessment). Prefer the k8s_* / service_name convention for new queries.

Log levels in production — many services run at WARN+ only

In environment="production", a lot of services are configured to log at WARN level and above — INFO and DEBUG lines are never ingested into Loki. So an INFO-level query against a prod service often returns empty even when the service is healthy and logging — the logs simply aren't there.

This is service-dependent, not universal (verified 2026-08-21):

Implications:

• Don't assume empty = broken. A production INFO query returning nothing usually means the service logs at WARN+, not that it's down. Verify with a stats query — if the stream has entries but your level filter matches nothing, the service is logging at a higher level.
• Start at WARN/ERROR for prod error hunting — | json | level=~\(?i).(warn|error|fatal).`` is the reliable production filter across services.
• Not all services emit JSON with a level field. Envoy and some infra components use their own format. If | json | level=~ matches nothing across ALL levels but the stream has data, fall back to a raw-line filter: |= "error" or |~ "(?i)error|warn".
• Some service_name values have zero logs in Loki. If stats return 0 entries/streams, the service isn't ingesting to Loki — don't keep retrying with different filters; use Mimir metrics instead.
• Non-prod (development, staging) typically DO log at INFO/DEBUG — use those environments if you need verbose logs to reproduce an issue.

Query patterns (battle-tested from synapse-dashboards)

Error/warning log hunting — the canonical pattern

Parse JSON, filter on extracted level with case-insensitive regex:

{k8s_cluster_name=~"$cluster", k8s_namespace_name=~"$namespace", service_name=~"ml-service-medical-coding-.*"} | json | level=~`(?i).*error.*`

For warnings: | level=~\(?i).warn.``

keep operator — THE cardinality-reduction pattern (use this always)

keep drops all extracted labels except the ones named, BEFORE filtering/aggregating. Always use it after | json to avoid exploding label cardinality:

{k8s_cluster_name=~"$cluster", k8s_namespace_name=~"$namespace", service_name=~"ml-service-medical-coding-.*"} | json | keep level | level=~`(?i).*error.*`

Multi-field keep (then numeric filter on an extracted field):

... | json | keep route_name, response_code | response_code<200 or response_code>=500

Keep for a distinct count:

sum(count by (interaction_id) (count_over_time({service_name="copilot-audio-bridge", environment=~"$environment", k8s_cluster_name=~"$cluster", k8s_namespace_name=~"$namespace"} | json | keep interaction_id | interaction_id != "" [$__range])))

Count errors over time (log-derived metric)

sum(count_over_time({k8s_cluster_name=~"$cluster", k8s_namespace_name=~"$namespace", service_name=~"ml-service-medical-coding-.*"} | json | keep level | level=~`(?i).*error.*` [$__interval]))

Cheapest count — substring filter, no JSON parse

sum(count_over_time({namespace=~"$namespace", app=~".*code-api.*"} |= `error` [$__interval]))

Latency percentiles FROM log fields (unwrap + quantile_over_time)

Extract a numeric field and compute p90 over time (divide by 1e9 for ns→s):

(quantile_over_time(0.9, {k8s_cluster_name=~"$cluster", service_name="traefik"} |= `/documents/` | json | keep RequestMethod, RequestPath, Duration | RequestMethod=`POST` | RequestPath=~`/v2/.+/.+/documents/` | unwrap Duration [$__interval]) by ()) / 1e9

Extract a field with regexp + line_format (non-JSON logs)

sort_desc(topk(15, sum by (clientId, k8s_cluster_name) (count_over_time({service_name="keycloak", k8s_cluster_name=~"$cluster"} | json | environment=~"$environment" | level="WARN" | line_format `{{.message}}` | regexp `clientId="(?P<clientId>[^"]+)"` | keep clientId, k8s_cluster_name [$__range]))))

Normalize a label before group-by (label_format + regexReplaceAll)

... | json | keep route_name, response_code | response_code<200 or response_code>=500 | label_format route_name=`{{ regexReplaceAll "^httproute/shared/ai-gateway-route-(.+?)/rule/.*$" .route_name "${1}" }}`

Raw log volume / rate

sum(rate({k8s_cluster_name=~"$cluster"}[5m]))
sum by (k8s_cluster_name) (rate({k8s_cluster_name=~"$cluster"}[$__interval]))

Week-over-week comparison (offset 7d)

sum(rate({k8s_cluster_name=~"$cluster"}[$__interval]))
sum(rate({k8s_cluster_name=~"$cluster"}[$__interval] offset 7d))

Empty-result fallback

count(sum by (k8s_cluster_name) (count_over_time({...} |~ "(?i)(barman.*error|wal not found)" [1h])) > 0) or on() vector(0)

Loki optimization rules

1. Narrow with label matchers FIRST, then line filters — {k8s_cluster_name=..., k8s_namespace_name=..., service_name=...} |= "error" is cheaper than {k8s_cluster_name=...} |= "error".
2. keep after | json — always. Parsed JSON labels explode cardinality; keep limits to what you aggregate on.
3. |= "substr" is cheaper than |~ "regex" is cheaper than | json — use the cheapest filter that works. Substring |= for a known token; regex |~ for patterns; | json only when you need extracted fields.
4. count_over_time / rate for charts; raw query for browsing — if you only need a count or rate, don't pull raw lines.
5. Stats before logs — run a cheap index-level stats query first; if it reports 0 entries, the stream is empty — don't run an expensive log query.
6. Group by k8s_cluster_name to control cardinality across many clusters.

Common pitfalls

• Wrong label namespace — k8s_cluster_name in Mimir or cluster in Loki → empty results. See the gotcha table above.
• environment="sandbox" in Loki — returns empty; sandbox is Mimir-only. Use k8s_cluster_name to find sandbox/on-prem logs.
• Production INFO/DEBUG queries return empty by design — many prod services log at WARN+ only (e.g. authorization-service). Empty INFO results ≠ broken. Start prod error-hunting at level=~\(?i).(warn|error).``. See the "Log levels in production" section.
• Scanning an empty Loki stream — always run a stats query first; 0 entries means don't bother with the log query.
• Not all services log JSON with a level field — envoy, some infra components use their own format; fall back to raw-line |= filters.