Runbook for Grafana failing to load provisioned dashboards, triggered by the frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts (LogQL → Loki ruler → Mimir Alertmanager → incident.io).

Meaning

Counts Grafana log lines at error level containing failed to load dashboard from in the monitoring-system namespace over a 10m window. Any occurrence sustained for 15m fires the alert.

The failing log line looks like:

logger=provisioning.dashboard type=file name=sidecarProvider level=error msg="failed to load dashboard from "
file="/tmp/dashboards/Team Platform/vllm-envoy-latency.json" error="invalid character '\"' after object key:value pair"

Why `service_name="synapse-ui-grafana"` scope

The Loki ruler ships its own evaluation logs (including the alert's query string) to the same Loki. A bare |= "failed to load dashboard from" filter without the service_name scope would match the ruler's own evaluation logs — the alert would count itself and fire permanently regardless of real errors. Scoping to Grafana's service_name (labelled by the Alloy log pipeline from k8s_deployment_name) removes that self-referential feedback, and keeps the query a cheap substring match (no logfmt parser needed).

Why `detected_level=`error``

Grafana logs more than dashboard-provisioning failures, and the substring failed to load dashboard from also appears in unrelated Grafana log lines at level=info — e.g. the Loki plugin logs queries verbatim (logger=plugin.loki ... level=info ... query="{...} |= failed to load dashboard from"), so anyone inspecting this alert in Grafana Explore would be counted by the alert. Restricting the match to detected_level="error" (assigned by the Alloy log pipeline) keeps only genuine provisioning failures — real failed to load dashboard from lines are always emitted at level=error.

Why the `dashboard` label

The regexp parser extracts the dashboard file path from the log line's file= field, stripping the /tmp/dashboards prefix — the alert name carries the broken dashboard, so diagnosis starts from the file instead of a log dig. Lazy matching (.*?) is required: a greedy .* would swallow the trailing " error="… into the label. The path is <grafana folder>/<file>.json (e.g. Team Platform/vllm-envoy-latency.json); the first path segment is the grafana_folder annotation on the source ConfigMap.

Why `for: 15m`

The k8s-sidecar re-syncs dashboards and Grafana re-reads its provisioning directory roughly every 10–15 min, so a genuinely broken file produces recurring errors — for: 15m rides out a single transient read (e.g. Grafana reading while the sidecar is mid-write) while still firing quickly on a persistent misconfiguration.

Impact

Missing dashboards. A dashboard whose JSON Grafana cannot parse is silently skipped — it never appears in the UI, and no data panels are rendered for it. Everything else keeps working.
Silent by default. Grafana logs the failure at level=error but nothing else surfaces it; this alert is the only signal.
Recurring noise until fixed. The broken file fails on every provisioning cycle (~every 10–15 min), so the alert stays firing until the JSON is corrected. That is intentional — it is a real misconfiguration.

Real-world instance: vllm-envoy-latency.json on dev-par01 (2026-09) — the ConfigMap synapse-dashboards-vllm-envoy-latency (chart synapse-dashboards-0.22.8, folder Team Platform) contained JSON malformed from line 19, so Grafana failed to load it on every sidecar sync (~92 errors/24h) and the dashboard was absent from the UI.

Diagnosis

From the alert labels, note k8s_cluster_name and environment.

1. Find the affected file(s)

{k8s_cluster_name="<cluster>", k8s_namespace_name="monitoring-system", service_name="synapse-ui-grafana"} |= "failed to load dashboard from" | detected_level=`error`

The file= field names the broken JSON under /tmp/dashboards/<grafana folder>/<name>.json.

2. Find the source ConfigMap

The files under /tmp/dashboards are written by the k8s-sidecar from ConfigMaps labelled grafana_dashboard: synapse. The file name maps to the ConfigMap data key, and the folder to the grafana_folder annotation:

export KUBECONFIG=~/.kube/<cluster>.yaml
# by data key (file name without path), or search all dashboard ConfigMaps:
kubectl -n monitoring-system get cm -o json \
  | jq -r '.items[] | select(.metadata.labels.grafana_dashboard == "synapse")
           | "\(.metadata.name)"'

The matching ConfigMap is the synapse-dashboards-* one whose data key equals the file name (e.g. synapse-dashboards-vllm-envoy-latency → vllm-envoy-latency.json).

These ConfigMaps are defined in code: the synapse-dashboards chart in the charts repository stores every dashboard as a JSON file under charts/synapse-dashboards/dashboards/<grafana folder>/<name>.json (folder names are lowercase, e.g. team platform), which the chart template renders into the ConfigMaps. There is nothing to find or edit in the cluster that is not already in that repo.

3. Validate the JSON

kubectl -n monitoring-system get cm <configmap> -o json \
  | jq -r '.data | keys[]' \
  | while read k; do
      kubectl -n monitoring-system get cm <configmap> -o "jsonpath={.data.$k}" | jq -e . >/dev/null \
        && echo "OK: $k" || echo "INVALID: $k"
    done

jq reports the line/column of the first syntax error.

Mitigation

The fix is to correct the dashboard JSON at its source. The synapse-dashboards-* ConfigMaps are rendered by the synapse-dashboards Helm chart (charts repo) and delivered by ArgoCD — manual kubectl edit is reverted by self-heal. Fix the JSON file under charts/synapse-dashboards/dashboards/<grafana folder>/ in the charts repository, cut a new chart tag, let the tag bump flow through to the cluster's config.json, and ArgoCD syncs the corrected ConfigMap.

Common causes of malformed JSON:

Hand-edited or generated JSON with a syntax error (trailing comma, unescaped quote, truncated export) — fix the syntax.
Template/CI escaping bug — a build step that injects values without escaping quotes; fix the generator.

There is no runtime mitigation; the alert clears once the corrected ConfigMap syncs and Grafana loads the file on its next provisioning cycle.

Verify

The alert clears once Grafana loads the corrected file on the next sidecar/provisioning cycle, sustained for the for: 15m window.
The dashboard appears under its grafana_folder (e.g. Team Platform) in the Grafana UI.
Re-run the diagnosis LogQL: zero failed to load dashboard from lines in the last 30m.

Notes

Grafana also reports provisioning errors in the UI (Alerting → Provisioning / dashboard list shows a stale-sync indicator), but only for signed-in admins.

Grafana Dashboard Provisioning Error