Grafana PromQL guide

Aimed at devs who write dashboards here but aren't strong on math/sets/rates. No prior stats needed. If you've ever asked "why is Successful Requests showing 85.4 instead of 85" — start here.

1. A metric is a set of numbered timestamps, not a number

A Prometheus metric isn't one value — it's many time series, each a list of (timestamp, value) samples. A time series is picked out by its metric name + label set:

vllm_request_success_total{cluster="c1", namespace="ns1", model_name="ultra"}
vllm_request_success_total{cluster="c1", namespace="ns1", model_name="fast"}

Two different series (different model_name), same metric name. When a query says sum(...), it's collapsing a set of series into one number — that's the "sets" part. Nothing exotic: it's just "add up all series matching these labels."

metric: vllm_request_success_total
├─ {model_name=ultra}  samples over time  ─┐
└─ {model_name=fast}   samples over time  ─┴─→ sum(...) → one number

2. Two metric types you'll actually meet

• Counter — only ever goes up (or resets to 0 on pod restart). Examples: _total suffixed metrics, vllm_request_success_total, vllm_e2e_request_latency_seconds_sum. You never read a counter's raw value on a dashboard — you read how fast it's rising.
• Gauge — goes up and down freely. Examples: vllm_num_requests_waiting, memory usage, queue depth. You often do read a gauge's raw value (sum(vllm_num_requests_waiting{...})), no rate needed.

Mixing these up is the #1 source of nonsense graphs: rate() on a gauge, or plain value on a counter, both produce garbage.

3. Instant vector vs range vector — the [5m] part

A query like vllm_num_requests_waiting{...} returns one value per series, right now — an instant vector.

Add a time window in square brackets — vllm_request_success_total{...}[5m] — and it becomes a range vector: not one value, but every sample recorded in the last 5 minutes, per series. A range vector can't be graphed directly; it's raw material for functions like rate(), increase(), delta(), deriv(), idelta(), irate() — every one of them takes a range vector in and hands an instant vector back out.

vllm_request_success_total{...}
├─ no brackets         → instant vector (1 value, right now)
└─ [5m] range selector → range vector (all samples, last 5m)
                          └─→ rate() / increase() / delta() /
                              deriv() / idelta() / irate()
                              └─→ instant vector (1 value, derived)

4. $__range vs $__rate_interval — this chart's two window variables

Grafana auto-fills these based on the dashboard's time picker (top-right "Last 1 hour" etc.):

• $__range — the whole visible time window. Use for a total over the dashboard's window: "how many successful requests in the last hour" (see vllm-overview.json, panel "Successful Requests").
• $__rate_interval — a short window (a few scrape intervals), constant regardless of the dashboard's zoom level. Use for per-instant rates plotted as a line/bar over time: "requests/sec right now, at each point on this graph."

Rule of thumb: single stat panel, one big number for the whole window → $__range. Line graph moving over time → $__rate_interval.

Example — same metric, two window variables, two different jobs:

sum(increase(kube_pod_container_status_restarts_total{...}[$__range]))
→ total restarts across the whole visible window — one number for a stat panel

sum(increase(kube_pod_container_status_restarts_total{...}[$__rate_interval]))
→ restarts per data point — a wiggly line that moves as you scrub the time picker

Use $__range in the first case (a fixed total shouldn't change shape when you zoom); use $__rate_interval in the second (a rate line needs a window short enough to show variation, no matter how far zoomed out you are).

5. The six functions, one at a time

All six take a counter or gauge range vector and produce a rate/delta as an instant vector. The difference is what they compute and which metric type they're for.

rate(counter[window]) — average per-second rise, smoothed

For counters. Takes the increase across the whole window, divides by the window length, extrapolated to cover the edges (see §6). Smooth, because it averages across every sample in the window — good default for graphs.

rate(vllm_request_success_total[$__rate_interval])
→ "requests/sec, averaged over the last few scrapes"

irate(counter[window]) — per-second rise, last two samples only

For counters. Ignores everything in the window except the last two samples — instantaneous, spiky, reacts fast but is noisy. Good for high-resolution "what's happening right now" graphs; bad for alerting thresholds (too jumpy) or long windows (throws away data).

samples:            s1     s2     s3     s4     s5
rate()  uses:        ●──────●──────●──────●──────●    (every sample → smoothed)
irate() uses:                             ●──────●    (last two only → instant)

increase(counter[window]) — total rise over the window

Literally rate(...) * window_seconds — an estimated total count, not a per-second rate. This is why increase() can return 85.4 instead of 85: it's not counting integer events, it's extrapolating a rate and scaling it back up (§6 explains the fractional part precisely). Use for "how many total X happened in this window" stat panels.

sum(increase(vllm_request_success_total{...}[$__range]))
→ "≈ how many successful requests happened in the visible window"

delta(gauge[window]) — total change over the window

increase()'s gauge counterpart. Can be negative (gauges go down too), and does not do counter-reset correction (because gauges don't "reset" the way counters do on restart). Use for "how much did this gauge change over the window" — e.g. queue depth net change.

idelta(gauge[window]) — change between the last two samples

irate()'s gauge counterpart: last-two-samples-only, instantaneous.

It also has a neat trick when applied to a counter: for a per-event bar chart (one clean bar per incident, not a smear), idelta() beats increase(). kubernetes/oom-killed.json restart-spike panel:

idelta(kube_pod_container_status_restarts_total{...}[$__rate_interval])

A restart counter jumps by exactly 1 the instant a container restarts, then sits flat until the next one. increase() over that same window would spread that single +1 across the entire lookback window — a low bar that lingers and fades as the window slides past the event, easy to misread as "restarts trickling in over time." idelta() only looks at the last two samples, so it reports the +1 as a single sharp bar exactly when it happened, and 0 everywhere else — because it isn't extrapolating across a window at all, there's nothing to smear.

deriv(gauge[window]) — per-second rate of change, via regression

For gauges. Fits a straight line through all samples in the window (linear regression) and reports its slope — smoother than delta()/window_length because it's least-squares, not just endpoint-to-endpoint. Used less often in this chart; reach for it when a gauge trend is noisy and you want a proper slope rather than a raw before/after delta.

Cheat sheet

6. Why increase() gives fractional numbers (the 85.4 question)

Prometheus doesn't actually know the exact integer count of events in your window — it only has samples at scrape intervals (e.g. every 30s), and the window you asked for ($__range) almost never starts/ends exactly on a scrape timestamp. So the very first and last sample Prometheus can see inside your window are a bit inside the edges, not sitting on them:

counter value:         40          41          43          44          45
scrape samples:        ●───────────●───────────●───────────●───────────●
                       t=-10s      t=5s        t=35s       t=65s       t=95s

requested window ($__range = 90s, say t=0s to t=90s):
                             |<───────────────────────────────────>|
                             t=0s                                  t=90s
                                    ▲                        ▲
                              first sample                last sample
                              INSIDE window               INSIDE window
                              (t=5s, value 41)            (t=65s, value 44)

increase() can only directly measure what happened between the first and last sample it can see inside the window — here, 41 → 44, a rise of 3, over 60 real seconds (t=5s to t=65s). But it owes you a number for the whole 90s window, including the two slivers it has no samples for: t=0s → t=5s and t=65s → t=90s. It extrapolates: assumes the same rate (3 per 60s) held during those slivers too, and adds the estimated partial rise on top of the 3 it actually measured. That's where a .4 (or any non-integer) comes from — it's an estimate of "how many would have happened if the observed rate applied for the entire requested window," not a literal SELECT COUNT(*).

The exact arithmetic on the diagram's numbers:

measured:     3 events over 60s  →  rate = 3 / 60 = 0.05 events/s
extrapolated: 0.05 × 90s window  =  4.5
result shown: increase(...) = 4.5   (though literally 3 events occurred)

So a stat panel reading 4.5 after 3 "real" restarts is Prometheus being honest about an estimate — scale that to 85.4 after 85 "real" requests and it's the same thing at a bigger window.

This is expected, not a bug. Two ways to force a whole number for display, without lying about the estimate:

• round(increase(...)) in the query — rounds the extrapolated estimate.
• decimals: 0 in the panel's fieldConfig.defaults — display-only, doesn't touch the underlying number or any alert/other panel reusing the same expression. Usually the better choice.

7. Label math: sum drops labels, sum by keeps them

Aggregators decide what happens to your labels:

• sum(vllm_request_success_total) — collapses everything into one series with no labels. One line, one number.
• sum by (model_name) (vllm_request_success_total) — collapses only the labels you didn't name. One series per model_name.
• without (pod) (...) — the inverse: drop just the listed labels, keep the rest.

The trap is forgetting that every unique label combination is its own series (§1). 3 models × 6 pods × 2 clusters = 36 series. Graph that raw and you get 36 wiggly lines where readers wanted one. This cardinality is multiplicative, which is also why high-cardinality labels (user_id, request IDs) in a metric are a monitoring-platform problem, not just a readability one.

And the classic ordering bug — with counters, always aggregate after rate, never before:

sum(rate(vllm_request_success_total[$__rate_interval]))   ✔ correct
rate(sum(vllm_request_success_total)[$__rate_interval])   ✘ wrong

rate does counter-reset correction per series (§15). Sum first and you've destroyed the per-series boundaries — pod A's restart mixes with pod B's samples, producing garbage. Same rule for increase: sum(increase(...)).

8. Combining two metrics: labels must line up

errors / total looks obvious but silently returns empty rows ("No data", no error message) whenever the two sides' label sets don't match. Vector arithmetic is a join on the label set: a series on the left is paired only with a series on the right that has exactly the same labels.

So this fails when one side carries a label the other doesn't:

vllm_errors_total{model="ultra", code="500"}      ← has code
vllm_requests_total{model="ultra"}                ← doesn't
── division finds no matching partner, returns nothing

Fix by aggregating both sides to the same label set first:

sum by (model_name) (rate(vllm_errors_total[$__rate_interval]))
  /
sum by (model_name) (rate(vllm_requests_total[$__rate_interval]))

For deliberate joins on a subset of labels there's on(...) (join keys) with group_left / group_right (which side's extra labels survive) — reach for those when you can't aggregate your way to matching sets. Rule of thumb: "No data" after adding arithmetic → mismatched labels until proven otherwise.

9. How Grafana actually runs your query

A panel is not a static picture — Grafana re-runs every query when the time picker, variables, or refresh timer change, on roughly your screen's pixel density:

1. Grafana divides the window width by the panel's Max data points (default ~1500) to decide a step — "give me one value every N seconds."
2. That step feeds Min step, $__interval, and $__rate_interval.
3. Prometheus evaluates the query at each step — the window and the resolution both change when you zoom.

Consequences that surprise people:

• Zooming changes the shape of the line, not just its scale — more pixels means finer steps means sharper spikes appear. A spike that "disappears" when you zoom out wasn't averaged away by your query; the new query simply can't see it at that resolution.
• Cranking up resolution in the query options makes prettier graphs and heavier queries — it literally asks Prometheus for more points.
• Refresh cadence matters: a 1m-refreshed "5m window" stat is 20% stale data by design.

The legend row is zoom-dependent too. The Max / Mean / Last values under a panel are computed over the points Grafana got back — at the current step and $__rate_interval. Zoom from 3 h to 7 d and Grafana re-runs the query coarser: short spikes get averaged away by the wider per-point window or skipped between steps, so the legend's Max shrinks even though nothing changed in the data. "3 h view says decode peaked at 26 s, 7 d view says 15 s" is expected behavior, not a contradiction.

If you need a zoom-invariant worst case, don't read it off the legend — use a max_over_time subquery in a stat panel, which inner-samples at a fixed step regardless of zoom:

max_over_time(
  (sum(rate(vllm_request_decode_time_seconds_sum{...}[1m]))
     / sum(rate(vllm_request_decode_time_seconds_count{...}[1m])))[7d:1m]
)
→ "worst 1-minute average decode time anywhere in the last 7 days"

Read this as "evaluate the per-1m average every 1 m across 7 d, keep the max" — the answer no longer depends on how far you're zoomed out.

10. Instant vs Range, and Format: table

Two Grafana query toggles worth knowing by name — the repo's own dashboards use both halves:

• Type: Instant — evaluate once, at the end of the window; one value per series. Right for stat panels (a total over $__range) and tables.
• Type: Range — evaluate at every step across the window; the wiggly line. Right for timeseries panels.
• Format: Table — return rows/columns instead of time series so a table panel (or a joinByField transformation) can use them.

The Runner Status table in team platform/github-actions-runners.json uses five instant: true, format: "table" queries — one per autoscaler metric — joined on the runner_group label by a transformation. That pattern only works because each query returns one row per label set; run the same query as Range and you'd get one row per sample per label set, an unreadable explosion. Stat panels are Instant for the same reason: you want the value at the end of the window, not 1500 of them.

11. Histograms (the _bucket/_sum/_count trio)

Some metrics (especially latencies) aren't counters or gauges but a histogram: three families of counters exported together from one measurement stream.

• _bucket{le="0.5"}, _bucket{le="1"}, … _bucket{le="+Inf"} — counters of observations ≤ each boundary. le = "less than or equal".
• _sum — counter of total observed value (all latencies added up).
• _count — counter of observations.

You never graph the buckets raw. The two useful shapes are:

# p99 latency: "the line 99% of requests finish under"
histogram_quantile(0.99,
  sum by (le) (rate(vllm_e2e_request_latency_seconds_bucket[$__rate_interval])))

# average latency over the window (not a quantile — just mean):
sum(rate(vllm_e2e_request_latency_seconds_sum[$__rate_interval]))
  /
sum(rate(vllm_e2e_request_latency_seconds_count[$__rate_interval]))

Two beginner traps:

• sum by (le) is not optional — without it, buckets from different pods/models are quantile'd separately (one line per pod) or, worse, mixed incorrectly. The le label must survive the aggregation or the math has no meaning.
• Quantiles are only as precise as the buckets. If all 99% of requests land between le="2" and le="5", a p99 can only say "somewhere between 2 and 5" — Prometheus interpolates inside the bucket. A p99 of "5.0" often just means "hit the largest bucket," i.e. some requests might be slower than anything you measure.

12. Averages hide tails, and tails don't add up

Two panels can be correct and still look mutually inconsistent, because means and quantiles are different cuts of the same distribution.

A mean compresses; a quantile isolates. "Avg Decode Time: 6.6 s" means the typical request took 6.6 s decoding — 1 request in 100 taking 4 minutes moves this number by barely anything. "p99 Request Latency: 3.8 min" means the worst 1% of requests in each window took up to 3.8 min. You cannot reconstruct one from the other, and you cannot add a mean to a p99. If your brain tries max p99 ≈ max prefill + max decode, both of these resist: the p99 is a max over windows of the 99th-percentile request; the prefill panel shows a mean of all requests in the window. Different axes, different populations — they're not arithmetically composable.

Legend Max is the worst window's aggregate, not the worst request. "Max 15 s" under a mean-timeseries panel means "the busiest window's average was 15 s", not "some request took 15 s".

Phase identities need every term. vLLM's request lifecycle satisfies, per request:

e2e latency ≈ queue wait + prefill + decode (+ scheduler overhead)

Measured on corti-llm-glm-52-nvfp4 over 7 d: 0.10 + 0.37 + 6.61 ≈ 7.1 s vs observed e2e mean 7.4 s — checks out. The easy term to forget is queue (waiting for a batch slot): in a saturated window, a 4-minute e2e tail can be 95% queue while prefill and decode means look totally healthy. When a latency mystery doesn't add up, start by listing all the phases on the dashboard and checking which one you're not seeing a term for.

Same-sounding metrics differ by a denominator. decode_time is a whole-sequence duration; time_per_output_token (TPOT) is per token. Neither replaces the other, and they relate as decode ≈ TPOT × output_tokens (verified live: 6.61 s ≈ 14.3 ms × ~460 tokens). Before trusting a panel's units, sanity-check the metric name's implied denominator — prom metric HELP text tells you.

13. Which function do I want? (the chooser)

The §5 table is for lookup; this is for the moment you open a blank query editor:

What are you measuring?
│
├─ counter, e.g. requests_total
│    What shape of answer?
│    ├─ one big number for the window  → increase(counter[$__range]) + stat panel + Instant
│    ├─ smooth rate line over time     → rate(counter[$__rate_interval])
│    ├─ one sharp bar per event        → idelta(counter[$__rate_interval])
│    └─ fast-reacting, noisy line      → irate(counter[$__rate_interval])
│
└─ gauge, e.g. queue depth
     What shape of answer?
     ├─ current value is fine          → no function needed — sum(gauge)
     ├─ net change over window         → delta(gauge[$__range])
     └─ smoothed trend slope           → deriv(gauge[$__rate_interval])

And three wrappers that apply to nearly every branch: aggregate with sum/sum by after the function (§7) · rescue missing series with or vector(0) (§14) · guard divisions with clamp_min(denominator, 1) (§14).

14. Query hygiene: zero baselines and safe division

Two small habits that make dashboards behave when data is missing or zero, rather than showing "No data" or blowing up on a division.

Render a zero baseline instead of "No data". If a series only exists when something bad happens (errors, restarts), a cluster with no errors has no series at all — Prometheus can't return "zero rows of a series that was never scraped." The panel shows "No data," which reads as "unknown" rather than "genuinely zero," and breaks anything (like a red/green threshold) that expects a real 0. Fix: or vector(0) appends a literal zero series whenever the left side returns nothing.

sum(rate(http_requests_total{status=~"5.."}[$__rate_interval])) or vector(0)

— from team speech recognition/ml-routing.json's non-2xx panel: no 5xx responses in the window → the panel now correctly reads 0 instead of "No data".

Guard division with clamp_min(). A denominator that's legitimately 0 (no PVC capacity reported yet, a gauge that hasn't populated) turns a division into NaN or +Inf, which Grafana renders as a blank or nonsensical spike. clamp_min(x, 1) floors the denominator at 1 so the division degrades to a small, sane number instead of exploding:

kubelet_volume_stats_used_bytes{...}
  / clamp_min(kubelet_volume_stats_capacity_bytes{...}, 1)
  * 100

— from team platform/redis-incident.json's PVC usage-percent panel.

15. Common pitfalls

• rate()/increase() on a gauge — nonsense. Gauges can go down; these functions assume monotonic counters and "fix" resets by adding the drop back, which corrupts a real decrease into a fake spike.
• irate() on a low-traffic counter — if scrapes are 30s apart and events are rare, "last two samples" can span minutes of real time, producing a wildly wrong instantaneous rate. Use rate() instead.
• Window shorter than ~4x scrape interval — not enough samples to extrapolate sensibly; results get noisy or NaN. This chart's $__rate_interval is sized by Grafana to avoid this automatically.
• Forgetting sum() around a rate/increase — you'll get one line per label combination instead of one aggregate number, e.g. one line per model_name instead of a single "Successful Requests" total.
• Counter reset from a pod restart — rate()/increase() detect and correct for this automatically (that's part of why they're preferred over doing subtraction by hand); delta() on a gauge does not need this because gauges don't "reset" the same way.
• rate(sum(...)) — aggregating counters before rate destroys per-series reset handling (§7). Always sum(rate(...)).
• Dividing metrics with different label sets — silently yields "No data"; align both sides with the same sum by (...) list first (§8).
• Same stat, different zoom = different number — expected: zooming reruns the query at a new resolution and window (§9), and increase() output shifts with extrapolation edges (§6).
• Range-type query on a table/stat panel — you asked Prometheus for 1500 points and the panel shows one row per sample or an arbitrary "last" value. Switch the query to Instant (§10).
• histogram_quantile without sum by (le) — buckets must be summed across pods/models keeping le, or the quantile is meaningless (§11).
• Adding a mean panel and a p99 panel — "max p99 ≈ max prefill + max decode" never holds: means and quantiles cut the distribution differently, and you may be missing a phase entirely (queue!) (§12).
• Reading legend Max as "worst request" — it's the worst window's aggregate; and it shrinks when you zoom out because Grafana re-runs the query coarser (§9, §12). Want the real worst case? max_over_time subquery (§9).

16. Try it yourself

Five-minute exercises against any Grafana + Prometheus datasource (a dev cluster is fine — these are read-only):

1. See a counter raw, then rate it. Graph http_requests_total-style metric (or any _total) without any function: an ever-rising staircase. Wrap it in rate(...)[$__rate_interval]: the staircase becomes "events per second." Now you know why nobody graphs counters raw.
2. Break it on purpose. Point rate() at an obvious gauge (memory/queue depth). Watch the invented spikes where the gauge dropped — that's the reset "correction" misfiring (§15).
3. Label math. Take any multi-pod metric. Switch between sum(...) and sum by (pod) (...): one line → N lines, same data. Then try sum by (pod) (rate(...)) per pod and sum(rate(...)) aggregate — and try rate(sum(...)) to see why order matters when a pod restarts.
4. Rescue "No data". Write a selector that matches nothing ({status="599"} or a nonsense label). Panel dies; append or vector(0); baseline appears (§14).
5. Watch increase() fractionate. Stat panel: sum(increase(some_counter[$__range])) on a quiet metric → decimals. Fix with decimals: 0 in field config — and notice the value is unchanged, only the display (§6).
6. Build the runners-table pattern. Two queries, both Instant + Format: table, then a joinByField transformation on the shared label — that's all the Runner Status table in github-actions-runners.json is (§10).

17. Glossary — Grafana UI ↔ PromQL terms

See also

• README.md — "Query practices" section: remaining house conventions not covered above (prefer native Prometheus metrics over LogQL, histogram_quantile / rate(_sum)/rate(_count) for histograms).
• dashboards/team platform/vllm-overview.json — the dashboard that prompted this doc (increase() "Successful Requests" panel).
• dashboards/kubernetes/oom-killed.json — idelta() per-event bar example.
• dashboards/team speech recognition/ml-routing.json — or vector(0) zero-baseline example.
• dashboards/team platform/redis-incident.json — clamp_min() safe-division example.
• dashboards/team platform/github-actions-runners.json — Instant + Format: table + joinByField pattern (§10), and aggregate-vs-by label math (§7) side by side: Availability panel uses sum(...), Runner Status table uses avg by (runner_group) (...).