CPU Throttling > 20%

Runbook for a container being CPU-throttled, triggered by the CpuThrottlingHigh Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). This mirrors the Datadog [platform/capacity] StatefulSet is being CPU throttled more than 20% alert, which fired on redis-node (prod-us, prod-weu) on 2026-08-05.

Meaning

Linux CFS (Completely Fair Scheduler) enforces a container's CPU limit by throttling: each scheduling period (default 100ms) the container is allowed limit × period of CPU time. If it exhausts that quota before the period ends, it is throttled — it cannot run until the next period. container_cpu_cfs_periods_total counts the number of CFS periods elapsed; container_cpu_cfs_throttled_periods_total counts the periods in which the container was throttled. The ratio throttled_periods / periods is the fraction of periods the container was capped — the standard CPU-throttling signal. > 0.2 means more than 20% of periods were throttled.

Both metrics are scraped from cAdvisor (kubelet :10250/metrics/cadvisor) by synapse-client's alloy-metrics and remote-written to synapse-server Mimir. The alert groups by k8s_cluster_name, environment, namespace, pod, container so each throttled container is its own alert instance.

# via the Mimir datasource proxy (or Grafana Explore):
count(container_cpu_cfs_throttled_periods_total)   # > 0 once the metric flows

cAdvisor metrics carry pod but not a kube_stateful_set label, so the alert is per-pod. The Datadog alert grouped by kube_stateful_set (e.g. redis-node); to map a firing pod to its StatefulSet, the pod name prefix is the STS name (redis-node-0 → StatefulSet redis-node).

Impact

• The container is being capped at its CPU limit and is running slower than its workload demands. Latency rises; throughput drops.
• For a serving container (API, exporter, sidecar), request latency degrades. The metrics container (redis-exporter) throttling is usually low-impact (scrapes are bursty but small), but signals the limit is mis-sized.
• For a worker container (processing queue items, ASR inference), throughput drops and backlogs grow. Sustained >20% throttling means the limit is materially too low.
• Sustained throttling can cascade: a slow worker causes queue backlog, which causes memory growth, which can hit the memory limit → OOMKill (see 02-pod-crashloop.md).

Diagnosis

The alert labels carry k8s_cluster_name, namespace, pod, container, and environment. From the alert in Grafana, note all of them.

1. Confirm the throttling and see the CPU limit

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prod-us-admin.yaml
NS=<namespace>
POD=<pod>
kubectl -n $NS get pod $POD -o jsonpath='{range .spec.containers[?(@.name=="<container>")]}{.name}: limit={.resources.limits.cpu} request={.resources.requests.cpu}{"\n"}{end}'

Confirm the live throttling ratio via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

QUERY='sum by (k8s_cluster_name,namespace,pod,container) (rate(container_cpu_cfs_throttled_periods_total{k8s_cluster_name="<cluster>",namespace="<namespace>",pod="<pod>"}[5m])) / sum by (k8s_cluster_name,namespace,pod,container) (rate(container_cpu_cfs_periods_total{k8s_cluster_name="<cluster>",namespace="<namespace>",pod="<pod>"}[5m]))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

A value > 0.2 confirms the alert. If the query returns no series, the metric is not yet flowing (see the IMPORTANT note above) — check whether chart #2923 has synced to this cluster.

2. Check whether the workload is bursty or mis-sized

Look at the CPU usage vs the limit over the last hour:

QUERY='sum by (pod,container) (rate(container_cpu_usage_seconds_total{k8s_cluster_name="<cluster>",namespace="<namespace>",pod="<pod>",container="<container>"}[5m]))'

Compare to the limit (cores). If usage is consistently near or above the limit, the limit is too low. If usage is bursty (spikes above the limit, low average), a higher limit or removing the limit (if the node has headroom) is the fix.

3. Identify the owning workload (StatefulSet / Deployment)

kubectl -n $NS get pod $POD -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}'

For a pod named redis-node-0, the owner is StatefulSet/redis-node. The CPU limit lives in the StatefulSet's container.resources.limits.cpu (or the chart values that rendered it).

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Remove the CPU limit (preferred fix)

We generally prefer not to set CPU limits — CPU limits cause CFS throttling, which degrades latency and throughput even when the node has plenty of spare CPU. A container with a CPU request and no limit is scheduled (guaranteed its request) and can burst into node headroom when other containers are idle, which is the normal case on our clusters. The CPU request is what drives scheduling decisions; the limit only adds throttling.

This is the documented Kubernetes recommendation and a well-established operational practice:

"If you do specify CPU limits, you must ... ensure that the application is not being throttled by the CPU limit. ... CPU throttling can cause application performance problems." — Kubernetes docs, Manage CPU Resources

"CPU limits ... cause more harm than good. ... removing CPU limits improves application latency and reduces CPU throttling without impacting other workloads." — Robusta: Stop using CPU limits

The fix is a deployments PR dropping limits.cpu, keeping requests.cpu:

# example: redis-node metrics sidecar
metrics:
  resources:
    requests:
      cpu: 100m
    # limits.cpu omitted — container is scheduled at 100m and can burst
    # into node headroom when other containers are idle, without throttling.

For an ArgoCD-managed workload, the fix is GitOps: edit the values, merge, ArgoCD syncs and the pod restarts. Do not kubectl set resources directly (ArgoCD self-heal reverts it in ~15s).

Before removing the limit, confirm the node has spare CPU: kubectl top nodes and the node's kube_node_status_allocatable vs usage. On a saturated node, removing limits can let one container starve others — but this is rare on our clusters, which are sized for headroom.

Step 3 — Raise the CPU limit (if a limit is required)

In the rare case a limit must be kept (e.g. a known-greedy workload on a shared node, or a contractual cap), raise it in a deployments PR so the throttling ratio drops below 20%:

resources:
  limits:
    cpu: 250m   # was 100m — raise to clear the 20% throttle ratio
  requests:
    cpu: 100m

This is the fallback when removing the limit is not acceptable. The container will still throttle at the new limit under bursts; size it above the 99th-percentile burst.

Step 4 — Verify the fix

After the new limit takes effect, the throttling ratio drops below 20% within a few scrape intervals and the alert clears (after the for: 10m window). Re-run the Mimir query from Diagnosis step 1 to confirm the ratio is under 0.2.

Verify

• The alert clears once rate(throttled_periods) / rate(periods) <= 0.2 sustained for the for: 10m window.
• kubectl -n $NS get pod $POD -o jsonpath='{.spec.containers[?(@.name=="<container>")].resources}' shows the changed resources (no limits.cpu for the preferred fix, or a raised limit for the fallback).
• The Mimir query from Diagnosis step 1 returns a value <= 0.2.
• For a serving container, request latency returns to normal (check the service's latency dashboard/SLI).

Metric-availability note

Until chart corticph/charts#2923 syncs to a cluster, container_cpu_cfs_throttled_periods_total is absent for that cluster and this alert will not fire there. The container_cpu_cfs_throttled_seconds_total metric (wall-clock seconds throttled) is already collected and can be used as a fallback signal in Explore: rate(container_cpu_cfs_throttled_seconds_total[5m]) / <cpu_limit> > 0.2 gives a similar (periods-based vs seconds-based) view.