Runbook for a cluster-autoscaler nodegroup whose 24h-average utilization reaches 75% of its max size. This is the warning tier of the nodegroup utilization alert — the critical tier (>= 90%) is 08-capacity/05-nodegroup-utilization-high.md.

Meaning

Same mechanism as 08-capacity/05-nodegroup-utilization-high.md — the ratio cloudprovidertarget_total / cloudprovidermaxsize_total * 100 measures how close a nodegroup is to its ceiling. avg_over_time(...[24h:]) >= 75 means the 24h-average utilization has reached 75%.

Differences from the critical tier (05):

Threshold — >= 75 (warning) vs > 90 (critical). This fires earlier, giving time to act before the nodegroup hits the ceiling.
All nodegroups — no kafka exclusion. Kafka nodegroups are included here so their growth is tracked; the critical tier excludes them (kafka is sized independently).
Aggregation — avg by (DataDog avg: rollup) vs max by in the critical tier.

Impact

The nodegroup is approaching its max. If demand rises further, the autoscaler will be unable to add nodes and pods will go Pending.
A 24h-average of 75%+ means the nodegroup has been consistently near capacity — not a transient burst.

Diagnosis

See 08-capacity/05-nodegroup-utilization-high.md § Diagnosis — the same procedure applies. Use this Mimir query instead (threshold 75, no kafka exclusion):

QUERY='avg_over_time((avg by (k8s_cluster_name, nodegroup, environment) (cluster_autoscaler_status_cloudprovidertarget_total{k8s_cluster_name="<cluster>", nodegroup="<nodegroup>"}) / avg by (k8s_cluster_name, nodegroup, environment) (cluster_autoscaler_status_cloudprovidermaxsize_total{k8s_cluster_name="<cluster>", nodegroup="<nodegroup>"}) * 100)[24h:])'

Mitigation

See 08-capacity/05-nodegroup-utilization-high.md § Mitigation — same steps (activate PIM, raise nodegroup max, un-strand idle nodes, verify).

Verify

The Mimir query above returns a value under 75.
No pods Pending due to max node group size exceeded.
The critical alert (08-capacity/05-nodegroup-utilization-high.md) is not firing for this nodegroup.

Notes

Warning tier — >= 75, severity warning. The critical tier (> 90, severity critical) is 08-capacity/05-nodegroup-utilization-high.md.
All nodegroups — kafka nodegroups are included (unlike the critical tier).
All environments — k8s_cluster_name=~"prd-.*|stg-.*|dev-.*": production, staging, and development.
24h average — avg_over_time(...[24h:]); takes up to 24h to clear after the ratio drops.
Related — 08-capacity/05-nodegroup-utilization-high.md (critical tier); 08-capacity/02-hpa-at-max-replicas.md (paired HPA signal); 07-kubernetes/12-cluster-autoscaler-unhealthy.md (autoscaler broken).