The cluster-autoscaler tracks, per nodegroup, the current desired node count (cloudprovidertarget_total) and the nodegroup's configured maximum (cloudprovidermaxsize_total). The ratio target / maxsize * 100 is the nodegroup utilization — how close the nodegroup is to its ceiling. When a nodegroup is at its max, the autoscaler cannot add more nodes to that nodegroup; pods that would schedule there go Pending instead.

avg_over_time(...[24h:]) > 90 means the 24h-average utilization has exceeded 90%. The for: 5m window prevents momentary spikes in the 24h average from flapping the alert. Unlike a continuous for: 24h condition, the 24h average can tolerate brief dips below 90% — it fires when the average over the full day is above 90%, meaning the nodegroup has been near its ceiling for most of the day.

Kafka nodegroups (nodegroup=~".*kafka.*") are excluded — they are sized independently and monitored via Kafka-specific alerts.

The nodegroup label is the cloud-provider nodegroup identifier:

kommodity/Cluster-API (par clusters) — MachineDeployment/<namespace>/<machinedeployment-name>, e.g. MachineDeployment/default/prd-par01-worker-stateful-workloads-nodepool-fr-par-2.
AKS GPU clusters — the VMSS name, e.g. aks-gpuh100x71w0-25472401-vmss.

Scoped to k8s_cluster_name=~"prd-.*|stg-.*|dev-.*" — all par (Scaleway/kommodity) and AKS clusters that run cluster-autoscaler, across production, staging, and development.

Impact

The nodegroup is near its max. If workload demand rises further, the autoscaler cannot add nodes to this nodegroup and pods go Pending (unschedulable).
A nodegroup stuck at high utilization for 24h also cannot scale down — there is always enough workload to keep the nodes busy. This may be legitimate (the workload genuinely needs the capacity) or a sign of a stranded/underutilized node that the autoscaler cannot remove (PodDisruptionBudget, kube-system pods, or a scale-down-disabled annotation blocking removal).
Cost: running at 90%+ of max is fine if the capacity is used; the concern is the ceiling — if max needs to rise, this is the signal to do it before a Pending-pod incident.

Diagnosis

The alert labels carry k8s_cluster_name, environment, nodegroup. From the alert, note all of them.

1. Confirm the utilization and see the nodegroup's limits

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml

Confirm via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

QUERY='avg_over_time((max by (k8s_cluster_name, nodegroup, environment) (cluster_autoscaler_status_cloudprovidertarget_total{k8s_cluster_name="<cluster>", nodegroup="<nodegroup>"}) / max by (k8s_cluster_name, nodegroup, environment) (cluster_autoscaler_status_cloudprovidermaxsize_total{k8s_cluster_name="<cluster>", nodegroup="<nodegroup>"}) * 100)[24h:])'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

A value > 90 confirms the alert. Check the nodegroup's configured min/max:

# kommodity (MachineDeployment)
kubectl get machinedeployment <nodegroup-name> -n <nodegroup-namespace> -o jsonpath='{.spec.replicas} min/max not in spec; check the autoscaler config'
# AKS (VMSS) — the min/max is in the cluster-autoscaler deployment args:
kubectl -n kube-system get deploy -l app.kubernetes.io/name=cluster-autoscaler -o yaml | grep -A30 'args:'

The autoscaler's --nodes=<min>:<max>:<nodegroup> flags define the min/max per nodegroup.

2. Is the nodegroup full because of real demand or stranded nodes?

Check whether the nodes are actually busy or just unable to scale down:

kubectl get nodes -l <node-label-for-the-nodegroup> -o wide
kubectl describe nodes | grep -A5 'Allocated resources'   # per-node CPU/memory pressure

Nodes busy (high allocated %) — real demand; the nodegroup legitimately needs more capacity. Raise the max (Mitigation step 2).
Nodes not busy but not scaling down — stranded nodes. Check for:
- A cluster-autoscaler.kubernetes.io/scale-down-disabled=true annotation on a node (kubectl get node <node> -o jsonpath='{.metadata.annotations}').
- PodDisruptionBudgets blocking eviction (kubectl get pdb -A).
- kube-system / daemonset pods that pin a node (the autoscaler will not remove a node with a non-evictable pod).

3. Are pods Pending because the nodegroup is full?

kubectl get pods -A --field-selector=status.phase=Pending -o wide
kubectl get pods -A --field-selector=status.phase=Pending -o jsonpath='{range .items[*]}{.metadata.name} unschedulable={.status.conditions[?(@.type=="PodScheduled")].message}{"\n"}{end}'

If Pending pods have 0/6 nodes are available: 6 node(s) had max node group size exceeded (or similar), the nodegroup's max is the blocker — this alert's condition is causing real Pending pods.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Raise the nodegroup max (the durable fix for real demand)

If the nodegroup is full because of legitimate demand (Diagnosis step 2, nodes busy), raise the max so the autoscaler can add nodes when demand spikes. This is a GitOps change to the autoscaler config (the --nodes=<min>:<max>:<nodegroup> arg) in the synapse-client / cluster-autoscaler chart values:

# the autoscaler deployment args, e.g.
--nodes=<min>:<new-max>:<nodegroup>

Do not kubectl edit the deployment (ArgoCD self-heal reverts it). Open a deployments PR. After the merge, the autoscaler picks up the new max on its next sync.

Before raising, confirm the cloud provider has the quota (Scaleway instance quota, Azure VMSS vCPU quota) for the larger max — a max the cloud provider cannot fulfill is silently capped.

Step 3 — Un-strand nodes (if the nodegroup is full but nodes are idle)

If the nodes are not busy (Diagnosis step 2, stranded), remove the scale-down blocker so the autoscaler can remove idle nodes and the utilization reflects real demand:

Remove cluster-autoscaler.kubernetes.io/scale-down-disabled annotations that are no longer needed.
For a PDB that is too strict, relax it in a GitOps change.
For daemonset-pinned nodes, the only lever is the daemonset's scheduling — usually not worth changing; accept the stranded node.

Step 4 — Verify

The alert clears once the 24h-average target / maxsize * 100 drops below 90% sustained for for: 5m. Note the 24h average: even after you raise the max or free nodes, the alert takes up to 24h to clear (it requires the whole 24h window average to drop below threshold). Use the Mimir query in Diagnosis step 1 to confirm the current ratio has dropped immediately, even though the alert stays firing until a full 24h of lower utilization brings the average below 90%.

Verify

The Mimir query from Diagnosis step 1 returns a value under 90 (immediate signal, even though the alert's 24h average has not yet elapsed).
kubectl get pods -A --field-selector=status.phase=Pending shows no pods blocked by max node group size exceeded.
The autoscaler logs show ScaleUp succeeding when demand rises (after raising the max).

Notes

Scope — k8s_cluster_name=~"prd-.*|stg-.*|dev-.*": all par (Scaleway/kommodity) and AKS clusters that run cluster-autoscaler, across production, staging, and development.
Kafka excluded — nodegroup!~".*kafka.*": Kafka nodegroups are sized independently and monitored via Kafka-specific alerts.
24h average — avg_over_time(...[24h:]) computes the mean utilization over the past day. The alert fires when this average exceeds 90%, meaning the nodegroup has been near its ceiling for most of the day. It takes up to 24h to clear after the ratio drops.
nodegroup label shape — MachineDeployment/<ns>/<name> on kommodity, <vmss-name> on AKS. Parse accordingly when looking up the nodegroup config.
Related — 08-capacity/02-hpa-at-max-replicas.md (HPA at max often pairs with a full nodegroup); 07-kubernetes/12-cluster-autoscaler-unhealthy.md (the autoscaler itself broken, not just at capacity).