Cluster Autoscaler Unhealthy

Runbook for a cluster-autoscaler that has stopped being healthy, covering the ClusterAutoscalerUnhealthy alert (this runbook) and the ClusterAutoscalerUnregisteredNodes alert (both route here — they share the autoscaler deployment and the diagnosis).

Meaning

The cluster-autoscaler runs in kube-system and scales node groups up/down based on Pending pods. A cluster-autoscaler-status-exporter (deployed in the shared namespace) scrapes the autoscaler's status and exposes cluster_autoscaler_status_* metrics, remote-written to Mimir.

• ClusterAutoscalerUnhealthy — cluster_autoscaler_status_health_status{status="Healthy"} != 1 for 5m. The status="Healthy" series is 1 when healthy, 0 when unhealthy. min by (k8s_cluster_name) ... != 1 fires when any autoscaler instance on the cluster reports unhealthy.
• ClusterAutoscalerUnregisteredNodes — cluster_autoscaler_status_long_unregistered_total >= 1 for 5m. Nodes the autoscaler tried to add but never registered with the Kubernetes API — almost always a cloud-provider authentication issue.

Both are scoped to k8s_cluster_name=~"prd-.*|stg-.*|dev-.*" — the par (Scaleway/kommodity) and AKS GPU clusters that run cluster-autoscaler, across production, staging, and development. The AKS general-service clusters (prod-weu, prod-us) do not run cluster-autoscaler and have no series. The metrics are already scraped into Mimir — no chart PR is needed.

Impact

• The cluster stops scaling in response to Pending pods — workloads that need more nodes sit Pending indefinitely. HPAs and KEDA ScaledObjects scale pods, but the pods cannot schedule if the autoscaler is not adding nodes.
• For ClusterAutoscalerUnregisteredNodes specifically: the autoscaler is requesting nodes from the cloud provider but they never join the cluster — capacity is requested but never delivered, and the autoscaler may be stuck retrying.
• Scale-down is also affected — the autoscaler may not remove underutilized nodes, leaving capacity (and cost) stranded.

Diagnosis

The alert label carries k8s_cluster_name (and environment). From the alert, note both.

1. Confirm the autoscaler is unhealthy / has unregistered nodes

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml

Confirm via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

# health (should be 1; 0 = unhealthy)
QUERY='min by (k8s_cluster_name) (cluster_autoscaler_status_health_status{status="Healthy", k8s_cluster_name="<cluster>"})'
# unregistered nodes (should be 0)
QUERY2='max by (k8s_cluster_name) (cluster_autoscaler_status_long_unregistered_total{k8s_cluster_name="<cluster>"})'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

2. Check the autoscaler logs

kubectl -n kube-system logs -l app.kubernetes.io/name=cluster-autoscaler --tail=100

Common failure patterns:

• failed to register node / node failed to register — nodes the autoscaler added are not joining the cluster. For ClusterAutoscalerUnregisteredNodes this is the signature. Causes: cloud-provider auth (the node's kubelet cannot authenticate to the API server), a broken node bootstrap script, a misconfigured cloud-provider config, or the node never reaching Ready.
• cloud provider error / Failed to ensure nodes — the autoscaler cannot talk to the cloud provider (Scaleway / Azure) to request nodes. Check the autoscaler's cloud-provider credentials (secret/Kubeconfig) and egress to the cloud API.
• Cluster autoscaler is not healthy (self-reported) — the autoscaler's internal health check failed; the logs before this line name the failing component.
• Scale-up stuck — PodUnschedulable Pending pods but no ScaleUp activity in the logs: the autoscaler sees the pods but is not acting. Check for a --scale-down-delay, a paused autoscaler, or a max-node cap reached (see 08-capacity/05-nodegroup-utilization-high.md).

3. Check the autoscaler deployment

kubectl -n kube-system get deploy -l app.kubernetes.io/name=cluster-autoscaler -o wide
kubectl -n kube-system get pods -l app.kubernetes.io/name=cluster-autoscaler -o jsonpath='{range .items[*]}{.metadata.name} restarts={.status.containerStatuses[0].restartCount} phase={.status.phase}{"\n"}{end}'

A crashlooping autoscaler pod reports unhealthy. See 02-pod-crashloop.md for crashloop diagnosis.

4. For unregistered nodes — check the node and the MachineDeployment

kubectl get nodes -o wide | grep -iE 'NotReady|unknown'
# For kommodity/Cluster-API clusters, the nodegroup is a MachineDeployment:
kubectl get machinedeployment -A | grep <cluster>
kubectl get machines -A | grep -iE 'failed|error'

An unregistered node shows as NotReady or never appears in kubectl get nodes. The MachineDeployment/Machine status names the registration failure.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the autoscaler

The fix depends on the failure mode from Diagnosis:

• Cloud-provider auth / credentials — the autoscaler's cloud-provider config is a Secret/Kubeconfig mounted into the pod. If expired or rotated, update it via a GitOps change to the autoscaler chart values. The Scaleway clusters use a Scaleway access key; the AKS GPU clusters use the AKS cloud-provider config.
• Crashlooping autoscaler — see 02-pod-crashloop.md. A restart (kubectl -n kube-system rollout restart deploy/cluster-autoscaler) may recover a transient crash, but identify the crash reason first.
• Unregistered nodes (kommodity) — the node bootstrap (talos/kommodity) failed to join the node to the API server. Check the Machine status and the node's bootstrap logs. This is usually a Scaleway auth or network issue at node boot, not an autoscaler bug — the autoscaler requested the node correctly; the node failed to register. Fix the bootstrap / cloud config, then the autoscaler's next scale-up succeeds.
• Unregistered nodes (AKS) — the VMSS failed to provision or the node never reached Ready. Check the Azure VMSS activity log for provisioning failures.

Kommodity-specific fix (Scaleway sovereign clusters)

For kommodity-managed clusters (prd-par01, prd-par02, prd-ric02, stg-par01, stg-par02, dev-par01, dev-par02), the autoscaler wedge is often caused by an expired or stale signing key / kubeconfig secret. The recovery procedure:

1. Delete the *-kubeconfig secret in the downstream cluster (the cluster the autoscaler runs on):

   kubectl -n kube-system delete secret <cluster>-kubeconfig
   

2. Delete the signing key secret in kommodity (the management cluster):

   kubectl delete secret <cluster>-signing-key
   

3. Wait for automatic recreate — kommodity's reconciler regenerates the signing key and the kubeconfig secret automatically.
4. Validate the *-kubeconfig secret is recreated in the downstream cluster:

   kubectl -n kube-system get secret <cluster>-kubeconfig
   

5. Restart the autoscaler:

   kubectl -n kube-system rollout restart deploy/cluster-autoscaler
   

Step 3 — Verify

The alert clears once cluster_autoscaler_status_health_status{status="Healthy"} returns to 1 (and long_unregistered_total drops to 0 for the nodes alert) sustained for the for: 5m window. Confirm the autoscaler is scaling: Pending pods get nodes, and the logs show ScaleUp activity.

Verify

• ClusterAutoscalerUnhealthy clears once min by (k8s_cluster_name) (...health_status{status="Healthy"}) is 1 sustained for 5m.
• ClusterAutoscalerUnregisteredNodes clears once max by (k8s_cluster_name) (...long_unregistered_total) is 0 sustained for 5m.
• kubectl get nodes shows the previously-unregistered nodes as Ready (or removed if they were garbage-collected).
• The autoscaler logs show ScaleUp / ScaleDown activity resuming.

Notes

• Scope — k8s_cluster_name=~"prd-.*|stg-.*|dev-.*": covers all par clusters (Scaleway/kommodity) and AKS GPU clusters that run cluster-autoscaler, across production, staging, and development. The AKS general-service clusters do not run it.
• Two alerts, one runbook — ClusterAutoscalerUnhealthy (autoscaler self-reported health) and ClusterAutoscalerUnregisteredNodes (nodes the autoscaler added never registered) both point at the autoscaler deployment and share the diagnosis.
• All environments — production, staging, and development clusters are all covered. An unhealthy autoscaler in dev blocks dev work; in staging it blocks pre-prod validation. No environment= filter — the k8s_cluster_name=~"prd-.*|stg-.*|dev-.*" filter is the scope.
• Related — 08-capacity/05-nodegroup-utilization-high.md for nodegroup capacity ceiling; 08-capacity/02-hpa-at-max-replicas.md for the HPA-at-max signal that often pairs with autoscaler scaling.