AKS Upgrades

This document describes the process for upgrading AKS clusters.

Prerequisites

Review the deprecated API migration guide and address any deprecated/removed APIs before proceeding.

Clusters are configured with max surge, allowing AKS to temporarily add extra nodes during upgrades. Production clusters use 20% and staging uses 50%. This lets workloads be drained and rescheduled without disruption.

Procedural environmental upgrades

In development

For the first upgrade to a new Kubernetes version, start by upgrading the dev-weu cluster to identify any risks for a disruptive upgrade.

Parallel cluster upgrades should be limited to clusters within the same environment. Consider upgrading clusters one at a time if you have identified risks associated with the upgrade.

Upgrading

Enable forceful AKS upgrades. A ready-to-use command is available in the cluster overview documentation, or use the template below[^1]
Bump kubernetes.version in clusters/<cluster-identifier>/terraform.tfvars. Raise a PR and review the Terraform plan to validate the version change
Monitor the CI/CD job for successful completion

Handling `terraform apply` failures

If the CI/CD job fails, you can retry the workflow. Common failure scenarios:

PDBs preventing node drain: Workloads with Pod Disruption Budgets can block the upgrade. Review topology spread constraints and pod anti-affinity rules to ensure pods can be rescheduled

[^1]: Template for enabling forceful upgrades for the next 4 hours (macOS date):

az aks update --resource-group <resource-group> --name <cluster-name> --subscription <subscription-id> --enable-force-upgrade --upgrade-override-until $(date -v +4H -u +"%Y-%m-%dT%H:%M:%SZ")