A HorizontalPodAutoscaler (HPA) scales a workload's replica count to keep a metric (CPU, memory, or a custom metric) at a target. It is bounded by minReplicas and maxReplicas. kube_horizontalpodautoscaler_status_desired_replicas is the replica count the HPA wants to scale to; kube_horizontalpodautoscaler_spec_max_replicas is the configured ceiling. The ratio desired / max * 100 is the HPA utilization — how close the HPA is to its ceiling.

avg_over_time(...[12h:]) > 75 means the 12h-average desired/max ratio has exceeded 75%. The HPA is running out of headroom — it has been wanting more than 75% of its max for most of the last 12 hours. The for: 5m window prevents momentary dips in the 12h average from flapping the alert.

Using desired_replicas (not current_replicas) is forward-looking: desired is what the HPA controller computed it needs, before the workload actually scales. When desired == max, the HPA is capped — it wants more but cannot.

namespace!="kube-system" excludes infrastructure HPAs (descheduler, etc.) that are intentionally pinned at min=max=1 and would permanently fire.

Both metrics are gauges from kube-state-metrics (deployed by the synapse-client chart, scraped and remote-written to synapse-server Mimir).

Impact

The HPA is approaching its max. If traffic/queue depth rises further, the HPA cannot scale out and per-pod load grows unbounded → latency rises, pods throttle (see 08-capacity/01-cpu-throttling.md) or OOMKill (see 07-kubernetes/02-pod-crashloop.md).
A 12h-average of 75%+ means the workload's steady-state demand is near the configured max — a capacity-planning signal, not a transient spike.
If the workload is a queue consumer, queue depth grows while the consumer count is capped → backlog.
Scaling this out increases load on downstream dependencies. Every extra replica is another client of the workload's dependencies — the database, Keycloak, the kube-apiserver, and any internal service the workload calls. Raising maxReplicas can push that load onto a dependency sized for the current replica count.

Diagnosis

The alert labels carry k8s_cluster_name, environment, namespace, and horizontalpodautoscaler. From the alert, note all of them.

1. Confirm the HPA utilization and identify the workload

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=<namespace>
HPA=<horizontalpodautoscaler>
kubectl -n $NS get hpa $HPA
kubectl -n $NS describe hpa $HPA | sed -n '/^Conditions:/,$p'

kubectl get hpa shows REFERENCE (the target Deployment/StatefulSet), TARGETS (current/target ratio), MINPODS, MAXPODS, REPLICAS. Confirm REPLICAS is close to MAXPODS. The Conditions section shows ScalingLimited with Reason: TooManyReplicas when the HPA is explicitly capped.

2. Is the per-pod load legitimately high, or is there a bottleneck?

kubectl -n $NS get deploy <target-deployment>   # or sts
kubectl -n $NS top pods -l app=<target>

Legitimate demand — traffic/queue depth is genuinely high; the workload needs more replicas. Raise maxReplicas (Mitigation step 2).
Per-pod inefficiency — a downstream bottleneck (DB, slow API), a missing cache, an N+1 query, or a hot path makes each pod do more work than it should. Fix the inefficiency so each pod handles more load and the HPA does not need to scale out as far.
Wrong HPA metric — the HPA targets CPU but the real bottleneck is memory, or targets a custom metric that is miscalibrated. The describe hpa Metrics section shows the current/target per metric.

3. Check the HPA's ScalingLimited condition

kubectl -n $NS get hpa $HPA -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason} msg={.message}{"\n"}{end}'

ScalingLimited / TooManyReplicas — confirmed capped at or near max.
AbleToScale / ScaleDownStabilized — normal; the HPA is not actively blocked, just near the ceiling.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Raise maxReplicas (if the workload legitimately needs more capacity)

If the per-pod load is legitimate and the workload needs more replicas, raise maxReplicas in the chart values:

# example: common-chart HPA values
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 30   # was 15 — raise to clear the 75%-of-max plateau
  targetCPUUtilizationPercentage: 70

For an ArgoCD-managed workload, the fix is GitOps: edit the values, merge, ArgoCD syncs and the HPA controller immediately allows scaling out to the new max. Confirm the node pool has capacity for the extra pods (or the cluster autoscaler will add nodes — see 08-capacity/05-nodegroup-utilization-high.md).

Caution

Check downstream capacity before raising the max. Every extra replica is another client of the workload's dependencies — scaling N→2N pods roughly doubles the fan-out load on:

The database (CNPG) — more connection-pool pressure, more concurrent queries. A CNPG cluster sized for the current pod count can hit CPU/connection limits when the HPA doubles the replicas.
Keycloak — more token-introspection / userinfo requests.
The kube-apiserver — every pod's controllers/informers open watches and list calls; N extra pods = N× watch load.
Internal services the workload calls (gRPC/HTTP downstreams, the AI Gateway, Kafka brokers) — each sees N× client connections / N× request rate.

Raise maxReplicas gradually (e.g. 15→20, watch the downstream metrics for 30m, then 20→25) rather than jumping to 30. If a dependency is already near its limit, raising the HPA max trades this alert for a downstream alert — fix the dependency's capacity first.

Step 3 — Fix the per-pod load (if there is a bottleneck)

If the per-pod load is high due to a bottleneck, raising maxReplicas only buys time — the real fix is the efficiency:

Downstream bottleneck — DB, slow dependency, rate-limited upstream. Fix the downstream or add caching.
Inefficient code path — profile the hot path (the app's pprof / tracing) and optimize.
Wrong HPA metric — retarget the HPA at the metric that actually correlates with load.

Step 4 — Lower the target (if the HPA is overly aggressive)

If the HPA's targetCPUUtilizationPercentage is very low (e.g. 50%), the HPA scales out aggressively and hits max before the pods are actually overloaded. Raising the target (e.g. to 70–80%) lets pods run hotter before scaling, staying under max with the same real capacity.

Verify

The alert clears once the 12h-average desired / max * 100 drops below 75% sustained for for: 5m. Note the 12h average: even after you raise the max or reduce load, the alert takes up to 12h to clear (it requires the 12h window average to drop below threshold). Use the Mimir query to confirm the current ratio has dropped immediately.
kubectl -n $NS get hpa $HPA shows REPLICAS < MAXPODS.
kubectl -n $NS describe hpa $HPA no longer shows ScalingLimited / TooManyReplicas.

Notes

Metric — desired_replicas (not current_replicas): desired is what the HPA wants to scale to, forward-looking. When desired == max, the HPA is capped.
12h average — avg_over_time(...[12h:]); takes up to 12h to clear after the ratio drops.
kube-system excluded — namespace!="kube-system": infrastructure HPAs (descheduler) pinned at min=max=1 would permanently fire.
All environments — k8s_cluster_name=~"prd-.*|stg-.*|dev-.*": production, staging, and development.
No product label — the DataDog query groups by product; Mimir has no product label, so namespace is used instead.
Related — 08-capacity/05-nodegroup-utilization-high.md (full nodegroup pairs with HPA at max); 08-capacity/01-cpu-throttling.md (per-pod overload); 07-kubernetes/02-pod-crashloop.md (OOM/throttle).