Runbook for when a KEDA-scaled HPA's max replicas exceeds the minimum Kafka topic partition count on a production cluster. This is a capacity-planning alert — the consumer group cannot usefully scale beyond the number of partitions (each partition is assigned to at most one consumer in a group).

Meaning

A Kafka consumer group assigns one partition per consumer pod. If the HPA scales to more pods than the topic has partitions, the extra pods are assigned no partitions and sit idle — they consume resources without contributing throughput. This alert fires when an HPA's spec.maxReplicas divided by the minimum topic partition count (across all Kafka topics on the cluster) exceeds 1.

Metrics

kube_horizontalpodautoscaler_spec_max_replicas — the maxReplicas field of a Kubernetes HPA. The value IS the max replica count. Filtered to keda-hpa-.* (KEDA-scaled HPAs are the Kafka consumer autoscalers) and excluding keda-hpa-descheduler.* (not a Kafka consumer).
kminion_kafka_consumer_group_topic_assigned_partitions — the number of partitions assigned to a consumer group for a topic. max by (k8s_cluster_name, topic_name) gives the partition count per topic (assuming at least one consumer group is assigned to all partitions); min by (k8s_cluster_name) gives the smallest topic partition count on the cluster. This replaces DataDog's kafka.kafka_topic_partitions (kminion exposes partition count as a label on kminion_kafka_topic_info, not as a metric value, so the consumer-group-assigned-partitions metric is the closest value-based equivalent).

The on (k8s_cluster_name) group_left join matches each HPA series against the single per-cluster minimum partition count, producing one ratio per HPA per cluster.

What changed from DataDog

The DataDog monitor was scoped to product:co-pilot topics and excluded beta-eu and keda-hpa-deepgram*. This migrated version is generic across all production Kafka clusters — it covers any cluster where kminion reports metrics (prd-par01, prod-weu, prod-us) and any KEDA-scaled HPA, not just co-pilot products. On-prem clusters without kminion metrics in Mimir are naturally excluded by the join.

Impact

When the HPA scales to max, pods beyond the partition count sit idle. This wastes resources (CPU, memory, GPU if applicable) and can mask a capacity ceiling — the service appears to scale but throughput plateaus. No data loss occurs; the consumers with partitions process normally.

This is a P4 informational alert — not urgent, but indicates a misconfiguration that should be corrected in the next capacity review.

Diagnosis

1. Confirm the ratio

The alert fires when maxReplicas / minPartitions > 1. From the alert labels, note horizontalpodautoscaler, cluster, and namespace.

Check the current values (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

# HPA max replicas
QUERY='max by (k8s_cluster_name, horizontalpodautoscaler, namespace) (kube_horizontalpodautoscaler_spec_max_replicas{k8s_cluster_name="<cluster>", horizontalpodautoscaler="<hpa>"})'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

# Minimum topic partition count on the cluster
QUERY='min by (k8s_cluster_name) (max by (k8s_cluster_name, topic_name) (kminion_kafka_consumer_group_topic_assigned_partitions{k8s_cluster_name="<cluster>", environment="production", topic_name=~".*-.*"}))'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

2. Identify the limiting topic

Find which topic has the fewest partitions:

QUERY='max by (k8s_cluster_name, topic_name) (kminion_kafka_consumer_group_topic_assigned_partitions{k8s_cluster_name="<cluster>", environment="production", topic_name=~".*-.*"})'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result | sort_by(.value[1] | tonumber) | .[0]'

3. Check which topic the HPA consumes from

The HPA name usually maps to the service name. Check the KEDA ScaledObject trigger to find the Kafka topic:

export KUBECONFIG=~/.kube/<cluster>.yaml
kubectl get scaledobject -n <namespace> <hpa-name> -o jsonpath='{.spec.triggers}' | jq

Look for the topic or topicList in the trigger spec. The partition count of THAT topic is what matters — the alert uses the cluster-wide minimum as a conservative proxy.

Mitigation

Choose one:

Option A — Increase topic partitions (preferred)

If the service needs to scale to the current maxReplicas, increase the Kafka topic partition count to at least match. Partition count can only increase (never decrease) and requires the consumers to rebalance:

export KUBECONFIG=~/.kube/<cluster>.yaml
kubectl -n kafka exec -it kafka-kafka-0 -- kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic <topic> --partitions <new-count>

This is a Kafka Strimzi cluster — alternatively, update the KafkaTopic CR and let the operator handle it:

kubectl get kafkatopic -n kafka <topic> -o yaml
# edit spec.partitions
kubectl edit kafkatopic -n kafka <topic>

Option B — Lower the HPA maxReplicas

If the service doesn't need that many replicas, lower maxReplicas in the KEDA ScaledObject (or the HPA) to match or be below the partition count. This is a GitOps change in the deployments repo.

Verify

The ratio query returns <= 1 for the HPA.
No idle consumer pods when the HPA is at max (kubectl get pods -n <namespace> — all pods should have assigned partitions).