Orca ORCA v0.174.2
Welcome Developer Guide Infrastructure Platform Engineering Runbooks Archive
Documentation navigation
On Premise Customers
SFD Hamilton County CommonSpirit GMR Region Nord On-Prem Synapse API-Key Deploy
Dns
DNS Flush
Kafka
Kafka Consumer Lag in Voice-Vault Kafka Consumer Group Lag High (> 800) HPA Max Replicas Exceed Kafka Topic Partitions
Storage
Ceph OSD Full / BlueFS Allocation Wedge Query Ceph RGW (S3) Bucket Usage Ceph Mon Quorum Loss / Node-Replacement Recovery PVC Near Full Warning (>= 80%) PVC Near Full Critical (>= 90%) PVC Disk Usage Forecast > 95% Ceph Cluster Storage Utilization High (> 90%) Ceph Cluster Error Ceph Cluster Warning
Observability
Alloy Operator Wedge Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild Grafana Dashboard Provisioning Error Mimir Ingester Reaching Series Limit Mimir Ingest Storage Wedged Mimir Ingester PVC Near Full Loki PVC Near Full Mimir Self-Monitoring Scrape Stalled Synapse Cluster Metrics Silent Mimir Discarding Samples (Series Limit) Loki Discarding Log Entries Trivy Metrics Stale Envoy Proxy Heap Near Ceiling Envoy Proxy Heap Shedding Envoy Proxy Overload Rejecting Requests Traefik Elevated Error Spike (4xx/5xx) Envoy Elevated Error Spike (4xx/5xx) Scrape Targets Failing Traefik 402 Payment Failure (Multiple Distinct Users) Azure Audit Ingestion Silent
Databases
CNPG Cluster Has No Backup CNPG Replication Unhealthy CNPG Cluster Backup Stale (> 48h) CNPG Cluster Has No WAL Recovery Point
Kubernetes
Pod ImagePull / ImagePullBackOff Pod CrashLoopBackOff Deployment Has No Running Replicas Deployment Rollout Stuck DaemonSet Has No Running Pods StatefulSet Has No Running Replicas Kubernetes Job Failed Kubernetes CronJob Suspended Certificate Expiring Soon Node Memory/Disk Pressure (Kubelet Evicting Pods) Container OOMKilled Cluster Autoscaler Unhealthy Trivy Critical Vulnerability One Week Trivy Critical Vulnerability Four Weeks Trivy High Vulnerability One Week Trivy High Vulnerability Six Weeks Certificate Renewal Stuck Certificate Never Issued Trivy Critical Vulnerability Immediate API v2 Debug Endpoint Called in Production Cluster Autoscaler Unregistered Nodes Platform API Server Errors Platform API Client Errors Pod Pending Over 30 Minutes Pod Pending (GPU) Over 60 Minutes Cilium Policy Drops Burst
Capacity
CPU Throttling > 20% HPA Utilization Critically High GPUs Underutilized For 48h Container Memory Leak (Rising Near Limit) Nodegroup Utilization Critically High Nodegroup Utilization High Warning
Identity
Keycloak Client Misconfigured
Synthetic Monitoring
Synthetic Feature Failing Synthetic Monitor Silent Speech to Text – Synthetic Check Failing Corti Models – Synthetic Check Failing Text Generation – Synthetic Check Failing
Platform
Orca GitHub Rate-Limit Budget Low Email Service Minute Quota High Kyverno Admission API Server Unreachable Email Service Hourly Quota High Email Service Send Throttled Email Service Bounce Rate High

Deployment Rollout Stuck

Runbook for a Deployment whose rollout has stopped progressing, triggered by the DeploymentRolloutStuck Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). There is no matching Mimir-ruler alert.

This is the sibling of 03-deployment-no-running-replicas.md. The two are distinct:

  • No running replicas (spec > 0 AND ready == 0) — zero pods serving, service is down. Severe.
  • Rollout stuck (this alert) — the new ReplicaSet is not progressing, but old pods may still be serving. The service is often still up on the previous version. Lower urgency.

Meaning

A Deployment rollout works by creating a new ReplicaSet and scaling it up while scaling the old ReplicaSet down. kube_deployment_status_replicas_updated is the number of pods on the new spec; kube_deployment_spec_replicas is the desired total. kube_deployment_status_replicas_unavailable is the number of pods not yet available (Ready + minReadySeconds elapsed).

The alert fires when, for 30m+:

  • updated < spec — the new ReplicaSet has not finished rolling out, AND
  • unavailable > 0 — some pods are not available.

The unavailable > 0 condition is what distinguishes "stuck" from "briefly mid-rollout": a healthy rolling update has updated < spec for a short window while pods come up, but unavailable stays at 0 (old pods still serving while new ones Ready up). A stuck rollout has unavailable replicas — the new pods cannot reach Ready, so the rollout cannot complete. The for: 30m rides out healthy rollouts (which converge in seconds to a few minutes).

All metrics are gauges from kube-state-metrics (deployed by the synapse-client chart, scraped on the cluster and remote-written to synapse-server Mimir). The and on (k8s_cluster_name, environment, namespace, deployment) matches the series per-Deployment. Each leg is pre-aggregated with sum by (...) to strip the scrape-integration labels (job, instance, source), so the alert carries only k8s_cluster_name, environment, namespace, and deployment and reads as a workload problem — not as a problem with kube-state-metrics itself. Each stuck Deployment becomes its own alert instance.

Impact

  • The service is usually still up on the old ReplicaSet — old pods keep serving. So the user impact is often "the new version is not deployed" rather than "the service is down."
  • If the rollout was triggered by an urgent fix (security, crash), the fix is not reaching production — the old (buggy/vulnerable) version keeps running.
  • The new ReplicaSet's pods are failing (crash, pull, probe) and the controller keeps trying to scale them up, burning CPU/restarts on the node.
  • If the old ReplicaSet was already scaled to 0 (e.g. by a config that also broke the old pods), this can degrade to the no-running-replicas condition — check 03-deployment-no-running-replicas.md if ready == 0.

Diagnosis

The alert labels carry k8s_cluster_name, namespace, deployment, and environment. From the alert in Grafana, note all of them.

1. Confirm the rollout is stuck (not just slow)

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=<namespace>
DEPLOY=<deployment>
kubectl -n $NS get deploy $DEPLOY
kubectl -n $NS get rs -l app=<deployment> -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas,AGE:.metadata.creationTimestamp'

kubectl get deploy shows READY and UP-TO-DATE (updated). A stuck rollout shows UP-TO-DATE < DESIRED for a long time. List ReplicaSets: the newest is the one being rolled out (at DESIRED > 0 but READY low/zero); older ones may still have pods.

2. Inspect the new ReplicaSet's pods

The new ReplicaSet is the one not progressing. Its pods hold the blocker:

NEW_RS=$(kubectl -n $NS get rs -l app=<deployment> --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
kubectl -n $NS get pods -l pod-template-hash=<new-rs-template-hash> -o wide
kubectl -n $NS get pods -l app=<deployment> -o jsonpath='{range .items[*]}{.metadata.name}: {range .status.containerStatuses[*]}{.state.waiting.reason}{" "}{end}{"\n"}{end}'

The pod states route to sibling runbooks:

  • ErrImagePull / ImagePullBackOff → 07-kubernetes/01-pod-imagepull.md (bad image tag is the most common rollout-stuck cause — a new tag that was never pushed).
  • CrashLoopBackOff → 07-kubernetes/02-pod-crashloop.md (a config change that only the new pods hit — missing env var, bad config, a migration that fails on the new code).
  • Running but not Ready → readinessProbe failing on the new code (a probe path that changed, a dependency the new version needs but is not wired). Fix the probe target or the dependency.
  • Pending → unschedulable (new pod requests more than available) or PVC/config missing. See 03-deployment-no-running-replicas.md step 3.

3. Check rollout blockers (quota, PDB, progressDeadlineSeconds)

The Deployment controller can also stall without the pods being obviously broken:

kubectl -n $NS describe deploy $DEPLOY | sed -n '/^Conditions:/,$p'
kubectl -n $NS get resourcequota
kubectl -n $NS get poddisruptionbudget
  • ProgressDeadlineExceeded condition → the rollout hit spec.progressDeadlineSeconds (default 600s). The controller has given up. The pods are failing (step 2) — fix the root cause, then trigger a new rollout.
  • Resource quota — a ResourceQuota in the namespace caps CPU/memory/pod count and the new ReplicaSet cannot schedule. Check kubectl describe resourcequota for hard vs used.
  • PodDisruptionBudget — a PDB can block a rollout if it prevents old pods from being removed (minAvailable too high). Rare for a stuck rollout (PDBs mostly affect voluntary disruptions), but check.

4. Confirm the old ReplicaSet is still serving (impact check)

kubectl -n $NS get rs -l app=<deployment> -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas'

If an old ReplicaSet still has READY > 0, the service is up on the old version. If all ReplicaSets show READY 0, the service is down — switch to 03-deployment-no-running-replicas.md.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the new ReplicaSet's blocker

Follow Diagnosis steps 2–3 to the root cause:

  • Bad image tag — roll the chart config.json / image tag back to the last known-good tag (or push the missing tag to ACR). See 07-kubernetes/01-pod-imagepull.md.
  • Config / env var / secret only the new pods hit — fix the chart values or ExternalSecret in a PR. See 07-kubernetes/02-pod-crashloop.md.
  • Resource quota — raise the quota or lower the new pod's requests in a chart-values PR.
  • ProgressDeadlineExceeded — fix the pod root cause (step 2), then trigger a new rollout so the controller retries: kubectl -n $NS rollout restart deploy $DEPLOY (or push a new chart revision; ArgoCD self-heal reverts a manual restart, so prefer the GitOps path).

Step 3 — Roll back to the last good ReplicaSet (emergency)

If the new version is bad and the fix will take too long, roll back to the previous ReplicaSet so the rollout converges on the old (working) version:

kubectl -n $NS rollout undo deploy $DEPLOY
# or explicitly:
kubectl -n $NS get rs -l app=<deployment> -o custom-columns='NAME:.metadata.name,READY:.status.readyReplicas'
kubectl -n $NS rollout undo deploy $DEPLOY --to-revision=<last-good-revision>

rollout undo flips spec.template back to the previous revision; the controller scales the new (bad) RS down and the old (good) RS up. This is the fast path to get the rollout un-stuck. ArgoCD will reconcile to the desired state on the next sync — make the chart-values change match (roll back the tag) in a PR so GitOps and the live state agree.

Step 4 — Verify the rollout completes

After the fix, updated rises to match spec and unavailable drops to 0:

kubectl -n $NS rollout status deploy $DEPLOY
kubectl -n $NS get deploy $DEPLOY -o jsonpath='desired={.spec.replicas} updated={.status.updatedReplicas} ready={.status.readyReplicas}{"\n"}'

Verify

  • The alert clears once updated == spec AND unavailable == 0 sustained for the for: 30m window.
  • kubectl -n $NS get deploy $DEPLOY shows UP-TO-DATE == DESIRED and READY == DESIRED.
  • kubectl -n $NS rollout status deploy $DEPLOY reports "deployment successfully rolled out".
  • For a rollback, the live image matches the rolled-back chart tag.