Orca ORCA v0.174.2
Welcome Developer Guide Infrastructure Platform Engineering Runbooks Archive
Documentation navigation
On Premise Customers
SFD Hamilton County CommonSpirit GMR Region Nord On-Prem Synapse API-Key Deploy
Dns
DNS Flush
Kafka
Kafka Consumer Lag in Voice-Vault Kafka Consumer Group Lag High (> 800) HPA Max Replicas Exceed Kafka Topic Partitions
Storage
Ceph OSD Full / BlueFS Allocation Wedge Query Ceph RGW (S3) Bucket Usage Ceph Mon Quorum Loss / Node-Replacement Recovery PVC Near Full Warning (>= 80%) PVC Near Full Critical (>= 90%) PVC Disk Usage Forecast > 95% Ceph Cluster Storage Utilization High (> 90%) Ceph Cluster Error Ceph Cluster Warning
Observability
Alloy Operator Wedge Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild Grafana Dashboard Provisioning Error Mimir Ingester Reaching Series Limit Mimir Ingest Storage Wedged Mimir Ingester PVC Near Full Loki PVC Near Full Mimir Self-Monitoring Scrape Stalled Synapse Cluster Metrics Silent Mimir Discarding Samples (Series Limit) Loki Discarding Log Entries Trivy Metrics Stale Envoy Proxy Heap Near Ceiling Envoy Proxy Heap Shedding Envoy Proxy Overload Rejecting Requests Traefik Elevated Error Spike (4xx/5xx) Envoy Elevated Error Spike (4xx/5xx) Scrape Targets Failing Traefik 402 Payment Failure (Multiple Distinct Users) Azure Audit Ingestion Silent
Databases
CNPG Cluster Has No Backup CNPG Replication Unhealthy CNPG Cluster Backup Stale (> 48h) CNPG Cluster Has No WAL Recovery Point
Kubernetes
Pod ImagePull / ImagePullBackOff Pod CrashLoopBackOff Deployment Has No Running Replicas Deployment Rollout Stuck DaemonSet Has No Running Pods StatefulSet Has No Running Replicas Kubernetes Job Failed Kubernetes CronJob Suspended Certificate Expiring Soon Node Memory/Disk Pressure (Kubelet Evicting Pods) Container OOMKilled Cluster Autoscaler Unhealthy Trivy Critical Vulnerability One Week Trivy Critical Vulnerability Four Weeks Trivy High Vulnerability One Week Trivy High Vulnerability Six Weeks Certificate Renewal Stuck Certificate Never Issued Trivy Critical Vulnerability Immediate API v2 Debug Endpoint Called in Production Cluster Autoscaler Unregistered Nodes Platform API Server Errors Platform API Client Errors Pod Pending Over 30 Minutes Pod Pending (GPU) Over 60 Minutes Cilium Policy Drops Burst
Capacity
CPU Throttling > 20% HPA Utilization Critically High GPUs Underutilized For 48h Container Memory Leak (Rising Near Limit) Nodegroup Utilization Critically High Nodegroup Utilization High Warning
Identity
Keycloak Client Misconfigured
Synthetic Monitoring
Synthetic Feature Failing Synthetic Monitor Silent Speech to Text – Synthetic Check Failing Corti Models – Synthetic Check Failing Text Generation – Synthetic Check Failing
Platform
Orca GitHub Rate-Limit Budget Low Email Service Minute Quota High Kyverno Admission API Server Unreachable Email Service Hourly Quota High Email Service Send Throttled Email Service Bounce Rate High

Cluster Autoscaler Unregistered Nodes

Runbook for when the cluster-autoscaler has nodes that were requested but never registered with the Kubernetes API. This is the ClusterAutoscalerUnregisteredNodes alert — a sibling of ClusterAutoscalerUnhealthy (07-kubernetes/12-cluster-autoscaler-unhealthy.md).

Meaning

The cluster-autoscaler runs in kube-system and scales node groups up/down based on Pending pods. The cluster-autoscaler-status-exporter (in the shared namespace) exposes cluster_autoscaler_status_long_unregistered_total — a counter of nodes the autoscaler tried to add that never registered with the Kubernetes API. When this counter is >= 1 for 5m, the alert fires.

This is almost always a cloud-provider authentication issue: the autoscaler successfully requests nodes from the cloud provider (Scaleway/Azure), but the nodes fail to join the cluster — the kubelet cannot authenticate to the API server, the bootstrap script fails, or the node never reaches Ready.

Scoped to k8s_cluster_name=~"prd-.*|stg-.*|dev-.*" — all par (Scaleway/kommodity) and AKS GPU clusters that run cluster-autoscaler, across production, staging, and development.

Impact

  • The autoscaler is requesting capacity that never arrives — Pending pods stay Pending, and the cluster appears to be scaling but isn't.
  • The autoscaler may be stuck retrying the unregistered nodes, not scaling up new ones.

Diagnosis

See 07-kubernetes/12-cluster-autoscaler-unhealthy.md § Diagnosis — specifically steps 1, 2, and 4 (check the autoscaler logs for failed to register node, check for NotReady nodes, and check the MachineDeployment/Machine status).

Mitigation

See 07-kubernetes/12-cluster-autoscaler-unhealthy.md § Mitigation — specifically the Kommodity-specific fix (delete the *-kubeconfig secret in the downstream cluster, delete the signing-key secret in kommodity, wait for automatic recreate, validate the kubeconfig secret is recreated, restart the autoscaler). This is the standard recovery for Scaleway sovereign clusters where the signing key has expired.

For AKS GPU clusters, check the Azure VMSS activity log for provisioning failures.

Verify

  • max by (k8s_cluster_name) (cluster_autoscaler_status_long_unregistered_total{k8s_cluster_name="<cluster>"}) returns to 0.
  • kubectl get nodes shows previously-unregistered nodes as Ready (or removed if garbage-collected).
  • The autoscaler logs show ScaleUp / ScaleDown activity resuming.

Notes

  • Sibling of 07-kubernetes/12-cluster-autoscaler-unhealthy.md — same autoscaler deployment, same diagnosis, same kommodity recovery procedure.
  • All environments — production, staging, and development clusters are all covered (no environment= filter).
  • Cloud-provider auth — the most common cause is an expired or stale signing key (kommodity) or VMSS provisioning failure (AKS).
  • Related — 08-capacity/05-nodegroup-utilization-high.md for nodegroup capacity ceiling.