Orca ORCA v0.174.2
Welcome Developer Guide Infrastructure Platform Engineering Runbooks Archive
Documentation navigation
On Premise Customers
SFD Hamilton County CommonSpirit GMR Region Nord On-Prem Synapse API-Key Deploy
Dns
DNS Flush
Kafka
Kafka Consumer Lag in Voice-Vault Kafka Consumer Group Lag High (> 800) HPA Max Replicas Exceed Kafka Topic Partitions
Storage
Ceph OSD Full / BlueFS Allocation Wedge Query Ceph RGW (S3) Bucket Usage Ceph Mon Quorum Loss / Node-Replacement Recovery PVC Near Full Warning (>= 80%) PVC Near Full Critical (>= 90%) PVC Disk Usage Forecast > 95% Ceph Cluster Storage Utilization High (> 90%) Ceph Cluster Error Ceph Cluster Warning
Observability
Alloy Operator Wedge Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild Grafana Dashboard Provisioning Error Mimir Ingester Reaching Series Limit Mimir Ingest Storage Wedged Mimir Ingester PVC Near Full Loki PVC Near Full Mimir Self-Monitoring Scrape Stalled Synapse Cluster Metrics Silent Mimir Discarding Samples (Series Limit) Loki Discarding Log Entries Trivy Metrics Stale Envoy Proxy Heap Near Ceiling Envoy Proxy Heap Shedding Envoy Proxy Overload Rejecting Requests Traefik Elevated Error Spike (4xx/5xx) Envoy Elevated Error Spike (4xx/5xx) Scrape Targets Failing Traefik 402 Payment Failure (Multiple Distinct Users) Azure Audit Ingestion Silent
Databases
CNPG Cluster Has No Backup CNPG Replication Unhealthy CNPG Cluster Backup Stale (> 48h) CNPG Cluster Has No WAL Recovery Point
Kubernetes
Pod ImagePull / ImagePullBackOff Pod CrashLoopBackOff Deployment Has No Running Replicas Deployment Rollout Stuck DaemonSet Has No Running Pods StatefulSet Has No Running Replicas Kubernetes Job Failed Kubernetes CronJob Suspended Certificate Expiring Soon Node Memory/Disk Pressure (Kubelet Evicting Pods) Container OOMKilled Cluster Autoscaler Unhealthy Trivy Critical Vulnerability One Week Trivy Critical Vulnerability Four Weeks Trivy High Vulnerability One Week Trivy High Vulnerability Six Weeks Certificate Renewal Stuck Certificate Never Issued Trivy Critical Vulnerability Immediate API v2 Debug Endpoint Called in Production Cluster Autoscaler Unregistered Nodes Platform API Server Errors Platform API Client Errors Pod Pending Over 30 Minutes Pod Pending (GPU) Over 60 Minutes Cilium Policy Drops Burst
Capacity
CPU Throttling > 20% HPA Utilization Critically High GPUs Underutilized For 48h Container Memory Leak (Rising Near Limit) Nodegroup Utilization Critically High Nodegroup Utilization High Warning
Identity
Keycloak Client Misconfigured
Synthetic Monitoring
Synthetic Feature Failing Synthetic Monitor Silent Speech to Text – Synthetic Check Failing Corti Models – Synthetic Check Failing Text Generation – Synthetic Check Failing
Platform
Orca GitHub Rate-Limit Budget Low Email Service Minute Quota High Kyverno Admission API Server Unreachable Email Service Hourly Quota High Email Service Send Throttled Email Service Bounce Rate High

CNPG Cluster Backup Stale (> 48h)

Runbook for a CNPG cluster whose last successful backup is older than 48h. This is the "backups used to work, now they don't" signal — distinct from 01-cnpg-backup-failure.md (which fires when a cluster has never backed up, timestamp == 0).

Meaning

cnpg_collector_last_available_backup_timestamp is the Unix epoch of the last successful base backup (CNPG postgres_exporter sidecar, scraped via the cnpg-cluster PodMonitor). The alert fires when (time() - last_backup) / 3600 > 48 — i.e. the last backup is more than 48h old — and the timestamp is > 0 (gated, so this alert does not duplicate CnpgClusterNoBackup which fires at == 0). for: 1h rides out a transient scrape gap.

Backups are scheduled daily (ScheduledBackup CR, default 0 0 0 * * *), so a 48h+ gap means at least two consecutive scheduled runs have failed. RPO has silently grown past the expected 24h.

The and on (job, k8s_cluster_name, environment) join keeps the alert from firing on the == 0 case (handled by CnpgClusterNoBackup); the max by (...) collapses to the primary's value (replicas report 0). The metric is already scraped into Mimir — no chart PR is needed.

Impact

  • A restore from a 48h+ stale backup loses up to 48h+ of writes — RPO has silently grown past the expected 24h.
  • If left unresolved this progresses to the CnpgClusterNoBackup failure mode (the timestamp eventually ages out / a recreated cluster resets to 0).
  • The cluster is still serving traffic — this is a backup-pipeline problem, not a service outage.

Diagnosis

The alert labels carry k8s_cluster_name, environment, namespace, job (the CNPG cluster name). From the alert, note all of them.

The backup pipeline, failure modes, and remediation are identical to 01-cnpg-backup-failure.md. Use its Diagnosis section end-to-end:

  • Step 1 — identify the CNPG cluster and its backup config (kubectl get cluster, scheduledbackup, backup).
  • Step 2 — check the Backup CR status for phase: failed or stuck running; inspect the CNPG operator logs.
  • Step 3 — check WAL archive health (first_recoverability_point advancing via Mimir); if absent, CnpgClusterNoWalRecoveryPoint (03-cnpg-no-wal-recovery-point.md) is also firing.
  • Step 4 — check egress (the most common root cause — IPv6 ::/0 missing for Scaleway S3, or the Azure Blob endpoint for AKS).

Confirm the staleness directly via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

QUERY='(time() - max by (job, k8s_cluster_name, environment, namespace) (cnpg_collector_last_available_backup_timestamp{job="<job>", k8s_cluster_name="<cluster>"})) / 3600'
curl -s -G -H "X-Scope-OrgID: corti" "http://localhost:9090/prometheus/api/v1/query" --data-urlencode "query=$QUERY" | jq -r '.data.result[0].value[1]'

A value > 48 confirms the alert (hours since last backup).

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the backup pipeline

Follow 01-cnpg-backup-failure.md Mitigation step 2 — the root causes are the same (egress blocked, stuck Backup CR, WAL archiving broken). The durable fix is a chart PR (cnpg-cluster egress NetworkPolicy), not a kubectl patch (ArgoCD self-heal reverts it).

After the fix, watch for a new Backup CR to reach phase: completed and the Mimir timestamp metric to advance.

Verify

  • The alert clears once time() - last_available_backup_timestamp is under 48h sustained for the for: 1h window.
  • kubectl -n <namespace> get backup -l cnpg.io/cluster=<cluster> shows a recent phase: completed Backup.
  • Re-run the Mimir query from Diagnosis to confirm the age is under 48h.

Notes

  • Relationship to 01-cnpg-backup-failure.md — that runbook fires when timestamp == 0 (never backed up); this one fires when timestamp > 0 but stale (> 48h). The backup pipeline, diagnosis, and mitigation are shared.
  • Production-only — matches the chart alert. Dev/staging clusters are routinely recreated without backups.
  • Metric already collected — cnpg_collector_last_available_backup_timestamp is scraped from the CNPG postgres_exporter sidecar. No chart PR required.