CNPG Runbook

General runbook for CloudNativePG (CNPG) operations and incident recovery.

Contents

• Prerequisites
• Cluster status overview
• Procedure: Recover a diverged replica
• Procedure: Recover from stuck backups crashlooping the operator
• Background: June 2026 incident
• Background: gophish backup wedge (June 2026, dev-neu)

Prerequisites

Tools

Access

Ensure you have kubeconfig access to the target cluster (prod-us, prod-weu, etc.). Elevate production access via the Azure PIM portal if needed.

Cluster status overview

Check the state of all CNPG clusters across contexts, including which instance is primary and which are replicas:

# From the deployments repo root
./scripts/cnpg/cnpg-status.sh [context1 context2 ...]

# Examples
./scripts/cnpg/cnpg-status.sh prod-us prod-weu
./scripts/cnpg/cnpg-status.sh dev-weu dev-par01

Degraded replicas show timeline=? — they have stopped reporting state to the CNPG operator.

You can also check a specific cluster:

kubectl --context <ctx> get cluster <cluster-name> -n postgres \
  -o custom-columns='PRIMARY:.status.currentPrimary,READY:.status.readyInstances,TOTAL:.status.instances,PHASE:.status.phase'

Procedure: Recover a diverged replica

Symptoms

Datadog dashboard

Open the CNPG health, status, load dashboard and filter by cluster and service. A diverged replica produces three correlated signals:

WAL receivers up drops by 1 (e.g. 2→1 on a 3-node cluster). The replica has stopped streaming from the primary. This is the earliest indicator — it appears as soon as the replica falls behind or crashes, often days before anyone notices.

PG WAL Size grows continuously. PostgreSQL cannot reclaim WAL segments until all replicas have consumed them. With one replica offline the WAL directory expands unbounded on the primary.

PVC Size (%) on the primary pod climbs above 100 % as the WAL accumulation fills the volume. In the June 2026 incident cnpg-voice-vault-cluster-2 (primary) reached 150 %+ while the crashed cluster-3 replica showed a flat ~2 % (frozen, not consuming any WAL).

WAL Archive Status shows a steadily growing backlog of unarchived WAL files.

If all four signals appear together, a replica is diverged and the cluster is at risk of the primary PVC filling completely.

Kubernetes

• A replica pod is in CrashLoopBackOff in the postgres namespace.
• Pod logs contain one of:
  • FATAL: requested timeline N is not a child of this server's history
  • PANIC: could not locate a valid checkpoint record
• pg_controldata in the logs shows Time of latest checkpoint weeks in the past.

Root cause

The replica's data directory has diverged from the current primary timeline. This happens when:

1. A failover occurred (promoting a new primary, creating a new timeline).
2. The replica was a streaming standby at the time and advanced past the fork point on the old timeline.
3. The replica was then restarted (e.g. by a rolling upgrade), forcing it to replay WAL from the archive — at which point PostgreSQL detects the divergence and refuses to start.

The replica was surviving before the restart by streaming directly from the primary, bypassing the archive. The restart exposed the pre-existing divergence.

Prerequisites

Before proceeding, verify:

1. The affected pod is NOT the primary. Use the status script or:

   kubectl --context <ctx> get cluster <cluster-name> -n postgres \
     -o jsonpath='{.status.currentPrimary}'
   

   Only proceed if the pod name differs from currentPrimary.

2. Backups are healthy. CNPG will re-bootstrap the replica from the latest backup:

   kubectl --context <ctx> get cluster <cluster-name> -n postgres \
     -o jsonpath='{.status.conditions[?(@.type=="LastBackupSucceeded")].status}'
   # Should return: True
   

Recovery

Use kubectl cnpg destroy to delete the pod and its PVC. The CNPG operator detects the missing volume and re-bootstraps the instance from the latest backup + WAL replay.

kubectl cnpg destroy <cluster-name> <pod-name> --namespace postgres --context <ctx>

Example (incident June 2026, prod-us and prod-weu):

# prod-us
kubectl cnpg destroy cnpg-copilot-api-cluster cnpg-copilot-api-cluster-3 --namespace postgres --context prod-us
kubectl cnpg destroy cnpg-voice-vault-cluster cnpg-voice-vault-cluster-3 --namespace postgres --context prod-us

# prod-weu
kubectl cnpg destroy cnpg-assistant-cluster    cnpg-assistant-cluster-3    --namespace postgres --context prod-weu
kubectl cnpg destroy cnpg-copilot-api-cluster  cnpg-copilot-api-cluster-3  --namespace postgres --context prod-weu
kubectl cnpg destroy cnpg-voice-vault-cluster  cnpg-voice-vault-cluster-3  --namespace postgres --context prod-weu

Verification

Monitor the pod coming back up and rejoining the cluster:

kubectl --context <ctx> get pod <pod-name> -n postgres -w

Once Running 1/1, confirm it is a replica on the correct timeline:

kubectl --context <ctx> get cluster <cluster-name> -n postgres \
  -o jsonpath='{.status.instancesReportedState}'

The recovered instance should appear with isPrimary: false and a matching timeLineID.

Timeline

Re-bootstrap time depends on backup size and WAL replay volume. Expect 5–30 minutes for typical database sizes.

Procedure: Recover from stuck backups crashlooping the operator

Symptoms

Kubernetes

• A cluster is stuck below full replica count (e.g. 1/3) with status Unable to create required cluster objects — the operator cannot create the cluster's RoleBindings/objects.
• The CNPG operator pod in cnpg-system has a high, climbing restart count and flaps Running ↔ Error (exit 1); its readiness probe on :9443 intermittently reports connection refused.
• Operator logs flood with many repeats, for one cluster, of: A backup is already in progress or waiting to be started, retrying.
• Replica pods crashloop with startup-probe failures showing apiServerReachable:false / context deadline exceeded to 10.0.0.1:443 (they were never fully provisioned because the operator is unstable).
• Many Backup objects for the cluster sit with an empty phase (never started), and the cluster reports ContinuousArchiving=False (ContinuousArchivingFailing).

Confirm

# Backups for the cluster (look for a large count stuck with empty PHASE)
kubectl --context <ctx> get backups.postgresql.cnpg.io -n postgres \
  -l cnpg.io/cluster=<cluster-name>

# Operator restart count
kubectl --context <ctx> get pods -n cnpg-system

# Live cluster conditions (archiving / last backup)
kubectl --context <ctx> get cluster <cluster-name> -n postgres \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

Root cause

CNPG runs one backup per cluster at a time. When backups start failing the scheduled daily Backup objects pile up unstarted. The single shared operator then churns the entire backlog on every reconcile, destabilises (restart loop), and can no longer reconcile the cluster — so it cannot create RoleBindings or provision replicas. One sick cluster degrades the operator for all clusters.

The underlying backup failure is often a WAL-archive wedge: barman-cloud-check-wal-archive ... ERROR: ... Expected empty archive. CNPG's pre-archive check expects an empty object-store prefix, but a cluster that has been recovered/failed-over many times (high timeline, e.g. 0000000A) finds old WAL already at its prefix and refuses to archive — failing every backup.

Recovery

1. Stop new stuck backups — suspend the schedule:

   kubectl --context <ctx> patch scheduledbackup <cluster-name>-daily-backup \
     -n postgres --type merge -p '{"spec":{"suspend":true}}'
   

2. Clear the backlog — delete the cluster's Backup objects. This removes only the Kubernetes tracking objects; base backups and WAL already in the object store are untouched:

   kubectl --context <ctx> delete backups.postgresql.cnpg.io -n postgres \
     -l cnpg.io/cluster=<cluster-name>
   

   The operator should stop crashlooping within a minute or two.
3. Let the operator reconcile. Once stable it recreates the cluster objects and re-provisions replicas; WAL archiving often recovers on its own once a healthy replica rejoins. If the operator does not settle, restart it cleanly:

   kubectl --context <ctx> delete pod -n cnpg-system \
     -l app.kubernetes.io/name=cloudnative-pg
   

4. If WAL archiving stays Failing with "Expected empty archive", tell CNPG to skip the empty-archive precheck so archiving resumes against the non-empty prefix:

   kubectl --context <ctx> annotate cluster <cluster-name> -n postgres \
     cnpg.io/skipEmptyWalArchiveCheck=enabled
   

5. Re-enable the schedule once the cluster is 3/3 healthy:

   kubectl --context <ctx> patch scheduledbackup <cluster-name>-daily-backup \
     -n postgres --type merge -p '{"spec":{"suspend":false}}'
   

Verification

kubectl --context <ctx> get cluster <cluster-name> -n postgres   # READY 3/3, healthy
kubectl --context <ctx> get cluster <cluster-name> -n postgres \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'  # ContinuousArchiving=True

Background: June 2026 incident

Trigger: Chart cnpg-cluster was bumped from 3.6.0 → 3.7.0 in prod-us and prod-weu. Version 3.6.2 had changed default pod memory limits (500Mi → 2000Mi), which prod-* had never received. The resource change caused the CNPG operator to roll all instances across all clusters simultaneously.

Why dev was unaffected: dev-* environments had already rolled through 3.6.2 weeks earlier, so 3.7.0 brought no pod spec change for them.

Pre-existing condition: Several replica pods had been silently diverged for weeks — data directories frozen since May 18–21, while the cluster had undergone multiple failovers. The replicas survived by streaming directly from the primary. The rolling restart forced WAL archive replay and exposed the divergence.

Related: The same rollout was preceded by a cert-manager v1.18 breakage (May 29) where the default privateKey.rotationPolicy changed from Never to Always, rotating CA keys and orphaning leaf certs. Fixed by PR #2542.

Background: gophish backup wedge (June 2026, dev-neu)

Symptom that surfaced it: Grafana (synapse-ui) stuck in CreateContainerConfigError — its synapse-ui-keycloak-client secret was missing because platform-service never reached its client-creation step (it hung on the platform tenant's gophish Postgres).

Causal chain (oldest → newest):

1. ~Mar 26: gophish replicas went unhealthy → daily backups began failing.
2. WAL archiving wedged — barman-cloud-check-wal-archive: Expected empty archive (cluster on timeline 0000000A after many failovers; object-store prefix …/gophish/ not empty). ContinuousArchiving=False.
3. Backups piled up — ~47 daily Backup objects stuck with empty phase (one-backup-at-a-time + no effective cleanup).
4. Operator crashlooped — churned the 47-backup backlog every reconcile, exit 1, 51 restarts. While unstable it could not create cluster objects (RoleBindings) or provision replicas -10/-11. Only the primary -9 survived on cached state.
5. Downstream: platform-service boot hung connecting to gophish → CreateClients never ran → Grafana's client secret never created.

Fix: suspended the ScheduledBackup, deleted the 47 stuck Backup objects → operator stopped crashlooping → reconciled gophish → replicas rejoined and WAL archiving self-recovered → 3/3 healthy → platform-service completed boot → secret created → Grafana recovered. ScheduledBackup re-enabled at 3/3. See Procedure: Recover from stuck backups crashlooping the operator above.

The Grafana login failure seen during the same window (Failed to get token from provider) was a separate NetworkPolicy egress bug, not part of this incident — fixed in charts PR #2575.

One-off, or can it recur?

Recurring risk — not a one-off. The trigger and amplifiers are generic:

• Trigger is generic. "Expected empty archive" can hit any cluster recreated/restored against a non-empty object-store prefix. gophish tripped first only because it was the most churned (timeline 10, instances -9/-10/-11 vs -1..-4 elsewhere) — the same object-store reuse pattern applies cluster-wide.
• Amplifier is structural. Failed backups accumulate Backup objects unbounded (no retention/GC), and the single shared operator chokes on the backlog and crashloops — so one sick cluster can degrade reconciliation for all clusters.
• Blast radius is wide. platform-service's sequential boot (tenant DBs → clients) means any unreachable tenant DB blocks all Keycloak client/secret creation.

Suggested hardening:

1. Backup retention/GC on ScheduledBackups so failures can't pile to dozens.
2. Alerts on ContinuousArchiving=False, on Backup objects stuck/empty-phase beyond N hours, and on CNPG operator restarts (the 51 restarts went unnoticed); consider raising the operator memory limit.
3. Decouple platform-service so client/secret provisioning does not hang behind tenant DB setup (run independently, or time out the tenant DB connect instead of blocking). Highest-value fix — it is what turned a gophish DB problem into a Grafana outage.