Runbook for a CloudNativePG (CNPG) cluster that has no successful backup, triggered by the CnpgClusterNoBackup Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). The matching Mimir-ruler alert (CnpgClusterNoBackup, group cnpg.backups in the synapse-server PrometheusRule) also fires and pages oncall.

Three related CNPG backup alerts all route here:

CnpgClusterNoBackup — last_available_backup_timestamp == 0 for 30h+. The cluster has never completed a backup. Worst state: no recoverable point exists.
CnpgClusterBackupStale — last backup > 48h old (timestamp > 0 but stale). Backups used to work and stopped.
CnpgClusterNoWalRecoveryPoint — base backup exists (last_available_backup_timestamp > 0) but no first_recoverability_point. The silent failure: WAL archiving is broken so the base backup cannot actually be recovered. This is the root cause of the 2026-07 prd-par01 incident (IPv6 egress to the S3 endpoint was blocked, so WALs 015/016/017 never archived).

Meaning

CNPG runs PostgreSQL on Kubernetes. A backup is a barman-cloud-backup base snapshot of the data directory, streamed to an object store (Azure Blob for AKS clusters, Scaleway S3 for prd-par01). WAL archiving (barman-cloud-wal-archive) ships write-ahead logs so the base backup can be recovered to a point in time. The two are inseparable: a base backup without WALs is a crash-consistent snapshot that can only restore to the exact moment of backup, and recovery from it will fail if WALs covering the checkpoint onward are missing.

CNPG exposes backup health via the cnpg_collector_* metrics (scraped from the postgres_exporter sidecar on port 9187 via the cnpg-cluster PodMonitor, remote-written to synapse-server Mimir):

cnpg_collector_last_available_backup_timestamp — Unix epoch of the last successful base backup. 0 means none. Only updated on the primary; replicas report 0.
cnpg_collector_first_recoverability_point — epoch of the earliest point the cluster can be PITR-recovered to. Requires at least one base backup and WALs archived past that point. Absent (not 0) when archiving never produced a point.

A backup is scheduled by a ScheduledBackup CR (daily by default). Each run creates a Backup CR that the CNPG operator reconciles: it boots a backup pod, runs barman-cloud-backup, and on success updates the timestamp metrics. WALs are archived continuously by the primary's sidecar.

Impact

No recoverable backup = data loss on the next failure. If the cluster's PVC is destroyed (the 2026-07 rook-ceph crash deleted prd-par01 PVCs) with no backup, the data is gone. The cluster must be recreated with a fresh initdb (empty database) and rebuilt upstream.
A stale backup (>48h) means RPO has silently grown past the expected 24h. A restore loses up to 48h+ of writes.
No WAL recovery point is the most dangerous: the dashboard shows a green backup timestamp, so oncall assumes the cluster is protected — but recovery will fail. Every prd-par01 base backup looked healthy right up until we tried to recover and discovered WALs 015–017 were never archived.
Stateful workloads (Keycloak, agent-api, copilot-api-v2, voice-vault) lose availability when their CNPG cluster is lost. Customer data (interactions, transcripts) can be permanently lost.

Diagnosis

The alert labels carry k8s_cluster_name (Kubernetes cluster), namespace, and job (the CNPG cluster name — job is the postgres_exporter scrape job, which equals the CNPG Cluster resource name). From the alert in Grafana, note all three.

1. Identify the CNPG cluster and its backup config

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01-admin.yaml
NS=<namespace>                             # usually "postgres"
CLUSTER=<job-label>                        # CNPG cluster name, e.g. cnpg-agent-api-cluster

kubectl -n $NS get cluster $CLUSTER -o wide
kubectl -n $NS get backup -l cnpg.io/cluster=$CLUSTER
kubectl -n $NS get scheduledbackup -l cnpg.io/cluster=$CLUSTER

Check the cluster's backup destination in its spec — spec.backup.barmanObjectStore shows the endpointURL, destinationPath, and provider (azure/s3). For prd-par01 this is Scaleway S3 (s3://prd-par01-db-backups/<release>/); for AKS clusters, Azure Blob (<environment>cnpgbackup.blob.core.windows.net).

2. Check the Backup CR status

A healthy completed Backup shows phase: completed. A stuck/failed backup shows phase: failed or is stuck in running:

kubectl -n $NS get backup -l cnpg.io/cluster=$CLUSTER \
  -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,START:.status.startedAt,END:.status.completedAt'

For a failed backup, inspect its status and the CNPG operator logs:

kubectl -n $NS describe backup <backup-name>   # status.errors, status.error
kubectl -n cnpg-system logs deploy/cnpg-controller-manager --tail=100 | grep $CLUSTER

Common failures:

barman-cloud-backup: unable to connect to endpoint — egress to the object store is blocked (see step 4).
could not upload WAL file / barman-cloud-wal-archive: connection refused — WAL archiving cannot reach the object store. The base backup may succeed but WALs pile up locally.
Expected empty archive, found WAL file — the barman-cloud-check-wal-archive pre-flight found WALs at the destination. This fires when a new cluster's backup destination overlaps an existing backup's destination (the recovery-mode conflict, see Mitigation).

3. Check WAL archive health (the silent failure)

WAL archiving is the part that breaks silently. Verify first_recoverability_point is advancing via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

curl -s -H "X-Scope-OrgID: corti" \
  "http://localhost:9090/prometheus/api/v1/query?query=cnpg_collector_first_recoverability_point{job=\"$CLUSTER\"}"

If last_available_backup_timestamp > 0 but first_recoverability_point is absent, WAL archiving is broken — the base backup cannot be recovered. This is the CnpgClusterNoWalRecoveryPoint condition.

Check the primary pod's WAL archive sidecar logs directly:

PRIMARY=$(kubectl -n $NS get cluster $CLUSTER -o jsonpath='{.status.currentPrimary}')
kubectl -n $NS logs $PRIMARY -c postgres --tail=200 | grep -i 'wal.*archive\|barman'

4. Check egress (the most common root cause)

The 2026-07 incident root cause: the cnpg-cluster chart's egress NetworkPolicy did not allow IPv6 (::/0) to the Scaleway S3 endpoint, and the pod resolved the endpoint to an IPv6 address. WALs silently failed to archive for days.

Check the egress NetworkPolicy on the cluster's pods:

kubectl -n $NS get networkpolicy -l cnpg.io/cluster=$CLUSTER
kubectl -n $NS get networkpolicy <np-name> -o yaml | grep -A20 'egress:'

For Scaleway/S3 clusters, confirm the egress rule permits the object store endpoint on 443 for both IPv4 and IPv6. The chart fix (#2868) added ::/0 egress. If the cluster predates that chart version, the missing IPv6 rule is the cause.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the backup pipeline (if the cluster is healthy, just not backing up)

If the cluster is running fine and only backups are broken, fix the root cause from Diagnosis:

Egress blocked — patch the cnpg-cluster chart values to add the missing egress rule (IPv6 ::/0 for Scaleway S3, or the Azure Blob endpoint for AKS) and roll a chart PR. This is the durable fix; do not kubectl patch the NetworkPolicy directly (ArgoCD self-heal reverts it in ~15s).
Stuck Backup CR — a Backup stuck in running blocks the ScheduledBackup from creating new ones. Delete the stuck CR: kubectl -n $NS delete backup <stuck-name>. The next scheduled run will create a fresh one.
WAL archiving broken but base backup works — same egress check; WALs use the same endpoint. Once egress is fixed, the primary's sidecar will catch up and first_recoverability_point will appear.

After the fix, watch for a new Backup CR to reach phase: completed and the Mimir timestamp metric to advance past 0 (or past the stale value).

Step 3 — Recover from the object-store backup (if the cluster is lost)

If the cluster's PVCs were destroyed (rook-ceph crash, node loss, accidental delete) and a recoverable backup exists, rebuild the cluster from the object store via recovery mode. This is a GitOps change to the cnpg-cluster chart values, merged via PR:

cluster:
  mode: recovery
  backups:
    enabled: false          # no spec.backup block → skips the "Expected empty archive"
                            # pre-flight (new cluster's backup dest == recovery source)
  recovery:
    method: object_store
    clusterName: <cnpg-cluster-resource-name>   # serverName in the bucket layout
    provider: s3            # or azure
    endpointURL: <object-store-endpoint>
    destinationPath: <backup-destination-path>
    # Reuse the existing backup-creds secret (ExternalSecret still renders via chart ≥3.14.1,
    # so ESO keeps the S3/Azure creds synced for recovery to read the object store):
    secret:
      create: false
      name: <release>-backup-creds

Key points (learned in the 2026-07 recovery):

backups.enabled: false is required when the recovery source is the same destination the cluster would back up to — otherwise barman-cloud-check-wal-archive finds WALs at the destination and fails the bootstrap. The backup-creds ExternalSecret still renders (chart ≥3.14.1, PR #2867) so ESO keeps the creds synced.
No PITR by default — recover to the latest available backup (recovery.recoveryTarget omitted). Only add a PITR target if you know the exact timestamp to stop at.
The ScheduledBackup does not render in recovery mode, so no immediate backup fires to overwrite the source.
ArggoCD syncs the Cluster CR; CNPG boots a new primary, runs barman-cloud-backup-restore + replays WALs to the recovery point, then joins replicas.

Once the primary is Ready and serving, flip back to normal mode in a follow-up PR: cluster.mode: "", cluster.backups.enabled: true, remove cluster.recovery. This re-enables scheduled backups so the freshly-recovered cluster is protected going forward.

Step 4 — Fresh initdb (last resort, when no recoverable backup exists)

If recovery fails (WALs missing, backup corrupt) or there was never a backup, the cluster must be recreated empty. This destroys all data in that database — only do this when recovery is confirmed impossible:

cluster:
  mode: ""          # normal initdb bootstrap
  backups:
    enabled: true
  # Remove the entire cluster.recovery block.

ArgoCD sync recreates the Cluster with an empty data directory. The application owners must rebuild data from upstream sources (Kafka replay, re-sync from another system, etc.). This is what shared and medical-coding-orchestrator required on prd-par01 (WALs 015–017 never archived, base backups unrecoverable).

Verify

CnpgClusterNoBackup clears in Grafana and the Mimir-ruler alert clears once a Backup reaches phase: completed and cnpg_collector_last_available_backup_timestamp advances past 0 (within the for: 30h window — practically within one schedule cycle after the fix).
CnpgClusterBackupStale clears once the last backup is < 48h old.
CnpgClusterNoWalRecoveryPoint clears once cnpg_collector_first_recoverability_point appears (is no longer absent) — confirm via the Mimir query in Diagnosis step 3.
The Backup CR reaches phase: completed with no errors: kubectl -n $NS get backup -l cnpg.io/cluster=$CLUSTER.
For a recovery (step 3), the recovered primary is Ready: kubectl -n $NS get cluster $CLUSTER shows Current Primary: <pod> and Ready: true. Confirm the application connects and data is present before flipping back to normal mode.
After flipping back to normal mode, the first scheduled backup completes and last_available_backup_timestamp updates to a recent epoch.