Runbook for a CNPG cluster that has a base backup but no WAL recovery point — the silent failure where the dashboard looks green (a backup timestamp exists) but recovery will fail. This is the root cause of the 2026-07 prd-par01 incident: every base backup looked healthy right up until recovery was attempted and WALs 015/016/017 were found never to have archived.

Meaning

A CNPG backup is two inseparable parts: a base snapshot (barman-cloud-backup) and continuous WAL archiving (barman-cloud-wal-archive). Recovery from a base backup requires WALs covering the checkpoint onward — without them, recovery fails.

cnpg_collector_last_available_backup_timestamp — epoch of the last successful base backup. > 0 means a base backup exists.
cnpg_collector_first_recoverability_point — epoch of the earliest point the cluster can be PITR-recovered to. Requires a base backup and WALs archived past that point. Absent (not 0) when archiving never produced a point.

The alert fires when a base backup exists (last_available_backup_timestamp > 0) but no recovery point exists (first_recoverability_point > 0 is absent) — using unless because the metric is absent, not zero, when archiving never produced a point. for: 2h rides out the transient window right after a backup before the first WAL recovery point appears.

The max by (job, k8s_cluster_name, environment, namespace) collapses to the primary's value (CNPG only updates these metrics on the primary; replicas report 0). The metrics are already scraped into Mimir — no chart PR is needed.

Impact

The base backup cannot be recovered. Despite a green backup timestamp, a restore will fail when WAL replay hits the missing segment. This is the most dangerous CNPG failure: oncall assumes the cluster is protected when it is not.
Every restore attempt wastes time discovering the gap mid-recovery; in an incident this is time you do not have.
If the PVC is destroyed before WAL archiving is fixed, the data since the last recoverable point (possibly never) is lost.

Diagnosis

The alert labels carry k8s_cluster_name, environment, namespace, job (the CNPG cluster name). From the alert, note all of them.

The WAL-archive pipeline and remediation are identical to 01-cnpg-backup-failure.md. Use its Diagnosis section end-to-end:

Step 3 — check WAL archive health (first_recoverability_point via Mimir); check the primary pod's WAL archive sidecar logs (kubectl logs <primary> -c postgres | grep -i 'wal.*archive\|barman').
Step 4 — check egress (the most common root cause — IPv6 ::/0 missing for Scaleway S3 blocked WALs for days on prd-par01 in the 2026-07 incident).

Confirm the absence directly via Mimir (port-forward synapse-server-mimir-gateway in monitoring-system, header X-Scope-OrgID: corti):

# The base backup exists (> 0)
QUERY1='max by (job, k8s_cluster_name, environment, namespace) (cnpg_collector_last_available_backup_timestamp{job="<job>", k8s_cluster_name="<cluster>"})'
# The recovery point is ABSENT (no result) when broken
QUERY2='max by (job, k8s_cluster_name, environment, namespace) (cnpg_collector_first_recoverability_point{job="<job>", k8s_cluster_name="<cluster>"})'

If QUERY1 returns a value > 0 but QUERY2 returns no result, WAL archiving is broken — this alert's condition.

Check the primary pod's WAL archive sidecar logs directly:

export KUBECONFIG=~/.kube/<cluster>.yaml
NS=<namespace>
PRIMARY=$(kubectl -n $NS get cluster <job> -o jsonpath='{.status.currentPrimary}')
kubectl -n $NS logs $PRIMARY -c postgres --tail=200 | grep -i 'wal.*archive\|barman'

Look for could not upload WAL file / barman-cloud-wal-archive: connection refused — the WAL sidecar cannot reach the object store.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix WAL-archive egress

Follow 01-cnpg-backup-failure.md Mitigation step 2 — the root cause is almost always egress. WALs use the same object-store endpoint as the base backup; if base backups succeed but WALs do not, the endpoint resolves to an IPv6 address that the egress NetworkPolicy does not allow (the 2026-07 prd-par01 root cause). The durable fix is a chart PR adding ::/0 egress (cnpg-cluster chart PR #2868); do not kubectl patch (ArgoCD self-heal reverts it).

Once egress is fixed, the primary's WAL sidecar catches up and first_recoverability_point appears. The alert clears within the for: 2h window.

Step 3 — If the cluster is lost and recovery is needed

If the PVCs were destroyed before WAL archiving was fixed, the base backups are unrecoverable. Follow 01-cnpg-backup-failure.md Mitigation step 4 (fresh initdb, last resort) — the cluster must be recreated empty and the application owners rebuild data from upstream. Do not attempt recovery from the unrecoverable base backup.

Verify

The alert clears once cnpg_collector_first_recoverability_point appears (is no longer absent) sustained for the for: 2h window.
Confirm via the Mimir QUERY2 above — it returns a value > 0 once WALs are archiving.
The primary pod's WAL sidecar logs show successful barman-cloud-wal-archive uploads.

Notes

Relationship to 01-cnpg-backup-failure.md — that runbook fires when last_available_backup_timestamp == 0 (no base backup at all); this one fires when a base backup exists but WALs are not archiving (the silent "green but unrecoverable" state). The backup pipeline, diagnosis, and mitigation are shared.
Production-only — matches the chart alert.
Silent failure — the base backup timestamp looks healthy; only the absence of first_recoverability_point reveals it. This is why the alert uses unless (the metric is absent, not 0, when broken).
Metric already collected — cnpg_collector_first_recoverability_point and cnpg_collector_last_available_backup_timestamp are scraped from the CNPG postgres_exporter sidecar. No chart PR required.
The 2026-07 incident — IPv6 egress to the Scaleway S3 endpoint was blocked; WALs 015/016/017 never archived; every prd-par01 base backup was unrecoverable. This alert was added specifically to catch that class of failure.