CNPG Replication Unhealthy

Runbook for a CloudNativePG (CNPG) cluster whose PostgreSQL replication is unhealthy, triggered by the CNPGReplicationUnhealthy Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). The matching Mimir-ruler alert (PGReplicationUnhealthy, group postgres.replication in the synapse-server PrometheusRule) also fires and pages oncall.

Meaning

CNPG runs PostgreSQL in a primary/replica topology. The primary streams write-ahead logs (WAL) to each replica over a PostgreSQL replication connection; the replica replays them to stay current. Replication lag is how far a replica is behind the primary, measured in bytes of WAL the primary has generated that the replica has not yet replayed.

cnpg_pg_stat_replication_lag_bytes is the LSN-based lag, scraped from the primary's pg_stat_replication view: pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) per connected replica. It is 0 for a caught-up replica (including an idle one — no new WAL to replay), and > 0 when a replica is falling behind. The metric is only present on the primary (replicas have no downstream replicas, so pg_stat_replication is empty there); one series per connected replica, labelled application_name (the CNPG pod name). It is scraped by the cnpg-cluster PodMonitor and remote-written to synapse-server Mimir.

The alert fires when the max lag across replicas exceeds 100 MiB (104857600 bytes) for 5m. max by (job, k8s_cluster_name, ...) collapses to the worst replica, so a single lagging replica triggers the alert. 100 MiB is a generous threshold — a healthy replica under active write load lags by only a few MB momentarily; 100 MiB+ sustained means the replica cannot keep up.

# via the Mimir datasource proxy (or Grafana Explore):
count(cnpg_pg_stat_replication_lag_bytes)   # > 0 once the metric flows

Why LSN-based, not WAL-receiver-count or time-based

This alert previously used sum(cnpg_pg_replication_is_wal_receiver_up) < 2, which is structurally broken for two cases:

• 2-instance clusters (1 primary + 1 replica, e.g. dev-weu corti-signal / revlab): the healthy WAL-receiver sum is 1 (only the replica has a receiver; the primary does not). The < 2 threshold fired constantly — both dev-weu clusters were perpetually alerting despite healthy replication.
• Idle clusters: the time-based cnpg_pg_replication_lag (now() - pg_last_xact_replay_timestamp()) grows without bound on an idle replica whose last replay timestamp is stale, so any time-based threshold flaps on idle dev clusters.

The LSN-based metric is 0 whenever a replica is caught up, regardless of instance count or idle time, so it does not flap on idle or 2-instance clusters. It only fires when a replica is genuinely behind by real bytes of WAL.

Impact

• No failover capability. A replica that is not receiving WALs is stale and cannot be promoted. If the primary then fails, there is no healthy replica to take over → the cluster goes down with potential data loss (WALs not yet replicated).
• Read serving degrades. Tenants routing read traffic to replicas (if configured) hit a stale or down replica.
• Silent. The primary keeps serving writes; applications see no error. The only signal is the metric. The 2026-07 prd-par01 recovery had recovered clusters whose replicas had not rejoined (PVCs recreated, replica pods stuck) — the primary was healthy but the cluster had no failover target until replication was repaired.
• If left unfixed, WAL retention on the primary grows (it keeps WALs until all replicas acknowledge them). A replica down for long enough can exhaust the primary's max_slot_wal_keep_size and cause the primary's PVC to fill (see 04-storage/04-pvc-near-full.md).

Diagnosis

The alert labels carry k8s_cluster_name (Kubernetes cluster), namespace, and job (the CNPG cluster name). From the alert in Grafana, note all three.

1. Check the CNPG cluster status

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=<namespace>                             # usually "postgres"
CLUSTER=<job-label>                        # CNPG cluster name, e.g. cnpg-agent-api-cluster

kubectl -n $NS get cluster $CLUSTER -o wide

A healthy cluster shows Ready: true, Current Primary: <pod>, and the same number of Ready instance pods as instances (typically 3). A replica that has not rejoined shows Ready: false or is missing.

2. Check replica pod health

kubectl -n $NS get pods -l cnpg.io/cluster=$CLUSTER -o wide
kubectl -n $NS get pods -l cnpg.io/cluster=$CLUSTER,role=replica -o wide

For any replica pod that is not Running or is crash-looping:

kubectl -n $NS describe pod <replica-pod>
kubectl -n $NS logs <replica-pod> -c postgres --tail=100

Common replica failures:

• PVC missing / Pending — the replica's PVC was destroyed (rook-ceph crash) and cannot reattach. The pod stays Pending or ContainerCreating. This is the 2026-07 recovery scenario.
• CrashLoopBackOff — the replica's data directory is corrupt or from an incompatible WAL position. The postgres process exits on startup.
• Stuck ReplicationSlots / not joining — the replica started but never reached in recovery streaming state.

3. Check the replication connection from the primary

Confirm whether the primary sees the replica's replication connection:

PRIMARY=$(kubectl -n $NS get cluster $CLUSTER -o jsonpath='{.status.currentPrimary}')
kubectl -n $NS exec $PRIMARY -c postgres -- psql -c \
  "SELECT application_name, state, sync_state, sent_lsn, replay_lsn
   FROM pg_stat_replication;"

• If pg_stat_replication lists the replica with state=streaming, replication is actually working and the metric may be stale (scrape gap) — recheck the alert after a few scrape intervals.
• If pg_stat_replication is empty, no replica is connected — the replica pod is down or cannot reach the primary on 5432.
• If replay_lsn is far behind sent_lsn, the replica is connected but lagging (applying WALs slowly, or stuck).

4. Check the network path (primary ↔ replica)

CNPG replicas connect to the primary via the cluster's <cluster>-rw and <cluster>-r services. A NetworkPolicy or service issue can break the replication connection:

kubectl -n $NS get svc -l cnpg.io/cluster=$CLUSTER
kubectl -n $NS get networkpolicy -l cnpg.io/cluster=$CLUSTER

The cnpg-cluster chart renders an ingress NetworkPolicy on the cluster pods that allows postgres (5432) from labelled namespaces. If a Cilium/CN policy changed, the replica's connection to the primary may be dropped. Check the replica pod's logs for connection refused or timeout to <cluster>-rw.

5. Gap: a completely disconnected replica

The LSN-based metric is only present for replicas that appear in pg_stat_replication (i.e. are connected to the primary). A replica that is completely disconnected (WAL receiver down, pod crashed, network broken) has no row in pg_stat_replication, so cnpg_pg_stat_replication_lag_bytes is absent for it — the LSN alert will not fire on a disconnected replica.

This is by design: the LSN metric catches lag (connected but behind); a disconnection is a different failure class. To detect a disconnected replica, check directly:

# On the primary: how many replicas are connected?
PRIMARY=$(kubectl -n $NS get cluster $CLUSTER -o jsonpath='{.status.currentPrimary}')
kubectl -n $NS exec $PRIMARY -c postgres -- psql -c \
  "SELECT count(*) FROM pg_stat_replication;"

# Expected: one row per replica. If 0 but the cluster has replicas, a replica is disconnected —
# go to step 2 (pod health) and step 4 (network).

A disconnected replica shows up as a Ready: false pod (step 2) and an empty pg_stat_replication on the primary (step 3). The CNPG cluster status (kubectl get cluster) and the pod-level alerts (03-deployment-no-running-replicas.md for the CNPG StatefulSet) cover the disconnected case — the LSN alert is specifically for the "connected but falling behind" case.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Repair the replica

The fix depends on why the replica is down:

• PVC missing (rook-ceph / node loss) — the replica's PVC must be recreated. For an ArgoCD-managed cluster, the durable fix is a chart-values PR; in an emergency, kubectl -n $NS delete pod <replica-pod> lets the StatefulSet recreate the pod, but if the PVC is gone the new pod needs a fresh volume (which means rejoining from the primary via full resync, or recovery from backup — see 01-cnpg-backup-failure.md).
• CrashLoopBackOff (corrupt data dir) — delete the replica's PVC so CNPG rebuilds it from the primary via a base backup + WAL streaming: kubectl -n $NS delete pvc <replica-pvc> && kubectl -n $NS delete pod <replica-pod>. CNPG will rejoin the replica automatically. Only delete a replica's PVC, never the primary's.
• Stuck rejoin — kubectl -n $NS delete pod <replica-pod> forces CNPG to restart the join sequence. Watch the new pod's logs to confirm it reaches streaming state.
• NetworkPolicy changed — fix the policy via a chart-values PR (ArgoCD self-heal reverts manual kubectl patch in ~15s). Confirm the replica can reach <cluster>-rw:5432.

After the repair, watch the replica pod reach Running and pg_stat_replication on the primary show state=streaming for that replica.

Step 3 — If the cluster was just recovered (2026-07 scenario)

A cluster recovered from an object-store backup (see 01-cnpg-backup-failure.md step 3) boots a fresh primary first; replicas rejoin afterward. If replicas have not rejoined within ~10 min of the primary becoming Ready, they are likely stuck on a stale or missing PVC — apply Step 2. Do not flip the cluster back to normal mode until all replicas are Ready and replication is healthy (this alert is clear).

Verify

• CNPGReplicationUnhealthy clears in Grafana once max(cnpg_pg_stat_replication_lag_bytes) <= 104857600 (100 MiB) for the cluster, sustained for the for: 5m window. Practically it clears within a few scrape intervals once the lagging replica catches up.
• The Mimir-ruler PGReplicationUnhealthy (still on the WAL-receiver-count check until the chart alert is separately updated) clears once sum(cnpg_pg_replication_is_wal_receiver_up) >= 2 for 3-instance clusters.
• kubectl -n $NS get cluster $CLUSTER shows Ready: true with all instances ready.
• On the primary, pg_stat_replication lists all replicas with state=streaming and replay_lsn close to sent_lsn:

  kubectl -n $NS exec $PRIMARY -c postgres -- psql -c \
    "SELECT application_name, state, sent_lsn, replay_lsn,
            pg_wal_lsn_diff(sent_lsn, replay_lsn) AS lag_bytes
     FROM pg_stat_replication;"
  

  lag_bytes should be small (a few MB at most under load, 0 when idle).
• For a recovered cluster, confirm replication is healthy before flipping back to normal mode in the chart values.