Runbook for a cert-manager Certificate nearing expiry, triggered by the Certificate Expiring Soon alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts → PrometheusRule CRD synced to the Mimir ruler). The matching Mimir-ruler alert (CertificateExpiringSoon, group cert-manager in the synapse-server PrometheusRule) also fires for production and pages oncall.

Meaning

cert-manager manages TLS certificates (ACME via Let's Encrypt / ZeroSSL, or self-signed / internal CAs). Each Certificate CR has an expirationTimestamp in its status — the moment the issued cert stops being valid. certmanager_certificate_expiration_timestamp_seconds is that timestamp as a Unix epoch, scraped from the cert-manager controller's metrics endpoint (port 9402) and remote-written to synapse-server Mimir. The metric carries labels name (the Certificate CR name), exported_namespace (the Certificate CR's namespace — see below), issuer_name, issuer_kind.

Namespace label caveat: the scrape pipeline relabels the metric's own namespace label to the namespace of the cert-manager controller pod (always cert-manager, via honor_labels-style flipping). The Certificate CR's real namespace survives as exported_namespace — the alert, its annotations, and the kubectl commands below use that. Looking in -n cert-manager will return NotFound.

The one-zero sentinel: cert-manager exports expiration_timestamp_seconds = 0 (Unix epoch) when the Certificate has no issued Secret — either the first issuance is still in flight (a healthy new cert, minutes after creation) or issuance has failed permanently (a zombie Certificate for a dead/placeholder domain). Zero unrealistically survived the old unguarded comparison as a value of minus ~20,700 days, making both runbook certificate alerts fire forever on 4 such resources (2026-10-16 investigation; see 18-certificate-never-issued.md for the list). The expr therefore filters > 0 before comparing: this alert now fires only for certificates that have a real issued cert expiring soon. Everything with no secret moves to the sibling 07-kubernetes/18-certificate-never-issued.md.

The alert fires when (expiration - now) < 15 days — i.e. the cert expires within 15 days (or has already expired, in which case $value is negative). 15 days is lead time: cert-manager should renew automatically well before expiry, so this alert catching a cert at <15d means the renewal has not happened and needs human attention.

cert-manager renews a cert at ~2/3 of its lifetime by default (so a 90-day cert starts renewing at ~60 days). A cert at <15d should already have been renewed; if it is still expiring soon, the renewal failed (ACME challenge, rate limit, issuer misconfig, or the Certificate is not Ready).

This alert mirrors the existing Mimir-ruler CertificateExpiringSoon (synapse-server PrometheusRule) but fires for all environments (the chart alert filters to environment="production"). Both run side-by-side; the production one pages oncall via Mimir ruler, this one covers dev/staging/sandbox too.

Impact

If the cert expires, TLS handshakes fail → the HTTPS endpoint (Ingress, internal service, webhook) becomes unreachable. Browsers show "not secure" / cert-error; clients fail with x509: certificate has expired.
cert-manager webhooks (cainjector, the cert-manager webhook itself) use certs too — an expired webhook cert can break cert-manager itself (cascading: no new certs can be issued). This is the most dangerous case.
CNPG replication TLS (*-replication-tls, *-client-ca) — an expired client/replication cert breaks CNPG replication and client connections. See 06-databases/.
rook-ceph RGW (rook-ceph-cluster-rgw-cert) — breaks the object-store TLS endpoint.
15 days lead time means this is a warning, not an outage — there is time to fix the renewal. But an expired cert is a hard outage for the affected endpoint.

Diagnosis

The alert labels carry k8s_cluster_name, exported_namespace (the Certificate's namespace), name (the Certificate CR name), issuer_name, issuer_kind, and environment. From the alert in Grafana, note all of them.

1. Confirm the Certificate's expiry and Ready condition

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=<exported_namespace>
CERT=<name>
kubectl -n $NS get certificate $CERT -o wide
kubectl -n $NS describe certificate $CERT | sed -n '/^Status:/,$p'

The Ready condition tells you the state:

Ready=True — the cert is issued and valid. If the alert still fires, the metric and the status disagree (scrape lag) — recheck after a few minutes.
Ready=False / Issuing — cert-manager is actively trying to issue/renew. Check the Order/Challenge (step 2).
Ready=False / Failed (with a reason) — the renewal failed. The reason and message tell you why (ACME challenge failed, rate limited, issuer not found).

Check the Not After in the status to confirm the expiry the metric reports.

2. Check the Order and Challenge (ACME renewals)

For an ACME-issued cert, cert-manager creates an Order and one or more Challenge resources to prove domain control:

kubectl -n $NS get orders
kubectl -n $NS get challenges
kubectl -n $NS describe challenge <challenge-name> | sed -n '/^Events:/,$p'

Common ACME failures:

HTTP-01 self-check timing out — cert-manager cannot reach the challenge URL (http://<domain>/.well-known/acme-challenge/<token>). The Ingress / HTTPRoute is not routing the challenge path, or the challenge pod is not reachable. Check the Ingress/HTTPRoute and the solver pod. This is the most common ACME failure.
DNS-01 self-check timing out — the DNS TXT record is not appearing (external-dns lag, wrong DNS provider creds, wrong zone). Check kubectl get dnsrecords or the DNS provider.
Rate limited — Let's Encrypt returns 429 / too many failed authorizations. Wait out the window (1h–24h) or switch to a staging issuer to debug. Don't keep retrying — it makes the rate limit worse.
Wrong issuer — the Issuer/ClusterIssuer referenced by the Certificate does not exist or is misconfigured. Check kubectl get issuer -n $NS / kubectl get clusterissuer.

3. Check the Issuer / ClusterIssuer

kubectl -n $NS get issuer <issuer_name>      # if issuer_kind=Issuer
kubectl get clusterissuer <issuer_name>     # if issuer_kind=ClusterIssuer
kubectl -n $NS describe issuer <issuer_name> | sed -n '/^Status:/,$p'

Ready=True — the issuer is configured. The issue is the Order/Challenge (step 2).
Ready=False — the issuer itself is broken (bad ACME account, missing secret, wrong server URL). Fix the issuer.

4. Check the cert-manager controller is running

A cert-manager controller that is down/crash-looping cannot renew anything:

kubectl -n cert-manager get pods
kubectl -n cert-manager logs deploy/cert-manager --tail=50 | grep $CERT

If the controller is crash-looping or the cert-manager webhook cert expired (the controller's own cert), cert-manager is broken — fix the controller first. The cert-manager webhook cert expiring is the most dangerous case (it can wedge all cert issuance); if kubectl get orders,challenges returns errors, suspect this.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the renewal blocker

Follow Diagnosis steps 2–4 to the specific cause:

HTTP-01 challenge unreachable — fix the Ingress/HTTPRoute routing for /.well-known/acme-challenge/*. For an ArgoCD-managed Ingress, the fix is a chart-values PR.
DNS-01 challenge missing — fix the DNS provider (external-dns lag, creds, zone). See 02-dns/01-dns-flush.md for DNS-related debugging.
Rate limited — switch the Issuer to the ACME staging URL (https://acme-staging-v02.api.letsencrypt.org/directory) temporarily to issue a test cert and debug the challenge without burning the production rate limit, then switch back. Don't keep retrying production.
Issuer broken — fix the Issuer/ClusterIssuer config (ACME account secret, server URL, solver).
cert-manager controller down — restart/fix the cert-manager deployment (kubectl -n cert-manager rollout restart deploy/cert-manager as a hold; fix the chart values in a PR).

Step 3 — Force a renewal (once the blocker is fixed)

Once the underlying issue is fixed, force cert-manager to retry the renewal immediately rather than waiting for its next check:

kubectl -n $NS certificate <cert-name> renew   # triggers a renewal now
kubectl -n $NS get certificate <cert-name> -w    # watch Ready flip to Issuing -> True

certificate renew creates a new Order immediately. Watch the Order/Challenge succeed and the Ready condition return to True with a new Not After.

Step 4 — If the cert is already expired (emergency)

An expired cert is a hard outage for its endpoint. If the renewal cannot be fixed in time:

Switch to the ACME staging issuer to get some valid cert issued (staging certs are not browser-trusted but unblock internal TLS). Only useful for internal endpoints.
Manually issue a cert via the issuer if the ACME path is wedged (self-signed for internal; commercial CA for public).
For a webhook cert that expired and wedged cert-manager, the recovery is more involved (the controller cannot issue its own cert while its webhook cert is bad) — follow the cert-manager upstream recovery procedure (delete the expired webhook secret so cert-manager re-issues on startup).

Verify

The alert clears once (certmanager_certificate_expiration_timestamp_seconds - time()) / 24 / 3600 >= 15 for the Certificate, sustained for the for: 5m window. Practically it clears within a scrape interval after the renewal completes and the metric updates.
kubectl -n $NS get certificate $CERT shows Ready=True and a Not After > 15 days in the future.
kubectl -n $NS get orders shows the most recent Order Valid / Ready=True.
For a public endpoint, curl -vI https://<domain> shows the new cert's expire date > 15 days out.
The matching Mimir-ruler CertificateExpiringSoon (production only) also clears.

Notes

> 0 guard (2026-10-16) — added after this alert (and its 17-certificate-renewal-stuck.md sibling) fired permanently on 4 Certificate resources that had never issued a secret (expiration/renewal = 0 → "expires in -20,742 days"): only resources with a real issued cert belong in this alert now; the no-secret state moved to 07-kubernetes/18-certificate-never-issued.md.
A brand-new Certificate in its first-issuance window also reads 0 — with the guard it no longer false-pages during the minutes between Certificate creation and first secret.
exported_namespace, not namespace — the scrape pipeline overwrites namespace with the cert-manager pod's namespace (cert-manager); the Certificate's real namespace is in exported_namespace.
Related — 17-certificate-renewal-stuck.md (renewal overdue, the earlier signal for the same failures) and 18-certificate-never-issued.md (no issued secret).