Runbook for a cert-manager Certificate with no issued Secret, triggered by the Certificate Never Issued alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts → PrometheusRule CRD synced to the Mimir ruler). Low priority by design: some of these are harmless leftovers, and the endpoints involved may not even exist.

Created (2026-10-16) as the third tier of the certificate alert family. This state used to be swallowed by the two sibling alerts as garbage values — cert-manager's zero sentinel (expiration_timestamp_seconds = 0, Unix epoch) read through their unguarded < 15 days / < 0 days comparisons as "expires in -20,742 days" / "renewal overdue by 20,742 days", so both siblings fired permanently on every never-issued Certificate. The siblings now guard > 0 (live certs only) and this alert owns the no-secret state explicitly, so a genuinely-broken issuance is one quiet P4 list instead of two permanent P3 warnings that everyone ignores.

07-kubernetes/09-certificate-expiring-soon.md — a live cert nearing expiry (renewal not happening).
07-kubernetes/17-certificate-renewal-stuck.md — a live cert whose scheduled renewal is overdue (the early signal).
This runbook — no cert at all.

Meaning

cert-manager exports certmanager_certificate_expiration_timestamp_seconds = 0 (Unix epoch) for a Certificate CR with no Secret containing an issued certificate. That state means one of two things:

First issuance in flight — the resource was created minutes ago and cert-manager is running its first Order/Challenge. Normal; completes in minutes. The for: 1h window is chosen so this never alerts.
Issuance permanently failing — the Order/Challenge has been failing since creation, so the resource has never had a secret. The Certificate cannot do its job: whatever references it (an Ingress TLS block, a Gateway listener, a webhook, a CNPG cluster) is either falling back to a default cert, broken, or — most commonly — gone entirely, leaving the Certificate as a zombie.

The metric carries name (the Certificate CR name), exported_namespace (the Certificate CR's namespace — the scraped namespace label always reads cert-manager, the controller pod's namespace), issuer_name, issuer_kind.

Impact

Usually none — most never-issued Certificates are leftovers for endpoints that no longer exist (see live examples below). The alert is P4 for exactly this reason.
If the hostname IS in use, the endpoint has no valid TLS from this Certificate: the ingress presents a default/fallback cert or refuses the handshake. Browsers see cert errors; clients fail x509.
Hidden cost even when unused — cert-manager keeps a live reconciliation loop for every Certificate: per failed challenge retry it spins a solver pod + Service + HTTPRoute/Ingress and re-attempts every backoff interval. A zombie burns controller cycles and ACME rate-limit headroom (repeated failed authorizations count against the Let's Encrypt rate limits that the healthy certs on the same domain/issuer need).
If the Certificate is webhook-backed (cert-manager's own, CNPG webhooks) and never issued, the corresponding feature is broken — raise priority accordingly.

Diagnosis

The alert labels carry k8s_cluster_name, exported_namespace, name, issuer_name, environment.

1. Is this a brand-new Certificate? (expected to self-resolve)

export KUBECONFIG=~/.kube/<cluster>.yaml
NS=<exported_namespace>
CERT=<name>
kubectl -n $NS get certificate $CERT -o jsonpath='{.metadata.creationTimestamp}{"\n"}'

If created minutes ago with Issuing=False→True in the status conditions, watch it — healthy first issuance completes in minutes and the alert never progresses past pending. A Certificate created months ago with zero secret is a zombie: continue.

2. Why is issuance failing?

kubectl -n $NS describe certificate $CERT | sed -n '/^Status:/,$p'
kubectl -n $NS get orders,challenges
kubectl -n cert-manager logs deploy/cert-manager --tail=200 | grep $CERT

The certificate status and cert-manager logs name the failure — same taxonomy as 07-kubernetes/09-certificate-expiring-soon.md § Diagnosis step 2 (HTTP-01 self-check 404/timeout, DNS-01 missing record, rate limit, broken issuer). Common zombie signatures:

Placeholder / undelegated dnsName — argocd.example.com, *-invalid-replace-me-*, or a domain with no DNS record at all (dig +short <dnsName> empty). Issuance can never succeed; nobody depended on the result.
HTTP-01 self-check 404 — the domain still resolves to the ingress, but no route serves /.well-known/acme-challenge/* for that host any more (the owner Ingress/Gateway/route was removed or the solver is not routed). curl -s -o /dev/null -w "%{http_code}" http://<dnsName>/.well-known/acme-challenge/probe returning 404 confirms it.

3. Is the Certificate actually used?

# Who references the secret the Certificate would produce?
kubectl -n $NS get certificate $CERT -o jsonpath='{.spec.secretName}{"\n"}'
kubectl -n $NS get ingress,gateway,httproute -o yaml | grep -B5 -A5 "<secretName>"
grep -rn "<secretName>\|$CERT" ~/cc/deployments --include="*.yaml" | grep -v .terraform

If nothing in the cluster and nothing in the deployments repo references the secret name (i.e. the resource is not even GitOps-managed — check the annotations for app.kubernetes.io/managed-by, or owner references from a still-existing Ingress), it is a leftover.

Note: Ingress-shim-created Certificates (ownerReference kind Ingress) live and die with their Ingress — if the Ingress is gone but the Certificate survived (shim adoption quirks can orphan them), handle the Ingress annotation, not just the Certificate.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production, as in 03-kafka/01-consumer-lag.md.

Step 2 — Pick the path

Zombie (nothing references it) — delete the Certificate CR. If it is GitOps-managed, delete it in the deployments repo (ArgoCD self-heals a kubectl delete on a managed resource); if hand-applied, confirm ownership first (search the deployments repo and ask the #platform channel for the domain), then kubectl -n $NS delete certificate $CERT. Deleting is also the ACME-friendliest option: it stops failed-authorization retries from burning rate limit that healthy certs on the same domain need.
Placeholder dnsName — either delete the resource, or fix the configuration that templated a placeholder into it (e.g. set the real ArgoCD domain in the chart values and let the Ingress-shim recreate a valid Certificate). Check siblings: the same chart on other clusters usually has the correct value to copy.
Legit hostname, failing challenge — follow 07-kubernetes/09-certificate-expiring-soon.md § Diagnosis step 2 / § Mitigation step 2 (fix the HTTP-01 route, DNS, rate limit, or issuer), then force a retry:

kubectl -n $NS certificate $CERT renew
kubectl -n $NS get certificate $CERT -w   # Ready flips Issuing -> True within minutes

Step 3 — Cleanup check

After deleting a zombie, confirm the challenge garbage is gone (cert-manager usually cleans up on delete): kubectl -n $NS get orders,challenges.

Verify

The alert clears once the Certificate gets a Secret (expiration timestamp becomes real) — the metric flips from 0 within a scrape interval of Ready=True.
Or the alert simply disappears when the zombie CR is deleted (series drops).
kubectl -n $NS get certificate $CERT shows Ready=True with a Not After in the future — from then on the siblings own it (09-certificate-expiring-soon.md for expiry drift, 17-certificate-renewal-stuck.md for stuck renewals).

Notes

The zero sentinel — expiration_timestamp_seconds = 0 is cert-manager's "no Secret" value; unguarded comparisons turn it into "expires in -20,742 days". The siblings now filter > 0; this alert explicitly matches == 0 and requires it for 1h so first-issuance races never alert.
exported_namespace, not namespace — the scrape pipeline overwrites namespace with the cert-manager controller pod's namespace (always cert-manager); the Certificate CR's real namespace is in exported_namespace. kubectl -n cert-manager get certificate ... returns NotFound — use the exported one.
Live examples (2026-10-16 investigation, all firing the old unguarded siblings for 30+ days each):
- infra-gateway-us-corti-app-tls (prod-us, shared) — dnsName infra.gateway.us.corti.app, HTTP-01 self-check returns 404 (domain resolves to the ingress but no route serves the challenge path), not in the GitOps repo — hand-applied leftover. TLS handshake for the hostname fails outright. Verdict pending ownership: delete.
- inference-gateway-invalid-replace-me-corti-app-tls (dev-weu, shared) — the name is the diagnosis. cert-manager has been retrialing its HTTP-01 for 30+ days, holding solver pods/HTTPRoutes. Verdict: delete.
- argocd-server-tls (dev-par02, argocd) — created 2026-07-02 with dnsName argocd.example.com, a placeholder domain than can never be issued (the same-named certs on every other cluster are fine — dev-par02's chart values never got the real domain). Verdict: set the real domain.
- chat-dev-weu-corti-app-cert (dev-par01, base) — the counter-example: created the same morning by the librechat Ingress-shim, first issuance in flight. Expected to self-resolve — the case the for: 1h window exists for.
Related — 07-kubernetes/09-certificate-expiring-soon.md (live cert nearing expiry) and 07-kubernetes/17-certificate-renewal-stuck.md (live cert, renewal overdue), same kubernetes.security group.