Runbook for a CronJob that is suspended, triggered by the CronJobSuspended Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). There is no matching Mimir-ruler alert.

This is a config-drift / silent-stop signal, not a failure. priority: 4 (info) reflects that — a suspended CronJob is often intentional; the alert exists to catch the case where a suspension was meant to be temporary and was never resumed.

Meaning

A CronJob with spec.suspend: true does not create new Jobs on its schedule — the controller skips every scheduled tick while suspended. Existing, already-running Jobs are not stopped; only new ones are prevented. kube_cronjob_spec_suspend is 1 when the CronJob is suspended, 0 otherwise.

kube_cronjob_spec_suspend is a gauge from kube-state-metrics (deployed by the synapse-client chart, scraped on the cluster and remote-written to synapse-server Mimir). The expr is pre-aggregated with sum by (k8s_cluster_name, environment, namespace, cronjob) to strip the scrape-integration labels (job, instance, source), so the alert carries only k8s_cluster_name, environment, namespace, and cronjob and reads as a workload problem — not as a problem with kube-state-metrics itself. Each suspended CronJob becomes its own alert instance.

The for: 1h rides out brief intentional suspensions (e.g. suspending a cleanup CronJob during a maintenance window). A CronJob suspended for 1h+ is either forgotten (an incident suspension never resumed) or deliberately long-term (in which case the alert is expected noise and can be silenced for that CronJob).

Impact

The scheduled task is not running. Depending on the CronJob:
- Backup / retention CronJob → no backup/retention happening; data accumulates or RPO grows.
- Cleanup CronJob (transcript cleanup, expired-data purge) → data accumulates; disk usage grows (may eventually trip 04-storage/04-pvc-near-full.md).
- Sync CronJob (catalog sync, cache refresh) → the synced resource goes stale; consumers see outdated data.
- Report / billing CronJob → reports not generated; downstream billing/usage tracking gaps.
The impact is silent — nothing crashes, no other alert fires (the scheduled task simply does not happen). This is exactly the failure mode this alert exists to catch.
A suspended CronJob does not affect already-running Jobs or the rest of the namespace.

Diagnosis

The alert labels carry k8s_cluster_name, namespace, cronjob, and environment. From the alert in Grafana, note all of them.

1. Confirm the CronJob is suspended and check why

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=<namespace>
CJ=<cronjob>
kubectl -n $NS get cronjob $CJ -o wide
kubectl -n $NS get cronjob $CJ -o jsonpath='suspend={.spec.suspend}{"\n"}schedule={.spec.schedule}{"\n"}lastSchedule={.status.lastScheduleTime}{"\n"}'

Confirm suspend=true. Check lastScheduleTime — how long since the last run (the alert fires after 1h suspended, but lastScheduleTime tells you when it last actually ran).

2. Was the suspension intentional?

Look for the reason. Common intentional reasons:

Incident hold — the CronJob was suspended during an incident (e.g. to stop a failing backup Job loop) and never resumed. Check the incident timeline / oncall notes; if the incident is over, unsuspend.
Maintenance window — suspended during a maintenance window and the window is over.
Deliberate disable — the CronJob is being decommissioned or paused long-term. In this case the alert is expected noise — silence it for that CronJob in Alertmanager/incident.io rather than unsuspend.

If none of these apply, the suspension is likely accidental — a bad rollout set suspend: true (a chart default flipped, a values override mistake). Unsuspend (Mitigation).

3. Check the chart values for the source

For an ArgoCD-managed CronJob, the suspend value comes from the chart values:

# Find the chart config that rendered this CronJob's suspend field
kubectl -n $NS get cronjob $CJ -o jsonpath='{.metadata.labels.app\.kubernetes\.io/managed-by}{"\n"}{.metadata.annotations.argocd\.argoproj\.io/sync-wave}{"\n"}'

If suspend: true is in the chart values (not a manual patch), the GitOps source intends it — fix the chart values in a PR to unsuspend. If it was a manual kubectl patch (not in the chart), ArgoCD self-heal will revert it on the next sync (~15s on prd clusters), so it should not persist long enough to fire this alert — if it is firing, the chart values set it.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Unsuspend (if not intended)

If the suspension was not deliberate, unsuspend via a chart-values PR (GitOps):

# in the chart values for the CronJob
schedule: "0 */6 * * *"
suspend: false   # or omit (defaults to false)

Do not kubectl patch cronjob ... -p '{"spec":{"suspend":false}}' directly — ArgoCD self-heal reverts it in ~15s. Make the change in the chart values, merge, and let ArgoCD sync.

Step 3 — Leave suspended (if intended) and silence the alert

If the suspension is deliberate (decommissioned, long-term pause), leave it and silence this alert for that CronJob in Alertmanager/incident.io routing so it does not page oncall. The for: 1h window means a deliberate suspension will alert once after an hour; a silence prevents repeat noise.

Step 4 — If the CronJob was suspended to stop a failure loop

If the CronJob was suspended because every run was failing (see 07-job-failed.md), the suspension is a temporary hold. Fix the underlying Job failure first (see 07-job-failed.md Mitigation), then unsuspend so the scheduled task resumes with the fix in place.

Verify

The alert clears once kube_cronjob_spec_suspend == 0 for the CronJob, sustained for the for: 1h window. Practically, the suspension flag flips to 0 within a scrape interval after the chart-values PR syncs; the alert clears 1h later.
kubectl -n $NS get cronjob $CJ -o jsonpath='{.spec.suspend}' returns <nil> or false.
The CronJob resumes scheduling: kubectl -n $NS get cronjob $CJ shows lastScheduleTime advancing on the next scheduled tick, and new Jobs appear (kubectl -n $NS get jobs -l job-name | grep $CJ).
For a deliberately-suspended CronJob, the alert is silenced in routing and does not page.