Runbook for a Job that has reached the Failed condition, triggered by the JobFailed Grafana-managed alert (frontmatter alert: block in this runbook, provisioned by the runbook_alerts Terraform module → infrastructure/prod/synapse-alerts). There is no matching Mimir-ruler alert.

Meaning

A Kubernetes Job runs a pod (or pods, with parallelism) to completion. It retries on pod failure up to spec.backoffLimit (default 6); once it exhausts the limit, the Job controller sets the Failed condition and the Job stops. kube_job_failed is 1 when the Job is in the Failed condition. kube_job_status_failed (the gauge) counts the failed pod attempts; kube_job_failed (the flag) is the terminal state — kube_job_failed == 1 is the cleanest "this Job gave up" signal.

kube_job_failed is a gauge from kube-state-metrics (deployed by the synapse-client chart, scraped on the cluster and remote-written to synapse-server Mimir). Each failed Job becomes its own alert instance with cluster, namespace, job_name, and environment labels preserved. Note: job_name is the Kubernetes Job name (e.g. my-backup-28614), not the Prometheus scrape job label — the workload is in the job_name label.

A Job may be standalone or owned by a CronJob (kube_job_owner records the owning CronJob). A CronJob-owned Job failing means the scheduled task failed this run; the CronJob will create a new Job on the next schedule, but if the failure is deterministic (bad image, bad config, missing dependency) every run will fail until the root cause is fixed. kube_job_failed does not clear when a new Job succeeds — it stays 1 for the failed Job until the Job is deleted (TTL or history limit), so the alert fires per failed Job. The for: 5m rides out the brief window between the last pod failure and the condition being set.

Impact

The task the Job was meant to do did not complete. Depending on the Job:
- Backup Job (CNPG ScheduledBackup creates a Backup CR, not a Job, but some backups run as Jobs) → no backup taken; the next backup window is the RPO gap. See 06-databases/01-cnpg-backup-failure.md.
- Migration Job (DB schema migration, run-once init) → the migration did not apply; the app may be waiting on it or running on the old schema. A failed migration can block a rollout.
- Cleanup / retention Job (transcript cleanup, retention) → the cleanup did not run; data accumulates. Usually low urgency but degrades over time.
- init / pre-install Job (Helm hook, argocd PostSync) → the install/upgrade's pre-step failed; the rollout may be blocked.
A CronJob-owned Job failing every run means the scheduled task is silently not happening. CronJob failures accumulate (up to failedJobsHistoryLimit, default 1) and each one fires this alert.
The Job's pods are gone (the controller deletes failed pods up to the history limit), so the failure reason must be read from the Job's Failed condition and the last pod's --previous logs before the pod is deleted.

Diagnosis

The alert labels carry cluster, namespace, job_name, and environment. From the alert in Grafana, note all of them.

1. Confirm the Job is Failed and get the failure reason

export KUBECONFIG=~/.kube/<cluster>.yaml   # e.g. prd-par01.yaml
NS=<namespace>
JOB=<job_name>
kubectl -n $NS get job $JOB -o wide
kubectl -n $NS describe job $JOB | sed -n '/^Conditions:/,$p'

The Failed condition's message and reason tell you the failure mode:

BackoffLimitExceeded — the pod failed backoffLimit times. The pod failures are the root cause — go to step 2.
DeadlineExceeded — the Job hit activeDeadlineSeconds. The pod ran too long; check for a stuck/hung process (deadlock, waiting on a dependency). Not a crash — the task did not finish in time.
Job was active longer than specified deadline — same as DeadlineExceeded.

2. Find the failing pod and read its logs

The Job's pods are named <job>-<hash>. List them and read the last (failed) pod's logs:

kubectl -n $NS get pods -l job-name=$JOB -o wide
# the most recent failed pod:
POD=$(kubectl -n $NS get pods -l job-name=$JOB --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')
kubectl -n $NS logs $POD --previous --tail=50
kubectl -n $NS logs $POD --tail=50
kubectl -n $NS describe pod $POD | sed -n '/^Events:/,$p'

--previous reads the crashed instance's logs — the failure reason is in the last lines. If the pod has already been deleted (history limit), the logs are gone; use the Job's Failed condition message and the kube_job_status_failed count as the only signal.

The pod's waiting.reason routes to the sibling runbooks:

ErrImagePull / ImagePullBackOff → 07-kubernetes/01-pod-imagepull.md (bad image tag — common for a Job whose image tag was bumped but the image was never pushed to ACR).
CrashLoopBackOff → 07-kubernetes/02-pod-crashloop.md (the container exits — bad config, missing env var/secret, migration failure, bad args).

3. Check if the Job is CronJob-owned (recurring failure?)

kubectl -n $NS get job $JOB -o jsonpath='{.metadata.ownerReferences[?(@.kind=="CronJob")].name}{"\n"}'

If non-empty, the Job is owned by a CronJob. Check the CronJob's recent Job history for a pattern of failures:

CRONJOB=<owner-cronjob-name>
kubectl -n $NS get jobs -l job-name | grep $CRONJOB   # or list jobs in the namespace
kubectl -n $NS get jobs -o custom-columns='NAME:.metadata.name,FAILED:.status.failed,SUCCESS:.status.succeeded,START:.status.startTime'

If the last several Jobs all Failed, the failure is deterministic — fix the root cause (step 2) before the next schedule, or the next run fails too. The CronJob keeps creating failing Jobs, each of which fires this alert, until fixed or the CronJob is suspended (kubectl patch cronjob $CRONJOB -p '{"spec":{"suspend":true}}' as a temporary hold).

4. Check for a missing dependency (Job-specific failures)

Jobs often fail for reasons distinct from serving workloads:

DB unreachable — a migration/backup Job cannot connect to the DB. Check the DB is up and reachable from the namespace (NetworkPolicy on 5432 — see 06-databases/).
Missing secret / config — the Job references a Secret key that does not exist (ExternalSecret not synced). Check kubectl get externalsecret -n $NS.
Permissions — the Job's ServiceAccount lacks a required RBAC permission (e.g. a cleanup Job that lists/deletes resources). Check the Job's serviceAccountName and its Role/RoleBinding.
Resource quota — the Job's pod cannot schedule (quota full, no capacity). The pod will be Pending rather than Failed — but a Job whose pods never start eventually hits BackoffLimitExceeded and goes Failed.

Mitigation

Step 1 — Activate PIM

Activate the required Azure Privileged Identity Management roles before touching production (MultiTenantEuOnlyAccessPimRoles / DevOpsPimRoles / Platform Team (privileged)), as in 03-kafka/01-consumer-lag.md.

Step 2 — Fix the root cause

Follow Diagnosis steps 2–4 to the specific cause:

Bad image tag — roll the chart image tag back to the last known-good tag, or push the missing image to ACR. See 07-kubernetes/01-pod-imagepull.md.
App crash (config / env var / secret / migration) — fix the chart values or ExternalSecret in a PR. See 07-kubernetes/02-pod-crashloop.md. For a migration failure, the owning team must fix/revert the migration; rolling back the image tag to pre-migration is the fast unblock.
Missing dependency (DB / secret / RBAC) — fix the chart values / ExternalSecret / Role in a PR. Confirm the DB is up (06-databases/01-cnpg-backup-failure.md) or the Secret is synced before re-running.
DeadlineExceeded — raise activeDeadlineSeconds in the chart values if the task legitimately needs longer, or fix the hung process (deadlock, dependency timeout).

Step 3 — Re-run the Job (one-shot Jobs)

A standalone (non-CronJob) Job does not retry after Failed. Once the root cause is fixed, create a new Job from the same spec:

kubectl -n $NS get job $JOB -o json | jq 'del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.status,.spec.selector,.spec.template.metadata.labels["controller-uid"])' | kubectl -n $NS apply -f -
# or, for a CronJob-owned Job, wait for the next schedule (or trigger manually):
kubectl -n $NS create job --from=cronjob/<cronjob-name> <manual-job-name>

For a CronJob, the next scheduled run creates a fresh Job automatically; only re-run manually if you cannot wait for the schedule.

Step 4 — Suspend a recurring-failing CronJob (emergency hold)

If a CronJob is failing every run and the fix will take too long, suspend it so it stops creating failing Jobs (and stops alerting) until the fix merges:

kubectl -n $NS patch cronjob <cronjob-name> --type=merge -p '{"spec":{"suspend":true}}'

This is a temporary hold — the CronJobSuspended alert (sibling runbook 08-cronjob-suspended.md) will fire on it, which is the correct signal ("this CronJob is intentionally off"). Unsuspend (suspend: false) once the fix is in. For an ArgoCD-managed CronJob, make the suspend change in a chart-values PR so GitOps and the live state agree (ArgoCD self-heal reverts a manual patch in ~15s unless the chart value also sets it).

Verify

The alert clears once the failed Job is deleted (TTL ttlSecondsAfterFinished, or failedJobsHistoryLimit for a CronJob-owned Job cleans it up). A new successful run does not flip kube_job_failed back to 0 — the failed Job stays 1 until deleted; the alert clears when the Job object is gone. Practically, for a CronJob, the failed Job is removed by the history limit after a new run.
kubectl -n $NS get job $JOB shows the Job gone, or (for a re-run) a new Job with SUCCEEDED 1.
For a CronJob, the next scheduled run completes: kubectl -n $NS get jobs -l job-name | grep <cronjob> shows a recent Job with SUCCEEDED 1.
For a re-run, the pod reaches Completed 0/1 (succeeded) rather than the failure state.