Runbook for the shared GitHub App rate-limit budget running low — Orca uses a single GitHub App installation (corticph GitHub integration, installation 64334890) for every environment. GitHub grants 15,000 requests/hour for the core resource (REST API) and 12,500 points/hour for graphql. All Orca deployments — dev-weu, staging-eu, prod-weu — authenticate through this one installation, so every workflow's GitHub calls in any cluster draw from the same hourly budget.

Meaning

Orca's GitHub HTTP transport records GitHub's X-Ratelimit-* response headers as Prometheus gauges (github_ratelimit_remaining, github_ratelimit_limit, github_ratelimit_reset_timestamp_seconds; deployments PR #26206 for PLA-7611). The alert divides remaining by limit and fires at < 0.20 — 80% of the window's budget consumed — with a for: 10m sustain so a short mid-window dip does not page.

min by (installation, resource) deduplicates across pods and clusters: the budget is shared, so the alert fires once per resource (core, graphql) regardless of which environment burned it. A github_ratelimit_remaining gauge only updates when that pod makes a GitHub call, so the ratio reflects the view of the most recently active reporter — correct, because the underlying counter is installation-global at GitHub's side.
The resource label distinguishes budgets: core = REST API requests, graphql = GraphQL points. Both reset on the same hourly cycle. A busy release pipeline usually burns core; GraphQL-heavy automation (e.g. autoapprove) burns graphql.
The window is a rolling hour anchored per installation; after github_ratelimit_reset_timestamp_seconds the budget refills to full.

Impact

Degraded, not down: if the budget reaches zero, GitHub starts answering 403 with X-RateLimit-Remaining: 0. Orca then fails every GitHub interaction until reset — but note the alert fires at 80%, while requests still succeed.
Release workflows fail or retry/back off at every GitHub step (branch creation, PRs, status checks). Long DBOS workflows survive (steps retry on recovery), but a release busy in its GitHub step errors visibly.
Config-repo reads (release plans, deployment manifests) stall, and autoapprove (PLA-7558) stops reviewing PRs.
Customer-facing services are unaffected — this alert is about the internal release/orchestration pipeline.

Diagnosis

1. Confirm the ratio and find the reset time

min by (installation, resource) (github_ratelimit_remaining / github_ratelimit_limit)
github_ratelimit_reset_timestamp_seconds - time()        # seconds until the budget refills

2. Identify which environment is burning the budget

deriv(github_ratelimit_remaining{resource="core"}[10m])

The most negative deriv is the heaviest consumer; each series carries cluster, environment, pod. Repeat for resource="graphql" if the alert fired on the GraphQL budget.

prod-weu burning: check for a release storm in the orca UI (https://orca.corti.app, Releases) — many concurrent releases all polling PRs/statuses multiply GitHub calls.
dev-weu / staging-eu burning: a looping or experimental workflow. Check pods' logs for HTTP retry storms:

kubectl --context dev-weu -n base logs deploy/orca --since=15m | grep -iE 'github|retry|rate limit' | tail -30

3. Check whether requests are already failing

increase(github_ratelimit_throttled_total[1h])

Any non-zero value means GitHub is already answering 403/429 — the budget is exhausted right now and impact is live, not incipient.

Mitigation

Step 1 — Reduce consumption

Identify the runaway consumer (Diagnosis step 2) and stop it: cancel releases in the orca UI (orca release cancel <id>) or stop the workflow loop in the offending environment.
A release storm from legitimate activity: let it run, but avoid triggering new releases until after the reset (Diagnosis step 1).

Step 2 — Wait out the window

There is no way to raise an installation's hourly limit. The window resets at github_ratelimit_reset_timestamp_seconds and the budget refills to github_ratelimit_limit. Verify with:

min by (installation, resource) (github_ratelimit_remaining / github_ratelimit_limit)

Past events and current burn are visible in Grafana with the github_ratelimit_* metrics (cluster label distinguishes environments).

Step 3 — Fix the root cause

If one workflow or environment repeatedly exhausts the budget, file an issue against corticph/orca with your findings (requests usually multiply through redundant polling or missing pagination caching).

Verify

min by (installation, resource) (github_ratelimit_remaining / github_ratelimit_limit) recovers above 0.20 and the alert clears after the for: 10m window.
increase(github_ratelimit_throttled_total[15m]) stays at zero — no 403/429 responses.
The guilty environment's release activity is back to normal (or intentional), and GitHub-dependent workflows (releases, autoapprove) complete without errors.