Runbook for a debug-logging-enable call to the API v2 admin endpoint in production. The /v2/admin/triggers/log/debugEnable endpoint enables debug-level logging in the API v2 service. It is high-risk because debug logs may contain sensitive request data (transcripts, prompts, PII) and the endpoint has poor access protection.

Meaning

Traefik (the ingress controller) logs every HTTP request it proxies as a JSON access log line. This alert counts log lines matching:

service_name="traefik" — Traefik access logs (not app logs)
environment="production" — prod clusters only
RequestHost=~"api.*.corti.app" — external-facing API v2 hosts
RequestPath="/v2/admin/triggers/log/debugEnable" — the debug-enable trigger

When the count in a 5m window exceeds 0 (any call), the alert fires immediately (for: 0m).

The DataDog monitor this replaces used a > 1 threshold (2+ calls in 5m), which would miss a single debug-enable call. Migrated to > 0 so any access alerts — a single call is sufficient to enable debug logging.

Impact

Debug logging in production can:

Expose sensitive data — debug-level logs may contain full request/response bodies, tokens, PII, or patient data written to Loki.
Increase log volume — debug level dramatically increases log throughput, which can overwhelm Loki ingestion and fill disk.
Signal unauthorized access — if the caller is not an internal employee, the admin endpoint may be exposed to the internet without adequate auth.

This is a P1 critical security alert — treat any firing as a potential security incident until verified.

Diagnosis

1. Identify the caller

The Traefik access log line carries the caller IP. Query Loki for the matching lines (port-forward synapse-server-loki-gateway in monitoring-system, header X-Scope-OrgID: corti):

{service_name="traefik", environment="production"}
| json
| keep k8s_cluster_name, ClientHost, RequestHost, RequestPath, RequestMethod, DownstreamStatus, StartUTC
| RequestPath="/v2/admin/triggers/log/debugEnable"

Note the ClientHost (caller IP) and StartUTC (timestamp) from the result.

2. Verify with the API v2 engineers

The DataDog message template says: "You MUST reach out to the API v2 engineers to verify that this has been performed by an internal employee!"

Contact the API v2 team (platform team) and confirm:

Was the debug-enable call authorized?
Who made the call (match the ClientHost IP to a known internal IP or VPN range)?
Why was debug logging needed (active debugging session, incident response)?

3. Check if debug logging is still enabled

If debug logging was enabled and not disabled, it should be turned off to stop sensitive data exposure and log volume. The API v2 service likely has a corresponding /v2/admin/triggers/log/debugDisable endpoint or a config toggle.

Mitigation

Step 1 — Disable debug logging

If the call was unauthorized or debug logging is no longer needed, disable it immediately:

Call the disable endpoint (/v2/admin/triggers/log/debugDisable) if available
Or restart the API v2 pods (debug logging is typically not persistent across restarts): kubectl rollout restart deployment/<api-v2-deployment> -n <namespace>

Step 2 — Investigate the caller

If the ClientHost IP is not a known internal IP:

Check Traefik logs for other admin-endpoint calls from the same IP around the same time
Check if the endpoint is exposed without auth (it should be behind authorization-service ForwardAuth)
If unauthorized access is confirmed, escalate as a security incident

Step 3 — Restrict the endpoint (longer term)

The endpoint has "poor protection" per the DataDog description. Work with the API v2 team to add proper authorization (Casbin RBAC via authorization-service) or restrict it to internal-only traffic (NetworkPolicy / Traefik middleware IP-allow-list).

Verify

The Loki query for the debug-enable path returns no new matches after the alert fired.
Debug logging is disabled (API v2 logs return to INFO level — check Loki for level=debug lines from the service).
The alert clears after the 5m count_over_time window rolls past the last call.

Notes

LogQL alert — uses logql: (not expr:); the Loki ruler evaluates this and dispatches via the Mimir Alertmanager.
keep drops stream labels — | keep RequestHost, RequestPath discards every unlisted label, including stream labels like k8s_cluster_name. This alert originally kept only the parsed fields; the sum by (k8s_cluster_name) then grouped on an empty label, merging all production clusters into one series and rendering the summary without a cluster name. Fixed by including k8s_cluster_name in the keep list (2026-10-28); any keep pipeline that groups by a stream label must repeat that label in the keep list.
Threshold — migrated from DataDog > 1 to > 0; a single debug-enable call is sufficient to fire.
for: 0m — fires immediately on any match; no persistence delay for this security alert.
Traefik JSON logs — the | json | keep RequestHost, RequestPath pipeline parses the Traefik access log JSON and keeps only the fields needed for the filter, minimizing cardinality.
Related — the Trivy vulnerability runbooks (07-kubernetes/13-18-trivy-*.md) are in the same platform.security group.