Documentation navigation
On Premise Customers
SFD
Hamilton County
CommonSpirit
GMR
Region Nord
On-Prem Synapse API-Key Deploy
Dns
DNS Flush
Kafka
Kafka Consumer Lag in Voice-Vault
Kafka Consumer Group Lag High (> 800)
HPA Max Replicas Exceed Kafka Topic Partitions
Storage
Ceph OSD Full / BlueFS Allocation Wedge
Query Ceph RGW (S3) Bucket Usage
Ceph Mon Quorum Loss / Node-Replacement Recovery
PVC Near Full Warning (>= 80%)
PVC Near Full Critical (>= 90%)
PVC Disk Usage Forecast > 95%
Ceph Cluster Storage Utilization High (> 90%)
Ceph Cluster Error
Ceph Cluster Warning
Observability
Alloy Operator Wedge
Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild
Grafana Dashboard Provisioning Error
Mimir Ingester Reaching Series Limit
Mimir Ingest Storage Wedged
Mimir Ingester PVC Near Full
Loki PVC Near Full
Mimir Self-Monitoring Scrape Stalled
Synapse Cluster Metrics Silent
Mimir Discarding Samples (Series Limit)
Loki Discarding Log Entries
Trivy Metrics Stale
Envoy Proxy Heap Near Ceiling
Envoy Proxy Heap Shedding
Envoy Proxy Overload Rejecting Requests
Traefik Elevated Error Spike (4xx/5xx)
Envoy Elevated Error Spike (4xx/5xx)
Scrape Targets Failing
Traefik 402 Payment Failure (Multiple Distinct Users)
Azure Audit Ingestion Silent
Databases
CNPG Cluster Has No Backup
CNPG Replication Unhealthy
CNPG Cluster Backup Stale (> 48h)
CNPG Cluster Has No WAL Recovery Point
Kubernetes
Pod ImagePull / ImagePullBackOff
Pod CrashLoopBackOff
Deployment Has No Running Replicas
Deployment Rollout Stuck
DaemonSet Has No Running Pods
StatefulSet Has No Running Replicas
Kubernetes Job Failed
Kubernetes CronJob Suspended
Certificate Expiring Soon
Node Memory/Disk Pressure (Kubelet Evicting Pods)
Container OOMKilled
Cluster Autoscaler Unhealthy
Trivy Critical Vulnerability One Week
Trivy Critical Vulnerability Four Weeks
Trivy High Vulnerability One Week
Trivy High Vulnerability Six Weeks
Certificate Renewal Stuck
Certificate Never Issued
Trivy Critical Vulnerability Immediate
API v2 Debug Endpoint Called in Production
Cluster Autoscaler Unregistered Nodes
Platform API Server Errors
Platform API Client Errors
Pod Pending Over 30 Minutes
Pod Pending (GPU) Over 60 Minutes
Cilium Policy Drops Burst
Capacity
CPU Throttling > 20%
HPA Utilization Critically High
GPUs Underutilized For 48h
Container Memory Leak (Rising Near Limit)
Nodegroup Utilization Critically High
Nodegroup Utilization High Warning
Identity
Keycloak Client Misconfigured
SFD
Runbook for connecting to the SFD (Seattle Fire Department) on-premise customer environment.
1Password Credentials
Authorized Users
All authorized Corti personnel may access this Call Recording instance.
Instructions
- Connect to a Windows VM.
- Open the Microsoft Edge browser.
- Navigate to https://connect.seattle.gov/ and authenticate with the credentials in 1Password.
- Once authenticated, you'll be given the option to connect to a machine — connect to the CART machine.
- While connecting, ensure the Pulse Secure Launcher is installed. If not, install it, then try to launch the session again.
- Once launched, an RDP session opens to the jumphost. Authenticate with the
SFD\Test1credentials. - From the jumphost, open MobaXterm, where the session to the CART machine exists.
- If needed, use the
corticredentials for sudo access.
Information about the deployment
The production deployment runs via docker-compose in /home/corti/production_cart/.
Datadog also runs as a container via docker-compose in /home/corti/datadog/.
Current CART runtime configuration:
{
"log-level": "DEBUG",
"capture": {
"device": "enp179s0f1",
"snapshot-length": 2048
},
"file": {
"save-path": "data",
"buffer-limit": 1000000,
"persistence-in-hours": "72h"
},
"filter": {
"calledparty": [
"^22.*", "^213[5-9]$", "^214[0-3]$"
],
"callingparty": [
"^22.*", "^213[5-9]$", "^214[0-3]$"
]
},
"voip": {
"protocol": "SIP",
"port": 5060
},
"sender": {
"type": "call-service-v2",
"async": true,
"call-service": {
"url": "https://api.sfdproduction.motocorti.io/call-service",
"token": "<CORTI API TOKEN>"
}
}
}
Replace <CORTI API TOKEN> with the value from the Call Service Token entry in 1Password (linked above).