Documentation navigation
On Premise Customers
SFD
Hamilton County
CommonSpirit
GMR
Region Nord
On-Prem Synapse API-Key Deploy
Dns
DNS Flush
Kafka
Kafka Consumer Lag in Voice-Vault
Kafka Consumer Group Lag High (> 800)
HPA Max Replicas Exceed Kafka Topic Partitions
Storage
Ceph OSD Full / BlueFS Allocation Wedge
Query Ceph RGW (S3) Bucket Usage
Ceph Mon Quorum Loss / Node-Replacement Recovery
PVC Near Full Warning (>= 80%)
PVC Near Full Critical (>= 90%)
PVC Disk Usage Forecast > 95%
Ceph Cluster Storage Utilization High (> 90%)
Ceph Cluster Error
Ceph Cluster Warning
Observability
Alloy Operator Wedge
Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild
Grafana Dashboard Provisioning Error
Mimir Ingester Reaching Series Limit
Mimir Ingest Storage Wedged
Mimir Ingester PVC Near Full
Loki PVC Near Full
Mimir Self-Monitoring Scrape Stalled
Synapse Cluster Metrics Silent
Mimir Discarding Samples (Series Limit)
Loki Discarding Log Entries
Trivy Metrics Stale
Envoy Proxy Heap Near Ceiling
Envoy Proxy Heap Shedding
Envoy Proxy Overload Rejecting Requests
Traefik Elevated Error Spike (4xx/5xx)
Envoy Elevated Error Spike (4xx/5xx)
Scrape Targets Failing
Traefik 402 Payment Failure (Multiple Distinct Users)
Azure Audit Ingestion Silent
Databases
CNPG Cluster Has No Backup
CNPG Replication Unhealthy
CNPG Cluster Backup Stale (> 48h)
CNPG Cluster Has No WAL Recovery Point
Kubernetes
Pod ImagePull / ImagePullBackOff
Pod CrashLoopBackOff
Deployment Has No Running Replicas
Deployment Rollout Stuck
DaemonSet Has No Running Pods
StatefulSet Has No Running Replicas
Kubernetes Job Failed
Kubernetes CronJob Suspended
Certificate Expiring Soon
Node Memory/Disk Pressure (Kubelet Evicting Pods)
Container OOMKilled
Cluster Autoscaler Unhealthy
Trivy Critical Vulnerability One Week
Trivy Critical Vulnerability Four Weeks
Trivy High Vulnerability One Week
Trivy High Vulnerability Six Weeks
Certificate Renewal Stuck
Certificate Never Issued
Trivy Critical Vulnerability Immediate
API v2 Debug Endpoint Called in Production
Cluster Autoscaler Unregistered Nodes
Platform API Server Errors
Platform API Client Errors
Pod Pending Over 30 Minutes
Pod Pending (GPU) Over 60 Minutes
Cilium Policy Drops Burst
Capacity
CPU Throttling > 20%
HPA Utilization Critically High
GPUs Underutilized For 48h
Container Memory Leak (Rising Near Limit)
Nodegroup Utilization Critically High
Nodegroup Utilization High Warning
Identity
Keycloak Client Misconfigured
Hamilton County
Runbook for connecting to the Hamilton County on-premise customer environment.
1Password Credentials
Authorized Users
This environment only permits CJIS authorized users access.
Instructions
- Install the Fortinet FortiClient VPN client.
- If you are using a Mac, navigate to Settings > General > Login Items and Extensions > Network Extensions and enable the FortiTray.app privilege.
- If you are using Windows, download the "FortiClient VPN only (DIY, no support)" option, not "VPN/ZTNA".
- Authenticate using the credentials in the Hamilton County VPN 1Password entry.
- Receive a One Time Passcode in the
#devops-at-corti-aiSlack channel. - Create an SSH session to the server IP from the Hamilton County Call Recording Server 1Password entry.
- Authenticate using the credentials from the Hamilton County Call Recording Server 1Password entry.
Information about the deployment
This was one of our first deployments using the Call Recorder Installation Template. CART is installed as a Linux service, running out of /home/cortiai/cart/. You can interact with the service using docker commands or with systemctl.
Datadog is installed as a host agent, accessed with the datadog-agent command.
Current CART configuration:
{
"log-level": "DEBUG",
"metrics-port": 9876,
"capture": {
"device": "eno2",
"snapshot-length": 2048
},
"voip": {
"protocol": "sip",
"port": 5060,
"callMetadataEnhancer": "unify",
"callStateManager": "default"
},
"negative_filter": {
"callingparty": [
"^2[0-9]{3}$",
"^MOTOME.*",
"^GRM001ADM.*"
],
"calledparty": [
"^89[0-9]{5}$",
"^2[0]{3}$",
"^MOTOME.*",
"^GRM001ADM.*"
]
},
"sender": {
"type": "call-service-v2",
"async": true,
"call-service": {
"url": "https://api.hamiltonctyoh.motocorti.io/call-service",
"token": "<CORTI API TOKEN>"
}
},
"file": {
"buffer-limit": 200000,
"save-path": "data",
"persistence-in-hours": "30h",
"unprocessed-pcap-files-limit": 3000
}
}
Replace <CORTI API TOKEN> with the value from the Call Service Token entry in 1Password (linked above).