Documentation navigation
On Premise Customers
SFD
Hamilton County
CommonSpirit
GMR
Region Nord
On-Prem Synapse API-Key Deploy
Dns
DNS Flush
Kafka
Kafka Consumer Lag in Voice-Vault
Kafka Consumer Group Lag High (> 800)
HPA Max Replicas Exceed Kafka Topic Partitions
Storage
Ceph OSD Full / BlueFS Allocation Wedge
Query Ceph RGW (S3) Bucket Usage
Ceph Mon Quorum Loss / Node-Replacement Recovery
PVC Near Full Warning (>= 80%)
PVC Near Full Critical (>= 90%)
PVC Disk Usage Forecast > 95%
Ceph Cluster Storage Utilization High (> 90%)
Ceph Cluster Error
Ceph Cluster Warning
Observability
Alloy Operator Wedge
Mimir Returns Random / Inconsistent Query Results After a Storage Rebuild
Grafana Dashboard Provisioning Error
Mimir Ingester Reaching Series Limit
Mimir Ingest Storage Wedged
Mimir Ingester PVC Near Full
Loki PVC Near Full
Mimir Self-Monitoring Scrape Stalled
Synapse Cluster Metrics Silent
Mimir Discarding Samples (Series Limit)
Loki Discarding Log Entries
Trivy Metrics Stale
Envoy Proxy Heap Near Ceiling
Envoy Proxy Heap Shedding
Envoy Proxy Overload Rejecting Requests
Traefik Elevated Error Spike (4xx/5xx)
Envoy Elevated Error Spike (4xx/5xx)
Scrape Targets Failing
Traefik 402 Payment Failure (Multiple Distinct Users)
Azure Audit Ingestion Silent
Databases
CNPG Cluster Has No Backup
CNPG Replication Unhealthy
CNPG Cluster Backup Stale (> 48h)
CNPG Cluster Has No WAL Recovery Point
Kubernetes
Pod ImagePull / ImagePullBackOff
Pod CrashLoopBackOff
Deployment Has No Running Replicas
Deployment Rollout Stuck
DaemonSet Has No Running Pods
StatefulSet Has No Running Replicas
Kubernetes Job Failed
Kubernetes CronJob Suspended
Certificate Expiring Soon
Node Memory/Disk Pressure (Kubelet Evicting Pods)
Container OOMKilled
Cluster Autoscaler Unhealthy
Trivy Critical Vulnerability One Week
Trivy Critical Vulnerability Four Weeks
Trivy High Vulnerability One Week
Trivy High Vulnerability Six Weeks
Certificate Renewal Stuck
Certificate Never Issued
Trivy Critical Vulnerability Immediate
API v2 Debug Endpoint Called in Production
Cluster Autoscaler Unregistered Nodes
Platform API Server Errors
Platform API Client Errors
Pod Pending Over 30 Minutes
Pod Pending (GPU) Over 60 Minutes
Cilium Policy Drops Burst
Capacity
CPU Throttling > 20%
HPA Utilization Critically High
GPUs Underutilized For 48h
Container Memory Leak (Rising Near Limit)
Nodegroup Utilization Critically High
Nodegroup Utilization High Warning
Identity
Keycloak Client Misconfigured
Region Nord
Runbook for connecting to the Region Nord on-premise customer environment.
Access Restrictions
Region Nord restricts access, and only permits authorized users.
Presently Authorized Users
- Pablo Redondo
Instructions
- Install Citrix Workspace: https://www.citrix.com/platform/citrix-workspace-app.html
- Log in to https://citrixstore.rn.dk/Citrix/ESAWeb/ with your RN login. (Contact Region Nord if you need an account.)
- Once connected, Citrix should start automatically and connect to the ESA 2 host (otherwise just click on it). It has a MobaXterm instance with the CART server saved.
Information about the deployment
The production deployment runs via docker-compose in /opt/corti/cart/.
Datadog runs as a service (sudo systemctl status datadog-agent).
Current CART runtime configuration:
{
"Deployment": "RegionNord",
"log-level": "INFO",
"call-ttl": "10s",
"capture": {
"device": "ens160",
"snapshot-length": 4096,
"bpf-filter": "ip host 172.18.234.97 or ip host 172.18.234.68"
},
"file": {
"save-path": "/datadisk/cart",
"buffer-limit": 3000000,
"persistence-in-hours": "48h"
},
"filter": {
"callingparty": [
"Plads"
],
"calledparty": [
"Plads"
]
},
"voip": {
"protocol": "SIPREC",
"port": 5060,
"callMetadataEnhancer": "unify"
},
"sender": {
"type": "call-service-v2",
"async": true,
"call-service": {
"url": "https://api.regionnordems.motocorti.io/call-service",
"token": "<CORTI API TOKEN>"
}
}
}