Tools
CLI tools available under orca tools. This list is auto-discovered from the command tree.
Commands
vllm-bench
Token throughput benchmark for vLLM deployments running in Kubernetes
Subcommands
run
Run a sustained throughput benchmark
--duration
(default: 2m0s)
— benchmark duration
--max-tokens
(default: 512)
— max completion tokens per request
--prompt
(default: Explain in detail how transformer attention mechanisms work in large language models.)
— prompt text
--workers
(default: 1)
— concurrent workers; each sends requests sequentially
sweep
Sweep context sizes and plot throughput vs latency
--out-tokens
(default: 128)
— output tokens per request
--poison-prefix-cache
(default: false)
— prepend a random ID to each prompt to defeat vLLM prefix caching
--requests
(default: 20)
— requests per context size (averaged)
--sizes
(default: 1024,4096,16384,32768,65536,131072)
— comma-separated input token targets