Command-line tool for distributed LLM inference benchmarks on SLURM clusters using TensorRT LLM, SGLang and vLLM.
Overview
Command-line tool for distributed LLM inference benchmarks on SLURM clusters using TensorRT LLM, SGLang and vLLM. Replace complex shell scripts and 50+ CLI flags with a declarative schema: 2 YAML recipe: engine: names the engine, roles: describes each worker role, and services: covers everything launched next to the workers. srtctl ships a skill that teaches Claude Code, Codex or Cursor how to set up a checkout, write srtslurm.yaml, author and validate recipes, submit them, and read the results. Install it into the checkout and hand the agent the checkout: srtctl-mcp is the matching MCP server: schema tools (schema_summary, explain_field, validate_config, resolve_config) anywhere, and job tools (submit_job, dry_run, job_status, job_logs, list_jobs, cancel_job) when it runs on a login node inside the checkout. See docs/README.md. - Installation - Setup and configuration - Examples - Runnable 2.
README
srtctl
Command-line tool for distributed LLM inference benchmarks on SLURM clusters using SGLang, vLLM, TensorRT LLM, TileRT, TokenSpeed and AMD’s ATOM. Replace complex shell scripts and 50+ CLI flags with a declarative schema: 2 YAML recipe: engine: names the engine, roles: describes each worker role, and services: covers everything launched next to the workers.
Documentation: https://nvidia.github.io/srt-slurm/ (source in docs/; agents: llms.txt)
Quick Start
git clone https://github.com/NVIDIA/srt-slurm.git
cd srt-slurm
uv sync --no-dev
# One-time setup: downloads etcd, NATS, uv and the tachometer binaries for the
# compute nodes' architecture and writes srtslurm.yaml (account, partition, GPUs per node)
make setup ARCH=aarch64 # or ARCH=x86_64
uv run srtctl dry-run -f examples/sglang/dynamo-agg.yaml # render without submitting
uv run srtctl apply -f examples/sglang/dynamo-agg.yaml # submit
Installation covers cluster-specific setup; examples/ has runnable recipes, one per frontend and topology.
Let an agent drive it
srtctl ships a skill that teaches Claude Code, Codex or Cursor how to set up a checkout, write srtslurm.yaml, author and validate recipes, submit them, and read the results. Install it into the checkout and hand the agent the checkout:
uv run srtctl skill --target claude # .claude/skills/srtctl/SKILL.md
uv run srtctl skill --target codex # .codex/skills/srtctl/SKILL.md
uv run srtctl skill --target cursor # .cursor/rules/srtctl.mdc
The srtctl-mcp server, the recipe JSON Schema for editors, and llms.txt are described in the documentation home.
Where things are documented
- Write a recipe: Recipe Guide, then the topic pages it links (engines, topology, frontends, benchmarks, services, …)
- Every field, type, default and allowed value: Schema Reference (generated from the code)
- Every command and flag: CLI Guide and CLI Reference (generated)
- Run and debug: Monitoring, SLURM FAQ
- Analyze: Profiling, DSight
- Old recipes: Legacy (v1) layout;
srtctl migraterewrites them
Recommended Tools
Try a different keyword or remove a filter.
Install
npx skillfish add nvidia/srt-slurm