ComputeSDK Benchmarks

The default platform for cloud infrastructure benchmarks.

ComputeSDK Benchmarks publishes reproducible, provider-neutral performance measurements for the infrastructure that AI agents and developer platforms run on: sandboxes, object storage, headless browsers, and AI gateways. Every test is automated, every result is committed as JSON, and the methodology is public.

Live leaderboards and full methodology: https://www.computesdk.com/benchmarks

For LLMs and agents: see llms.txt for a machine-readable index of this repo, and results/schema.json for the JSON Schema of every benchmark result file.

Writing and running benchmarks

• WRITING_BENCHMARKS.md — step-by-step guide to authoring benchmarks.
• examples/ — runnable *.bench.ts examples covering every benchsdk capability.
• packages/create-bench/README.md — scaffold a new benchmark project in one command.
• packages/benchsdk-runner/README.md — runner API and bench run reference.
• .agents/skills/benchsdk-cli/SKILL.md — full bench CLI reference (auth, org, runs, results, artifacts, export).

Partners

Our partners support our independent benchmarks.

★ SILVER

   

• BRONZE

     

     

     

BENCHMARKS POWERED BY

Become a sponsor →

What We Measure

We benchmark the infrastructure that AI agents and developer platforms rely on:

• Sandboxes — cold-start and concurrent Time to Interactive (TTI): API request to first successful command inside a fresh sandbox.
• Object storage — upload/download latency and throughput across providers and file sizes.
• Headless browsers — session creation, navigation, and step throughput.
• AI gateways — cold/warm connection latency and time to first token.

Methodology

Every benchmark uses the same open code against every provider, with fixed workloads and a fixed scoring ceiling. Tests run automatically on GitHub Actions, and every result is committed as JSON to this repo. Each metric includes min, max, median, P95, and P99; a composite score rewards both speed and reliability.

For full details on each suite, see:

• Sandbox TTI methodology →
• Browser and throughput methodology →
• AI gateway methodology →

Transparency

• 📖 Open source — All benchmark code is public
• 📊 Raw data — Every result committed to repo as JSON
• 🔁 Reproducible — Anyone can run the same tests
• ⚙️ Automated — Daily at 5pm Pacific (00:00 UTC) via GitHub Actions on Namespace runners
• 🛡️ Independent — Sponsors cannot influence results

Latest Benchmarks

Retired benchmarks: The sequential and staggered sandbox TTI tests have been retired. Historic results are still accessible in the results/ directory under sequential_tti/ and staggered_tti/.

Burst TTI

Pricing Comparison

Browser Sessions

Browser Step Throughput

Object Storage

Snapshot & Fork

DAX Sandbox Builds

AI Gateway - Anthropic

AI Gateway - OpenAI

AI Gateway - Gemini

AI Gateway - Kimi

Roadmap

• [x] computesdk.com/benchmarks
• [x] Add P95 & P99
• [x] TTI n=100 test
• [x] TTI n=100 concurrency test (staggered + burst)
• [x] 100,000 concurrent sandbox stress test
• [ ] Cold start vs warm start metrics
• [ ] Multi-region testing
• [x] Cost-per-sandbox-minute

Powered by ComputeSDK — We use ComputeSDK, a multi-provider SDK, to test all sandbox providers with the same code. One API, multiple providers, fair comparison.

MIT License