aa-mcp

MCP server for artificialanalysis.ai — live AI model benchmarks, pricing, speed, context/parameter specs, provider endpoints and the image/video/speech/music arenas. Every benchmark and index AA publishes is covered — the leaderboard suite, the Decision Models, the Multilingual Index, the on-device benchmark set and the eval-chart breakdowns — shaped for agents that pay per token.

No API key needed. One keyless data source: the site’s Next.js RSC flight payloads, scraped and cached locally — no quota, no backend selection, no fallback logic. Every response header shows aa:rsc.

Install

One line, no clone (uv fetches and runs it for you):

uvx --from git+https://github.com/HenryGetz/aa-mcp aa-mcp

Or install the aa-mcp executable once:

uv tool install --from git+https://github.com/HenryGetz/aa-mcp aa-mcp   # -> ~/.local/bin/aa-mcp

Local checkout / development:

git clone  && cd aa-mcp
uv sync                      # creates .venv with the mcp SDK
uv run aa-mcp                # serve on stdio

Register with an MCP client (omp, Claude Code, Codex, …):

{
  "mcpServers": {
    "aa": {
      "type": "stdio",
      "command": "/home/USER/.local/bin/aa-mcp",
      "timeout": 120000
    }
  }
}

Skills and MCP install

The JSON above installs the MCP server (callable tools, exposed as mcp__aa_). A skill is a different thing — markdown instructions an agent reads on demand via skill://, installed by copying a directory (or npx skills add …) into a skills root such as ~/.agents/skills/ or /.agents/skills/; it adds no tools, and omp only notices it after a restart. This repo ships one at skills/aa-mcp/SKILL.md.

Full commands, paths and troubleshooting for both: docs/SKILLS.md.

No MCP client? The same tools are reachable from a plain shell as aa-mcp-cli (same install as aa-mcp; python -m aa_mcp.cli from a checkout) — e.g. aa-mcp-cli call aa_leaderboard '{"top":3}' prints exactly what the MCP tool returns.

Tools

Resources (15): aa://reference/metrics (full definitions), one per topic (overview, indexes, benchmarks, pricing, speed, providers, media, fields, sources), aa://benchmarks (registry JSON), aa://rankables and aa://rankables.yaml (every rankable thing, for building your own queries), aa://comparisons/spec (a aa_compare spec= template) and aa://sources. Prompts: best_model_for_task, compare_models.

Comparing models

Everything AA’s own comparison pages do, without the pages:

# same models as artificialanalysis.ai/models/comparisons?compare=a,b,c,d
aa_compare '{"queries":["mimo-v2-6-pro","deepseek-v4-1-flash","gpt-6-luna-xhigh","muse-wonder-full"]}'

# every benchmark AA has for those models, with deltas vs the best
aa_compare '{"queries":[...],"metrics":"all","view":"diff"}'

# reproducible / reusable comparison (YAML or JSON); the output echoes the spec back
aa_compare '{"spec":"models: [mimo-v2-6-pro, kimi-k3]\nmetrics: [intelligence, cost_per_task, gpqa, terminalBench40]\nview: diff"}'

Trade-off questions always have one call:

aa_pareto '{}'                             # Intelligence vs $/task frontier + knee
aa_pareto '{"x":"index_total_cost"}'       # vs cost of the whole index run
aa_pareto '{"x":"out_tokens_per_task"}'    # vs verbosity
aa_pareto '{"score":"cyberIndex","x":"cost_per_task"}'

Finding the right benchmark for a domain, without knowing its name:

aa_benchmarks '{"domain":"security"}'   # -> CWE-Bench, CyberGym, Cyber Index, …
aa_benchmarks '{"query":"speech"}'      # -> STT/TTS/STS boards + the exact call to make

Token discipline

• Tables, not JSON — rounded values, one header line, no argument echo.
• Default 10-15 rows; every table tool accepts top and fields.
• Long-form documentation lives in resources/reference, so it never costs context unless the agent asks for it.
• aa_raw warns when used.

Measured live (12-call panel, one call per tool): ≈3.3k tokens total, ≈277 per call, e.g. a 5-row leaderboard is ~180 tokens, a full model digest ~470, a media board ~160, the aa_benchmarks search index ~235, aa_reference topic=overview ~915 (the one call that is deliberately long-form).

Example — aa_leaderboard {"mode":"value","top":5} (verbatim):

# 5 of 114 scored by value (703 tracked) · value = Intel per $/task, active only · 34m ago · aa:rsc
| Model | Slug | Creator | Intel | $/Task | tok/s | TTFT | Ctx | Intel/$ |
|---|---|---|---|---|---|---|---|---|
| GPT-6 Luna (Low) | gpt-6-luna-low | OpenAI | 21.5 | $0.0045 | 117.9 | 2.2s | 1.0M | 4766.0 |
| GPT-6 Luna (Medium) | gpt-6-luna-medium | OpenAI | 29.9 | $0.02 | - | - | 1.0M | 1712.3 |
| GPT-6 Luna (Non-reasoning) | gpt-6-luna-non-reasoning | OpenAI | 18.5 | $0.01 | 113.0 | 0.9s | 1.0M | 1627.8 |
| Granite 4.2 3B | granite-4-2-3b | IBM | 9.1 | $0.0060 | 219.0 | 0.5s | 131k | 1512.5 |
| Claude Haiku 5.5 (Low) | claude-haiku-5-5-low | Anthropic | 29.5 | $0.02 | 186.5 | 6.3s | 1.0M | 1204.1 |
— 508 rows skipped: no measured cost per task (AA has cost per task for 195/703 variants)

Coverage and limits

Covered: 703 model variants × ~50 leaderboard metrics merged with a 703 × 94 dataset (capability indexes, AA-Briefcase/GDPval Elo, AIME, LiveCodeBench, GDP.pdf, AutomationBench, Harvey LAB, MLCR, cost breakdowns, per-eval costs, licenses, effort labels); per-model digests with prompt-type speed breakdowns; 1,222 provider endpoints across 64 hosts with prices, quantization, latency and feature flags; 11 media arenas (the nine image/video/TTS/music boards plus the two silent video boards) each with per-category sub-boards; the full speech boards (speech-to-text 36 rows, streaming 45, speech-to-speech 27 — plus the agentic/non-agentic slices as their own rankables) with per-dataset WER, streaming latency, STS components and the TTS pronunciation/throughput suite; AA’s coding-agent, chatbot and search-API boards, with the Search API’s BrowseComp / DeepSearchQA / latency / $-per-query metrics; the Decision-Models suite, the Multilingual Index (16 languages), the on-device/mobile benchmark set (7 evals incl. BFCL-500/BFCL-Small) and the eval-chart breakdown sub-scores; a 223-entry rankable index (136 benchmarks plus indexes, derived metrics, arenas and views) with descriptions, units, live coverage and the exact tool call for each.

Counts were measured live on 2026-10-16 and drift as AA adds or retires models.

Deliberately out of scope (and the server says so instead of guessing):

• Benchmarks AA does not track (MathVista, DocVQA) — no data exists. BFCL is tracked only on the on-device/mobile set (mobile.bfcl_500, mobile.bfcl_small), not as a server-grade leaderboard.
• Cross-model signature leaderboards for benchmarks AA only exposes as a chart subset: the tool labels these not a full leaderboard and reports real coverage.
• Model-level Coding Index and Agentic Index: AA publishes these only through its paid data API (no public page, route or payload key exists), so the server does not provide them — use the constituent benchmarks (SciCode, Terminal-Bench 4.0, LiveCodeBench, FrontierCode, τ²-Bench/τ³-Banking, AutomationBench).
• Performance-over-time history and raw per-provider measurements are Commercial-tier API features; not implemented.
• Surfaces AA publishes no cross-model ranking for: HLE Diamond (model pages only) and MicroEvals (11,228 user-voted prompt responses, not an AA-run benchmark).
• Image/video/music model detail pages (per-family specs beyond arena rows) and the laptops/workstations and datacenter hardware pages are not exposed (the mobile on-device benchmark set is — see below).

Indexes

Free, keyless and scraped like everything else:

Not available: the model-level Coding Index and Agentic Index. AA publishes them only through its paid data API — no public page, route, chunk string or payload key exists — so this server does not provide them. The free substitutes are the constituent benchmarks (SciCode, Terminal-Bench 4.0, LiveCodeBench, FrontierCode, τ²-Bench/τ³-Banking, AutomationBench), which aa_benchmark_leaderboard ranks.

Benchmark suites

Beyond the classic leaderboard benchmarks, the server reads every published benchmark AA ships on its own eval pages and chart sets (counts measured 2026-10-16):

• Decision Models — aa_benchmark_leaderboard benchmark=when2call, wildguardtest, hleMultichoice, omniscienceDecision (and omniscienceDecisionNonHallucination). AA’s four Decision-Models eval pages each publish a 28-row array; the four pages share one 28-row score blob with five keys (when2callAccuracy, wildguardtestAverageF1, hleTextMcqAccuracy, omniscienceAccuracy, omniscienceNonHallucinationRate), so each model can be ranked on all four. Rows include decisionModel entries that are not in the 703-row leaderboard.
• Multilingual Index — aa_leaderboard sort=multilingual ranks the average; sort=multilingual. ranks one of the 16 languages (ar, bn, de, en, es, fr, hi, id, it, ja, ko, pt, sw, yo, zh, my) on /models/multilingual (13 chart-selected models). The same page carries globalMmluLite, which unions with the /evaluations/global-mmlu-lite page to 17 scored models.
• On-device / mobile benchmark set — aa_benchmark_leaderboard benchmark=mobile. (aa_leaderboard sort=mobile.) over the 90 rows × 7 evals on /hardware-inference-stack/mobile-phones: mobile.gpqa (57 scored), mobile.ifbench (55), mobile.ifstruct (11), mobile.math_500 (57), mobile.omniscience (58), mobile.bfcl_500 (17) and mobile.bfcl_small (57). This is where AA publishes BFCL — the old “AA does not track BFCL” claim was wrong.
• Silent video arenas — aa_media board=text_to_video_silent (31 rows) and board=image_to_video_silent (82 rows). These are separate boards from the audio-capable text_to_video (28) / image_to_video (23); the silent I2V board is the largest of the four.
• Media category sub-boards — aa_media board= category= ranks one of a board’s per-category subsets (e.g. text_to_image ships 19, image_editing 17, text_to_video 20, video_editing 5, text_to_speech_controlled 11). Omit category= for the overall board.
• Breakdown sub-scores — aa_model sections=benchmarks renders the per-eval breakdowns AA publishes on its chart sets (AA-Briefcase rubric pass rate / analytical-quality / presentation Elo, Harvey criteria pass, GDP.pdf macro pass, AutomationBench strict/completion, GDPval/ITBench/EnterpriseOps average turns, AA-3D median turns). The multi-model ones are also rankable sub-metrics: the 9 Openness components (openness.*), the 6 Omniscience domains (omniscience.domain.*) and 14 SWE languages (omniscience.language.*), the 5 Terminal-Bench-Science domains and 8 EnterpriseOps-Gym domains.
• Speech, voice & search (all rankable through aa_leaderboard sort= / aa_benchmark_leaderboard benchmark=, counts measured 2026-10-16):
  • Speech-to-text — aa_media board=speech_to_text serves the full /speech-to-text/non-streaming board (36 rows, was the 15-row homepage panel). Metrics: sttWerIndex (36), sttWordErrorRate (11), sttEarnings22Wer, sttVoxpopuliWer, sttAgenttalkWer (36 each; lower is better).
  • Streaming speech-to-text — board=speech_to_text_streaming, the full /speech-to-text/streaming board (45 rows): sttStreamingWerIndex, sttStreamingEarnings22Wer, sttStreamingAgenttalkWer (45 each, lower is better) plus sttStreamingTimeToFirstPartial and sttStreamingTimeToFinal (seconds, lower is better).
  • Speech-to-speech — board=speech_to_speech serves the arena’s overall slice (27 rows; the page’s 80 rows are the overall + agentic + non-agentic slices). Metrics: sts index (19), stsOverallElo/stsAgenticElo/ stsNonAgenticElo (27/27/26), stsTaskCompletionRate (27), stsBigBenchAudio/stsTauVoice/stsArenaScore/stsTaskSuccess (19 each).
  • TTS — aa_media board=text_to_speech category= ranks AA’s use-case sub-boards (assistants 90, knowledge-sharing 97, entertainment 90, customer-service 92; inline language boards en-us 96, en-gb 83). Metrics: ttsQualityElo (97), ttsCharactersPerSecond (79, chars/s), ttsGenerationTime (79 s, lower is better), and the pronunciation suite ttsPronunciationPassRate / ttsTermPronunciation / ttsTextNormalization / ttsSequenceFidelity / ttsContextualDisambiguation (30 each).
  • Search API — aa_agents board=search_api plus the rankable searchQualityIndex, searchBrowsecompAccuracy, searchDeepsearchqaF1, searchOmniscienceAccuracy, searchAvgLatency (s, lower is better), searchCostPerQuery (USD/query, lower is better), searchCostPerTask, searchQueriesPerTask and searchQualityLift (26 scored each, 25 for the cost/queries columns).

Environment

Behaviour worth knowing

• Freshness is labelled. Payloads are cached for an hour; the header shows the age and says stale when a network failure forced a cached copy. Pass refresh=true for freshly released models.
• Coverage is labelled. AA scores a benchmark on a subset of models; tables state how many of the 703 tracked variants carry a value, and warn when a filter dropped rows that simply have no measurement.
• Cost per task is cache-aware and must not be recomputed from $/1M rates; raw rates are separate fields.
• Effort variants (-low … -xhigh) are separate rows — always name the effort.
• Replies are bounded and cells are neutralized. Scraped text is sanitized (markdown metacharacters escaped, 120 chars per cell) and any reply is capped at 200 kB, so one hostile page cannot blow up your context; oversized tables say how many rows were kept.
• Speech/arena board rows are their own namespace. A TTS voice or an ASR host is not a language model: those rows publish no Intelligence-Index cost or speed, so aa_pareto score= reports that no frontier can be drawn and names the board’s row count. aa_search resolves board entries (tagged aa:arena with their board) and aa_compare compares board rows within one board — the board’s own columns and its sub-metrics (metrics=stsOverallElo) both work, and the footer says the rows are not language models.
• Hostile arguments are rejected up front, not passed to a fetch or a cache path: control characters / lone surrogates, over-long strings (>256 chars, 2000 for fields) and non-finite numbers (NaN/Infinity) return a ValueError naming the argument, and error text never echoes more than 60 chars of your input.
• AA does not track MathVista or DocVQA; BFCL is tracked only on the on-device/mobile set (mobile.bfcl_500, mobile.bfcl_small). The server says so instead of inventing.

Troubleshooting

• Cloudflare / bot challenge (403/503 or a tiny body): the error names the exact cache path. Render the URL in a browser, save the HTML there, re-run — or set AA_OFFLINE=1 to keep working from the last good capture.
• Site structure change: the error says which array was missing; the parsers live in src/aa_mcp/loaders.py (flight.py finds the arrays).
• AA redeploys often — never hardcode chunk or build-id URLs; the client always re-parses freshly fetched payloads.
• Cache size: payloads are stored verbatim (a rich model page is ~3 MB, so the cache settles around 50-120 MB). Clear it with python -c "from aa_mcp.fetch import clear_cache; print(clear_cache())" — worst case the next call refetches.

Development

uv run pytest -q                 # offline: fixtures in tests/fixtures (real payloads)
uvx ruff check .                 # lint (CI runs this)
.venv/bin/python eval/client.py tools
.venv/bin/python eval/client.py call aa_leaderboard '{"top":5}'
.venv/bin/python eval/client.py bench                 # per-call chars/tokens report
.venv/bin/python scripts/surface_sweep.py             # every tool × every documented enum, live
.venv/bin/python scripts/gen_reference_md.py --check  # docs/REFERENCE.md in sync

scripts/surface_sweep.py is the end-to-end guard: it drives the real stdio server through ~220 calls covering every tool, board, sort key, axis, section, topic and raw kind, and fails if anything errors unexpectedly (it is how the speech-board NameError was caught).

docs/REFERENCE.md is generated from src/aa_mcp/reference.py (scripts/gen_reference_md.py); docs/INTERFACES.md documents the internal data model. eval/ holds the evaluation harness and every audit report: the independent field-by-field parser audit (eval/reports/fidelity.md), the code review (eval/reports/audit_quality.md), the adversarial attack report (eval/reports/adversarial.md) and five rounds of agent usage reports (eval/runs/round*.md).