概述

MCP-Bench 是一个综合评估框架，旨在通过模型上下文协议（MCP）评估大型语言模型（LLM）在工具使用场景中的能力。该基准测试提供了一个端到端的流程，用于评估不同 LLM 在解决现实任务中发现、选择和使用工具的效率。

新闻

• [2025-09] MCP-Bench 已被纳入 NeurIPS 2025 代理扩展环境研讨会。

排行榜

MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

Overview

MCP-Bench is a comprehensive evaluation framework designed to assess Large Language Models’ (LLMs) capabilities in tool-use scenarios through the Model Context Protocol (MCP). This benchmark provides an end-to-end pipeline for evaluating how effectively different LLMs can discover, select, and utilize tools to solve real-world tasks.

News

• [2025-09] MCP-Bench is accepted to NeurIPS 2025 Workshop on Scaling Environments for Agents.

Leaderboard

Overall Score represents the average performance across all evaluation dimensions including rule-based schema understanding, LLM-judged (o4-mini as judge model) task completion, tool usage, and planning effectiveness. Scores are averaged across single-server and multi-server settings.

Quick Start

Installation

1. Clone the repository

git clone https://github.com/accenture/mcp-bench.git
cd mcp-bench

2. Install dependencies

conda create -n mcpbench python=3.10
conda activate mcpbench
cd mcp_servers
# Install MCP server dependencies
bash ./install.sh
cd ..

3. Set up environment variables

# Create .env file with API keys
# Default setup uses both OpenRouter and Azure OpenAI
# For Azure OpenAI, you also need to set your API version in file benchmark_config.yaml (line205)
# For OpenRouter-only setup, see "Optional: Using only OpenRouter API" section below
cat > .env << EOF
export OPENROUTER_API_KEY="your_openrouterkey_here"
export AZURE_OPENAI_API_KEY="your_azureopenai_apikey_here"