개요

MCP-Bench는 모델 컨텍스트 프로토콜(MCP)을 통해 도구 사용 시나리오에서 대형 언어 모델(LLM)의 역량을 평가하기 위해 설계된 종합 평가 프레임워크입니다. 이 벤치마크는 다양한 LLM이 실제 과제를 발견, 선택 및 활용하는 도구를 얼마나 효과적으로 발견, 선택, 활용할 수 있는지 평가하는 종단 간 파이프라인을 제공합니다.

뉴스

• [2025-09] MCP-Bench는 NeurIPS 2025 에이전트 환경 확장성 워크숍에 선정되었습니다.

리더보드

MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

Overview

MCP-Bench is a comprehensive evaluation framework designed to assess Large Language Models’ (LLMs) capabilities in tool-use scenarios through the Model Context Protocol (MCP). This benchmark provides an end-to-end pipeline for evaluating how effectively different LLMs can discover, select, and utilize tools to solve real-world tasks.

News

• [2025-09] MCP-Bench is accepted to NeurIPS 2025 Workshop on Scaling Environments for Agents.

Leaderboard

Overall Score represents the average performance across all evaluation dimensions including rule-based schema understanding, LLM-judged (o4-mini as judge model) task completion, tool usage, and planning effectiveness. Scores are averaged across single-server and multi-server settings.

Quick Start

Installation

1. Clone the repository

git clone https://github.com/accenture/mcp-bench.git
cd mcp-bench

2. Install dependencies

conda create -n mcpbench python=3.10
conda activate mcpbench
cd mcp_servers
# Install MCP server dependencies
bash ./install.sh
cd ..

3. Set up environment variables

# Create .env file with API keys
# Default setup uses both OpenRouter and Azure OpenAI
# For Azure OpenAI, you also need to set your API version in file benchmark_config.yaml (line205)
# For OpenRouter-only setup, see "Optional: Using only OpenRouter API" section below
cat > .env << EOF
export OPENROUTER_API_KEY="your_openrouterkey_here"
export AZURE_OPENAI_API_KEY="your_azureopenai_apikey_here"