modelscope/evalscope

中文 &nbsp ｜ &nbsp English &nbsp

📖 中文文档 &nbsp ｜ &nbsp 📖 English Documentation &nbsp ｜ &nbsp 🌐 Product Website

⭐ If you like this project, please click the “Star” button in the upper right corner to support us. Your support is our motivation to move forward!

📝 Introduction

EvalScope is a one-stop LLM evaluation framework built by the ModelScope Community. Just one command to start — it supports model capability evaluation, inference performance stress testing, and result visualization.

pip install evalscope
evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5

✨ Key Features

• 📚 Comprehensive Evaluation Benchmarks: Built-in multiple industry-recognized evaluation benchmarks including MMLU, C-Eval, GSM8K, and more.
• 🧩 Multi-modal and Multi-domain Support: Supports evaluation of various model types including Large Language Models (LLM), Vision Language Models (VLM), Embedding, Reranker, AIGC, and more.
• 🧭 System One Evaluation: Evaluate text Choice decision models via --eval-type systemone_api, with task instructions, few-shot examples and structured probabilities/confidence. See the usage guide.
• 🚀 Multi-backend Integration: Seamlessly integrates multiple evaluation backends including OpenCompass, VLMEvalKit, RAGEval to meet different evaluation needs.
• 🤖 Agent Evaluation Mode: Drives benchmarks (e.g. GSM8K, AIME, SWE-bench Agentic) inside a controlled multi-turn AgentLoop with pluggable strategies, tools and Docker sandbox; full per-sample Agent Trace is recorded and visualizable.
• ⚡ Inference Performance Testing: Provides powerful model service stress testing tools, supporting multiple performance metrics such as TTFT, TPOT.
• 📊 Interactive Reports: Provides a Web Dashboard for multi-dimensional model comparison, report overview and detailed inspection.
• ⚔️ Arena Mode: Supports multi-model battles (Pairwise Battle), intuitively ranking and evaluating models.
• 🔧 Highly Extensible: Developers can easily add custom datasets, models and evaluation metrics.

📊 Visualization Preview

EvalScope provides an interactive Web Dashboard for multi-dimensional model comparison and in-depth analysis.

For details, please refer to 📖 Visualizing Evaluation Results.

🎉 What’s New

• 🔥 [2026.10.09] Added System One Choice evaluation and ANLI, BoolQ, BANKING77, ContractNLI, RewardBench v1 benchmarks. All five default to ModelScope and also support standard LLM evaluation. See System One Evaluation.
• 🔥 [2026.08.24] v1.11.0 Introduced published evaluation versions for reproducible benchmark results; improved report semantics and incomplete-run handling; strengthened multimodal media loading and task-config validation.
• 🔥 [2026.08.13] Improved evaluation reports with unified metric semantics and more reliable Agent Trace step grouping and tool-call/result linking.
• 🔥 [2026.08.10] Added AutomationBench, JobBench, MiniWoB, OmniDocBench-v1.6, PerceptionBench, ScreenSpot-Pro, PLawBench, PMC-VQA, HiPhO, LogicVista, and CC-OCR-V2 benchmarks.
• 🔥 [2026.07.21] Added Claw-Eval, ResearchRubrics, Toolathlon (agent), TVBench (video), WideSearch, and PerspectiveGap benchmarks.
• 🔥 [2026.07.03] Added CharXiv & BabyVision (chart understanding, visual cognition) and ERQA & WorldVQA (entity-recognition QA with LLM-judge + CoT) multimodal benchmarks.
• 🔥 [2026.06.23] Major agent & code evaluation expansion: added BigCodeBench, SWE-bench Multilingual, BrowseComp, MCP-Atlas, GDPval benchmarks; added OpenCode / OpenHands runners; refactored adapter architecture with AudioLanguageAdapter, unified FunctionCallAdapter, and public run_agent_loop API.
• 🔥 [2026.06.16] Added full-reference image quality metrics (SSIM, PSNR, etc.), long-context benchmarks (LoCoMo QA, LongMemEval), Caption & Maritime-OCR-Bench benchmarks; perf module now supports unified --data-source and parallelized request generation.
• 🔥 [2026.06.02] Refactored RAG evaluation module: upgraded to MTEB 2.x and RAGAS 0.4.x, with unified Pydantic-based configs. See the RAGEval guide.
• 🔥 [2026.05.27] Added Trie agentic trace replay for perf benchmarking: three new dataset plugins (trie_agentic_coding / trie_code_qa / trie_office_work) replay real multi-turn agent traces with per-turn token caps and tool-call latency simulation. Also introduced a --duration wall-clock budget for all benchmark modes and a Turn dataclass for per-turn overrides.
• 🔥 [2026.05.27] Added Vendor Verifier benchmarks (k2_verifier, kimi_verifier, minimax_verifier) for validating whether third-party API deployments faithfully reproduce official model behavior, with a shared FunctionCallAdapter base class.
• 🔥 [2026.05.26] Added the GAIA agent benchmark (multi-turn ReAct + bash in a Docker sandbox, official rule-based scorer) and generic MCP server support — any NativeAgentConfig-driven benchmark can now plug in stdio / HTTP / SSE MCP servers (fetch, web search, GitHub, …) without per-benchmark wiring.
• 🔥 [2026.05.22] Introduced the External Agent Bridge mode: evaluate off-the-shelf agent CLIs such as Anthropic’s Claude Code and OpenAI’s Codex directly through EvalScope. The bridge transparently forwards each CLI’s LLM traffic (Anthropic Messages / OpenAI Chat / OpenAI Responses, including SSE streaming) to your evaluation model, while recording the full trajectory as an agent_trace. Bring-your-own-runner via @register_runner. See the External Agent Bridge guide.
• 🔥 [2026.05.19] Added support for SWE-bench_Pro and τ³-bench: SWE-bench_Pro is a more challenging multilingual long-horizon software-engineering benchmark from Scale AI (recommended over the original SWE-bench for less data contamination and broader language coverage; per-instance Docker images are pulled directly from DockerHub, no local image build required); τ³-bench is the v1.0.0 release of the tau-bench family, extending τ²-bench with a new banking_knowledge retrieval domain (RAG), 75+ task fixes across existing domains, and pluggable retrieval pipelines (BM25 / embeddings / rerankers / sandbox shell).

🚀 Quick Start

Installation

pip install evalscope

For detailed installation instructions (source install, extra dependencies, etc.), please refer to the 📖 Installation Guide.

Method 1. Evaluate an Online Model API (Recommended for beginners, no GPU required)

Supports any OpenAI API-compatible model service. Just set $OPENAI_API_BASE_URL and $OPENAI_API_KEY and you are ready to go:

evalscope eval \
 --model your-model-name \
 --api-url $OPENAI_API_BASE_URL \
 --api-key $OPENAI_API_KEY \
 --eval-type openai_api \
 --datasets gsm8k arc \
 --limit 5

Method 2. Evaluate a Local Model

Evaluate a local model (auto-downloaded from ModelScope):

evalscope eval \
 --model Qwen/Qwen2.5-0.5B-Instruct \
 --datasets gsm8k arc \
 --limit 5

Method 3. Using Python Code

from evalscope import run_task, TaskConfig

task_cfg = TaskConfig(
    model='your-model-name',
    api_url='https://your-openai-compatible-endpoint/v1',
    api_key='your_api_key',
    eval_type='openai_api',
    datasets=['gsm8k', 'arc'],
    limit=5
)

run_task(task_cfg)

Output Results

After evaluation completion, you will see a report in the terminal in the following format:

┌───────────────────────┬───────────┬────────────┬───────────────┬───────┬─────────┐
│ Model                 │ Dataset   │ Metric     │ Subset        │   Num │ Score   │
├───────────────────────┼───────────┼────────────┼───────────────┼───────┼─────────┤
│ Qwen2.5-0.5B-Instruct │ gsm8k     │ Accuracy ↑ │ main          │     5 │ 40%     │
├───────────────────────┼───────────┼────────────┼───────────────┼───────┼─────────┤
│ Qwen2.5-0.5B-Instruct │ arc       │ Accuracy ↑ │ ARC-Easy      │     5 │ 80%     │
├───────────────────────┼───────────┼────────────┼───────────────┼───────┼─────────┤
│ Qwen2.5-0.5B-Instruct │ arc       │ Accuracy ↑ │ ARC-Challenge │     5 │ 40%     │
└───────────────────────┴───────────┴────────────┴───────────────┴───────┴─────────┘

Launch the visualization dashboard:

pip install 'evalscope[service]'
evalscope service

Visit http://127.0.0.1:9000 to open the visualization interface.

📈 Advanced Usage

Custom Evaluation Parameters

You can fine-tune model loading, inference, and dataset configuration through command line parameters.

evalscope eval \
 --model Qwen/Qwen3-0.6B \
 --model-args '{"revision": "master", "precision": "torch.float16", "device_map": "auto"}' \
 --generation-config '{"do_sample":true,"temperature":0.6,"max_tokens":512}' \
 --dataset-args '{"gsm8k": {"few_shot_num": 0, "few_shot_random": false}}' \
 --datasets gsm8k \
 --limit 10

• --model-args: Model loading parameters such as revision, precision, etc.
• --generation-config: Model generation parameters such as temperature, max_tokens, etc.
• --dataset-args: Dataset configuration parameters such as few_shot_num, etc.

For details, please refer to 📖 Complete Parameter Guide.

⚔️ Arena Mode

Arena mode evaluates model performance through pairwise battles between models, providing win rates and rankings, perfect for horizontal comparison of multiple models.

# Example evaluation results
Model           WinRate (%)  CI (%)
------------  -------------  ---------------
qwen2.5-72b            69.3  (-13.3 / +12.2)
qwen2.5-7b             50    (+0.0 / +0.0)
qwen2.5-0.5b            4.7  (-2.5 / +4.4)

For details, please refer to 📖 Arena Mode Usage Guide.

🖊️ Custom Dataset Evaluation

EvalScope allows you to easily add and evaluate your own datasets. For details, please refer to 📖 Custom Dataset Evaluation Guide.

⚡ Inference Performance Evaluation Tool

EvalScope provides a powerful stress testing tool for evaluating the performance of large language model services.

• Key Metrics: Supports throughput (Tokens/s), first token latency (TTFT), token generation latency (TPOT), etc.
• Result Recording: Supports recording results to wandb and swanlab.
• Speed Benchmarks: Can generate speed benchmark results similar to official reports.

For details, please refer to 📖 Performance Testing Usage Guide.

🧪 Other Evaluation Backends

EvalScope supports launching evaluation tasks through third-party evaluation frameworks (we call them “backends”) to meet diverse evaluation needs.

• Native: EvalScope’s default evaluation framework with comprehensive functionality.
• OpenCompass: Focuses on text-only evaluation. 📖 Usage Guide
• VLMEvalKit: Focuses on multi-modal evaluation. 📖 Usage Guide
• RAGEval: Focuses on RAG evaluation, supporting Embedding and Reranker models. 📖 Usage Guide
• Third-party Evaluation Tools: Supports evaluation tasks like ToolBench.

❤️ Community & Support

Welcome to join our community to communicate with other developers and get help.

👷‍♂️ Contributing

We welcome any contributions from the community! If you want to add new evaluation benchmarks, models, or features, please refer to our Contributing Guide.

Thanks to all developers who have contributed to EvalScope!

📚 Citation

If you use EvalScope in your research, please cite our work:

@misc{evalscope_2024,
    title={{EvalScope}: Evaluation Framework for Large Models},
    author={ModelScope Team},
    year={2024},
    url={https://github.com/modelscope/evalscope}
}