🔔 업데이트

2026-05-13: 관심 있으실 수도 있습니다: ToolCUA: 컴퓨터 사용 에이전트를 위한 최적의 GUI-툴 경로 오케스트레이션을 향하여.

2026-05-09: LibreOffice MCP 도구의 버그 수정. 새로운 모델 결과 추가: Kimi-K2.5, Gemini-3.1-Pro, Claude-4.5-Sonnet, Qwen3.5-Plus.

2026-01-26: OSWorld-MCP가 ICLR 2026에 승인되었습니다! 🎉

2025-10-28: 논문과 프로젝트 페이지를 공개했습니다! 🎉

📄 논문  읽기 |  🌐 프로젝트 페이지 방문

---

📑 개요 및 주요 하이라이트

OSWorld-MCP는 실제 상황에서 컴퓨터 사용 에이전트를 평가하기 위한 포괄적이고 공정한 벤치마크입니다.
이 도구는 모델 컨텍스트 프로토콜(MCP) 도구 호출 기능, 그래픽 사용자 인터페이스(GUI) 운영 기술, 의사결정 성능을 함께 측정합니다.
OSWorld의 확장판으로 설계되어 평가의 현실성, 밸런스, 비교 가능성을 크게 향상시킵니다.

주요 특징 및 발견 사항

• 158개의 검증된 MCP 도구, 7개의 일반적인 애플리케이션**(LibreOffice Writer, Calc, Impress, VS Code, Google Chrome, VLC, OS 유틸리티)을 포함합니다. 그중에는 견고성 테스트를 위한 25개의 방해 도구가 포함됩니다

OSWorld-MCP: Benchmarking MCP Tool Invocation in Computer-Use Agents

🔔 Updates

2026-05-13: You Might also be interested: ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents.

2026-05-09: Fixed bugs in LibreOffice MCP tools. New model results added: Kimi-K2.5, Gemini-3.1-Pro, Claude-4.5-Sonnet, and Qwen3.5-Plus.

2026-01-26: OSWorld-MCP is accepted to ICLR 2026! 🎉

2025-10-28: We released our paper and project page! 🎉

📄 Read the Paper  |  🌐 Visit the Project Page

---

📑 Overview & Key Highlights

OSWorld-MCP is a comprehensive and fair benchmark for evaluating computer-use agents in real-world scenarios.
It jointly measures Model Context Protocol (MCP) tool invocation capabilities, graphical user interface (GUI) operation skills, and decision-making performance.
Designed as an extension of OSWorld, it significantly improves realism, balance, and comparability in evaluation.

Key Features & Findings

• 158 validated MCP tools, spanning 7 common applications (LibreOffice Writer, Calc, Impress, VS Code, Google Chrome, VLC, OS utilities). Among them, 25 distractor tools for robustness testing
• 250 tool-beneficial tasks → 69% of benchmark tasks benefit from MCP tools
• Multi-round tool invocation possible, posing real decision-making challenges
• MCP tools boost model accuracy & efficiency — e.g., OpenAI o3: 8.3% → 17.6% (15 steps)
• Highest observed Tool Invocation Rate (TIR) = 33.3% (Claude-4-Sonnet, 50 steps) → indicating ample room for improvement
• MCP tools improve agent metrics
• Higher tool invocation correlates with higher accuracy
• Combining tools introduces significant challenges

Architecture Overview

Figure: OSWorld-MCP evaluation framework integrating GUI actions and MCP tool invocations.

---

⚙️ Installation & Usage

1️⃣ Preparation: Code Setup

# Clone OSWorld base repo
git clone https://github.com/xlang-ai/OSWorld.git

# Clone OSWorld-MCP
git clone https://github.com/X-PLUG/OSWorld-MCP.git

Integrate OSWorld-MCP files into OSWorld to enable MCP support.

---

2️⃣ Preparation: Docker Environment

1. Copy MCP files into /home inside Docker:

/home/
└── mcp_server/
└── osworld_mcp_client.py

2. Install dependencies:

pip install -r requirements.txt

3. Install Node.js
4. Launch MCP server:

cd mcp_server
bash debug_server.sh

A successful launch opens the local MCP debug UI in your browser.

---

3️⃣ Running Evaluation

Example: Evaluate Claude 4 Sonnet (15 steps):

python run_multienv_e2e.py \
    --api_url  \
    --api_key  \
    --model 'claude-sonnet-4-20250514-thinking' \