Computer Use MCP

Give an AI agent the ability to discover apps, understand their controls, and work on a desktop. Computer Use MCP connects MCP-compatible agents to native tools on macOS, Windows and Linux.

Use app scripting and accessibility controls when available, and screenshots, mouse and keyboard input when the interface requires them.

See it paint

Astra painted this through real mouse strokes. The agent used the OpenAI Responses API to operate the bundled paint studio through this MCP: 105 strokes, seven colors and 15 model calls, followed by a verified PNG export. The result is an original Van Gogh-inspired landscape made through native pointer events.

# From a clone of this repository, after the setup below:
npm install --prefix agents/openai-agent
npx playwright install chromium
export OPENAI_API_KEY=your-key
node agents/openai-agent/showcase.mjs paint --studio --turns 24 --tokens 200000

This makes paid API calls. The recorded run used 191,734 input and 3,274 output tokens. Run the demos and read their limits.

See it read

Thirty-eight seconds, ten fields, no answer key. The invoice on the left is drawn on a canvas, so its text never reaches the accessibility tree — the only way to read it is to look. DeepSeek Flash reads it from screenshots, types each value through the accessibility API, and presses Verify; the local host scores the entries and tells it only which ones are wrong. This run scored 10 of 10 on the first Verify, over 12 model calls and 23 tool calls, at a 94% prompt-cache hit rate. Every entry is recorded with its isTrusted flag, so a submission that was not typed through the UI is refused.

npm install --prefix agents/deepseek-agent
npx playwright install chromium
export DEEPSEEK_API_KEY=your-key
node agents/deepseek-agent/showcase.mjs --seed 7             # the run above
node agents/deepseek-agent/showcase.mjs --receipt random     # a real receipt from the web

Full-resolution capture · how it works, and what it cannot fake

See it plan

Watch the work instead of reading tool calls. The run console is an MCP App the agent reports into. It declares a plan, marks one task active at a time, and attaches real screen captures as it goes — the frame above is a live capture of Blender’s own window.

Underneath the conversation is everything it is doing: each thought, each tool call and each result, with timings. That feed is reported by the host driving the run rather than by the model, so it is complete — Blender’s own calls appear in it too.

You can type while it works. The message lands in the transcript that every run tool returns, so the agent reads it on its next call and can change course mid-task. Attach an image and it gets both the picture and the path the file was saved at, which is how Blender can open your reference as a texture rather than the agent working from its own description of it. web_search and scrape let it check a fact it does not know — both run locally, so they work with any model.

Look at it right now, with no API key:

npx -y -p @zavora-ai/computer-use-mcp computer-use-mcp-console

That opens the console and drives a scripted walkthrough — the words are written into the host, but everything it looks at is real, so the frame on the right is a capture of your actual screen. For a live agent instead, start the host with --no-demo and point one at http://127.0.0.1:4517/mcp:

computer-use-mcp-console --no-demo

That endpoint serves the desktop tools and the run tools from the same server, so whatever the agent plans and captures appears in the browser. The screenshot above is an ADK-Rust agent on DeepSeek Flash refining a watch it modelled earlier in the same conversation.

The console is loopback-only and unauthenticated: /mcp grants desktop control to anything that can reach the port. Fine on your own machine, never on a shared host.

Set up your agent

Requires Node.js 20+ and an interactive desktop. The published package bundles native modules for macOS, Windows and Linux, on x64 and arm64.

Add this to your agent’s MCP server configuration:

{
  "mcpServers": {
    "computer-use": {
      "command": "npx",
      "args": ["-y", "@zavora-ai/computer-use-mcp"]
    }
  }
}

For a client with a different configuration format, use npx as the command and -y @zavora-ai/computer-use-mcp as its arguments. See agent configuration for Cursor, Claude Desktop, Codex and programmatic examples. No model API key is needed by the MCP server itself.

Before using desktop tools:

Ask your agent to run doctor to check capabilities. The computer-use-mcp command checks its own install first (dependencies, build, native module) and names the fix on stderr if something is missing. Office must be installed and activated separately; its folder-access prompts are normal setup requirements. Downloads is not guaranteed to bypass those prompts.

What it can do

The default profile, desktop, exposes 40 tools on macOS: observation, coordinates, OCR for apps that draw their own UI, waits and the session target. Set COMPUTER_USE_PROFILE=full for every tool (76 on macOS), or core, ax, scripting and windows-admin for other bounds. Tools that don’t exist on your platform are not listed. Optional host services add their own tools. The server’s instructions to the agent are generated for your platform and profile, so they only name tools that are there.

Try asking your agent:

• “Find my installed Office apps and explain how you can automate them.”
• “Open a new text document, write a short meeting agenda and save it.”
• “Inspect this application’s controls and fill out the form I describe.”
• “Create a new painting with a swirling night sky and export it.”

Always specify the intended app or window. Prefer connectors, APIs or filesystem operations for work that does not need a desktop. Platform and application support varies; an available tool is not a promise that every app exposes usable controls.

Use it from code

import { createComputerUseServer } from '@zavora-ai/computer-use-mcp'
import { connectInProcess } from '@zavora-ai/computer-use-mcp/client'

// The default profile is `desktop`; pass { profile: 'full' } for every tool.
const client = await connectInProcess(createComputerUseServer())
try {
  const apps = await client.discoverApplications({
    query: 'office',
    include_capabilities: true,
  })
  console.log(apps.content)
} finally {
  await client.close()
}

Application examples · AI agent examples · Responses showcases

Architecture

flowchart TD
  Agent[AI agent or MCP client] --> Server[MCP server]
  Server --> Policy[Tool permissions and target checks]
  Policy --> Session[Desktop session]
  Session --> Native[Native mouse, keyboard and screen tools]
  Session --> AX[Accessibility controls]
  Session --> Scripts[App scripting]
  Native --> Apps[Desktop applications]
  AX --> Apps
  Scripts --> Apps
  Server --> Hosts[Optional host services]
  Hosts --> Browser[Isolated browser]
  Hosts --> Tasks[Persistent tasks and session console]

The TypeScript server handles tool requests, permissions and cancellation. A Rust native module connects to the operating system. Optional host services provide longer-lived sessions and isolated execution. Results return to the agent so it can observe, act and verify.

For protocol details, deployment controls and the full component diagram, see Architecture and the host integration guide. Legacy MCP clients remain supported; newer protocol features are opt-in.

Develop locally

git clone https://github.com/zavora-ai/computer-use-mcp.git
cd computer-use-mcp
npm ci
npm run build:ts
npm test

For live desktop use from a clone, install Rust and build your platform’s native module: npm run build:native on macOS, npm run build:native:win on Windows x64, or npm run build:native:linux on Linux. Windows arm64 has build:native:win:arm64. The packaged npm install does not require a local Rust build.

npm run smoke checks the local desktop; npm run test:browser tests a disposable browser fixture. These live checks need their documented platform prerequisites. Development scripts lists the maintained commands.

Documentation and support

• v7.6.1 release notes: the review fixes and the desktop status card
• v7.6.0 release notes (includes 7.5.0: its notes)
• v7.4.0 release notes and rollout checklist
• Reviews: the 2026-10-09 review and design pass behind 7.6.0, and the 2026-10-11 three-pass review behind 7.6.1
• Tool usage and agent setup
• Architecture and platform support
• Sessions, Tasks, browser isolation and supervision
• Token and interaction efficiency
• Security and reporting vulnerabilities
• Report a bug or request a feature

MIT licensed. Desktop access can change real applications and files; configure permissions and target applications deliberately. The bundled HTTP server is loopback-only. Exposing a remote endpoint requires host-owned authentication.