hyprcat/mac-cua

mac-cua

The computer use agent that doesn’t take over your computer. An open-source MCP server for macOS that lets AI control desktop apps in the background — without ever touching your mouse or stealing your focus.

Quickstart  ·  Background-First  ·  Tools  ·  Setup  ·  Architecture

---

Branches

mac-cua ships two branches. Pick the one that matches your needs.

Both branches are early-stage software. Contributors are welcome on either branch – see Contributing below.

Use release if you want the base implementation with fewer moving parts.

Use confirmed-delivery-pipeline if you want the latest input delivery fixes, elimination of false-negative verification errors, and per-session isolation.

# Clone and use the stable branch (default)
git clone https://github.com/hyprcat/mac-cua.git
cd mac-cua && git checkout release

# Or use the experimental branch
git checkout confirmed-delivery-pipeline

---

Demo

Task: Search Google for the population of Tokyo, calculate what percentage that is of the world population (8.1 billion), write the result in Notes, then open the Wikipedia page for Tokyo in Safari and find the mayor’s name. Only CUA tools — no other tools allowed.

Result from Claude:

1. Searched Google for “population of Tokyo” — found approximately 14.1–14.2 million residents
2. Calculated the percentage: 14.15 million / 8.1 billion = 0.17% of the world’s population
3. Wrote the result in Notes — created a new note titled “Tokyo Population vs World Population”
4. Opened the Wikipedia page for Tokyo in Safari and found that the leader of Tokyo is Governor Yuriko Koike (Tokyo has a Governor rather than a mayor, since it’s officially the Tokyo Metropolitan Prefecture)

---

Most computer use agents take over your screen. They grab your cursor, steal window focus, and lock you out while they work. You sit and watch.

mac-cua works differently. It sends input events directly to target processes using CGEventPostToPid — a macOS API that delivers clicks, keystrokes, and gestures to a specific app without moving your cursor or activating any window. The AI works in the background. You keep working in the foreground. At the same time. On the same machine.

OpenAI and Perplexity both shipped computer use agents this week — locked to their platforms, behind paywalls. mac-cua is the same capability as an open MCP tool. Plug it into Claude Code, Cursor, Codex, or any MCP client. Free, open source, Apache 2.0.

Background-First

This is the core idea behind mac-cua, and it influences every design decision.

  Traditional computer use agent:            mac-cua:

  +----------------------------------+       +----------------------------------+
  |  YOUR SCREEN                     |       |  YOUR SCREEN                     |
  |                                  |       |                                  |
  |  +----------------------------+  |       |  +----------------------------+  |
  |  |                            |  |       |  |                            |  |
  |  |  [Agent controls this]     |  |       |  |  You're working here.      |  |
  |  |  You're locked out.        |  |       |  |  Writing code, browsing,   |  |
  |  |  Cursor hijacked.          |  |       |  |  whatever you want.        |  |
  |  |  Focus stolen.             |  |       |  |                            |  |
  |  |  Don't touch anything.     |  |       |  |  Your cursor. Your focus.  |  |
  |  |                            |  |       |  |                            |  |
  |  +----------------------------+  |       |  +----------------------------+  |
  |                                  |       |                                  |
  |  Cursor: [Agent's]               |       |  Meanwhile, in the background:   |
  |  Focus:  [Agent's]               |       |  mac-cua clicks, types, scrolls  |
  |  You:    Watching.               |       |  in Safari, Music, Finder...     |
  +----------------------------------+       +----------------------------------+

How it stays invisible

A note on focus

Most operations are fully invisible, but a few macOS APIs have limitations that may cause a brief, momentary focus flash:

• Launching an app — macOS activates apps when they start; mac-cua yields focus back immediately
• Scroll events — some apps require momentary focus to receive scroll input
• Key-window targeting — certain actions need the window to be key window briefly

These flashes are sub-second and mac-cua restores your previous focus automatically. The vast majority of interactions — clicks, typing, value setting, screenshots, tree reads — are completely invisible.

What this means in practice

• You can browse the web while mac-cua fills out a form in another app
• You can write code while mac-cua navigates System Settings to change a preference
• You can be in a video call while mac-cua organizes files in Finder
• The agent never interrupts you. If a conflict arises, you win — mac-cua detects user interruption and backs off

Why mac-cua?

Quickstart

Prerequisites

• macOS 13+ (Ventura or later)
• Python 3.13+
• uv package manager

Install

git clone https://github.com/hyprcat/mac-cua.git
cd mac-cua
uv sync

Run

uv run python main.py

On first launch, macOS will prompt for two permissions:

Grant both, and the MCP server starts on stdio — ready for your AI tool to connect.

Setup Your AI Tool

mac-cua is a standard MCP stdio server. It works with any tool that supports the Model Context Protocol — no plugins, no extensions, just config.

Note: Replace /path/to/mac-cua with the actual path where you cloned the repo.

How It Works

mac-cua reads apps through two complementary channels and acts through background-targeted input:

                        +-----------------------+
                        |      LLM Client       |
                        |  (Claude, GPT, etc.)  |
                        +-----------+-----------+
                                    |
                              MCP (stdio)
                                    |
                        +-----------+-----------+
                        |     mac-cua Server    |
                        +-----------+-----------+
                                    |
                    +---------------+---------------+
                    |                               |
          +---------+---------+           +---------+---------+
          |   Accessibility   |           |    Screenshots    |
          |    API (AXTree)   |           | (ScreenCaptureKit)|
          +---------+---------+           +---------+---------+
                    |                               |
          Structured element              Visual pixel-level
          tree with roles,                context via GPU-
          states, actions                 accelerated window
          (read-only, non-               capture (works even
           intrusive)                     behind other windows)
                    |                               |
                    +---------------+---------------+
                                    |
                        +-----------+-----------+
                        |   Background Input    |
                        |   CGEventPostToPid    |
                        |                       |
                        |  Your cursor: unmoved |
                        |  Your focus: untouched|
                        +-----------------------+

Every tool call returns a fresh snapshot — the accessibility tree and a screenshot together — so the LLM always sees the current state before deciding what to do next.

Tools

9 MCP tools that cover the full range of desktop interaction — all operating in the background.

Discovery

Interaction

Reliability Hierarchy

When multiple tools could accomplish the same thing, prefer them in this order:

  Most reliable                                          Least reliable
  +-------------------+------------------+-----------------+------------------+
  | AX secondary      | set_value        | click by        | click by         |
  | action            |                  | element         | coordinates      |
  +-------------------+------------------+-----------------+------------------+

Example Workflow

Here’s what a typical interaction looks like. Notice: every step happens in the background.

# 1. Discover what's running
list_apps()
# => Safari (running), Music (running), Finder (running), ...

# 2. Get the current state of Safari (screenshot + AX tree)
get_app_state(app="Safari")
# => You don't even see Safari activate. mac-cua reads it silently.

# 3. Click the URL bar (element 12 from the tree)
click(app="Safari", element_index="12")
# => Click delivered to Safari's process. Your cursor didn't move.

# 4. Set the URL
set_value(app="Safari", element_index="12", value="https://example.com")
# => Value set directly via AX API. No typing animation. No focus change.

# 5. Press Enter
press_key(app="Safari", key="Return")
# => Key event sent to Safari. You didn't feel a thing.

# 6. Verify it worked
get_app_state(app="Safari")
# => Fresh screenshot shows the page loaded. All in the background.

Architecture

Three clean layers. No framework magic.

Layer 1 ─ MCP Protocol         app/server.py         Thin. Validates, delegates, formats.
Layer 2 ─ Session Manager       app/session.py        Per-app lifecycle, snapshots, recovery.
Layer 3 ─ Platform Backend      app/_lib/             One module per macOS subsystem.

Platform Backend Modules

Key Design Decisions

• CGEventPostToPid, never CGEventPost — all input is process-targeted. The global event stream (your cursor, your keyboard) is never touched
• Window capture without activation — ScreenCaptureKit captures by window ID, even if the window is fully occluded
• User interruption detection — if you start using an app the agent is working in, mac-cua detects the conflict and yields to you
• Snapshot-local indices — element indices are valid only for the snapshot that produced them; no stale references
• Cross-app robustness — detects and adapts to Native Cocoa, Electron, Safari, Chrome, Java, and Qt apps
• Event-driven settling — wait_for_settle with per-tool timeouts and debounce, not fixed sleep() calls
• Per-app guidance — custom operational hints per bundle ID (e.g., app/guidance/com.apple.Music.md)

Cross-App Test Results

Tested across 4 apps and all 9 tools. ~30 tool invocations, zero crashes, zero focus theft, zero cursor hijacking.

App Compatibility

Tool Results

What works well

• AX element clicks are the most reliable cross-app interaction
• set_value is instant on Native Cocoa — replaces entire document contents without typing
• type_text delivers to the correct process even with multiple apps running
• press_key handles complex modifiers (Cmd+A, Cmd+Z, Shift+End) reliably
• Screenshots capture every window correctly regardless of visibility

Known Limitations

• Electron set_value — VS Code and other Electron apps reject AX value setting. Use type_text instead
• Electron double-click — click_count > 1 fails on some Electron elements
• Electron scroll — webview elements don’t expose scroll to AX. Coordinate-based scrolling needed
• macOS auto-correct — type_text triggers the system spell checker, which can mangle text
• Qt apps — apps with zero AX exposure (like Macs Fan Control) can’t be automated
• Background shortcuts — some complex key combos don’t trigger when sent via background events

Supported App Types

mac-cua works with any macOS application that exposes an accessibility tree:

• Native Cocoa — Finder, Safari, Music, System Settings, Notes, Calendar, TextEdit
• Electron — VS Code, Slack, Discord, Notion (click + type_text work; set_value limited)
• Chromium — Chrome, Arc, Edge
• Java — JetBrains IDEs (IntelliJ, PyCharm, WebStorm)
• Qt — Limited. Falls back to screenshot-based coordinate interaction

Safety

• App blocklist — prevents interaction with system security processes (Keychain, login)
• URL blocklist — SSRF protection for web-based interactions
• App approval flow — session and persistent approval gates before controlling new apps
• Step limits — per-turn cleanup and step tracking to prevent runaway loops
• Background-only — cannot inject events globally; input is always process-targeted
• User wins — interruption detection yields control back to you immediately

Development

# Install dependencies
uv sync

# Run tests
uv run pytest

# Run a specific test
uv run pytest tests/test_safety.py -v

# Run the server
uv run python main.py

Project Structure

mac-cua/
  main.py                  Entry point, permissions, logging
  app/
    server.py              MCP protocol layer
    session.py             Session lifecycle & orchestration
    response.py            Response dataclasses
    guidance/              Per-app operational hints
    _lib/                  Platform backend (17 modules, ~7300 LOC)
  tests/                   136 tests
  specs/                   Tool reference docs

Contributing

Contributions are welcome! mac-cua is a community-driven project and we’d love your help.

1. Fork the repo
2. Create a branch (git checkout -b my-feature)
3. Make your changes — add tests if applicable
4. Run the test suite (uv run pytest)
5. Open a Pull Request

Whether it’s a bug fix, new app guidance file, documentation improvement, or a whole new feature — all contributions are appreciated.

If you find mac-cua useful, consider giving it a star. It helps others discover the project.

License

Apache License 2.0 — use it, fork it, ship it, sell it. No strings attached.

Acknowledgments

mac-cua was inspired by Codex computer use (OpenAI, April 2026) and Personal Computer (Perplexity, April 2026). Both showed that background desktop automation is the future — mac-cua brings that capability to everyone as an open-source MCP tool that works with any LLM.

Built with MCP for universal LLM compatibility, PyObjC for macOS integration, and ScreenCaptureKit for GPU-accelerated background capture.