代理桌面

AI代理的计算机用途:通过OS访问树读取并操作桌面应用程序.

一个用于macOS,Windows和Linux的CLI和C库. 一个快照返回了 JSON , 并带有对一个特工可以操作的每个元素的参考。 代理器再次在回放和快照上进行检查, 没有像素匹配; 循环留在您的代理器中 。

在macOS上,CLI称每个快照命名空间和二进制构建都使用一个私有命令主机. 它保留了本地的无障碍对象,因此被移除的行无法默默地解决有相同标识的幸存行. 主机在闲置了15分钟后退出。 TMPDIR 必须由当前使用模式的用户拥有 [[T1] ; 标准的每个用户临时目录的macOS满足了这一要求 。 无头呼叫仍然禁止前景活化并合成输入.

抓图,剪贴板读取,截图,命令输出和调试文物可以包含私人桌面内容. 让他们远离公共日志和上传. 可选的JEV帮助器向已配置的模型提供者发送屏幕描述;它与原生的CLI是分开的. 参见隐私和数据处理.

□ 安装

在 macOS 、 Windows 或 Linux 上安装 npm :

agent-desktop

Computer use for AI agents: see and operate desktop apps on macOS, Windows, and Linux through the OS accessibility tree.

agent-desktop gives an AI agent computer use without screenshots or pixel matching. It reads the accessibility tree of a desktop app and returns it as JSON, with a ref for every element the agent can act on. The agent clicks or types on a ref, reads the UI again to check the result, and repeats. The loop lives in your agent; agent-desktop has no model inside. It ships as a CLI and a C library.

Snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content. Keep them out of public logs and uploads.

Install

npm install -g agent-desktop
agent-desktop status                # platform, permissions, supported surfaces
agent-desktop permissions --guide   # macOS: walks you through the Accessibility grant

Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64. For direct execution without a JavaScript launcher, download the matching native archive from GitHub Releases and keep its helper beside the executable.

On macOS and Linux, Homebrew also works: brew install agent-desktop. There is no official Scoop package yet. The repository no longer maintains a custom Homebrew tap or Scoop bucket.

Release archives and building from source are covered in Installation.

Quick start

agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to verify

Each command prints one JSON envelope. A trimmed snapshot:

{
  "version": "2.6",
  "ok": true,
  "command": "snapshot",
  "data": {
    "app": "Finder",
    "snapshot_id": "s8f3k2p9",
    "ref_count": 59,
    "complete": true,
    "tree": {