Deskhand — Windows desktop automation for AI agents

A Windows desktop-automation server that exposes UI Automation, screen capture, synthetic input, OCR, on-screen vision, and system control over MCP (stdio and Streamable-HTTP) and a localhost HTTP API, all behind a kill switch and an append-only audit log. Built on .NET 9 + FlaUI (UIA3). (See Deskhand.html for the full architecture doc.)

Both hosts (HTTP and MCP) are thin shells over one governed backend (IAutomationBackend), so they expose identical capabilities and record to the same audit trail. The current surface is ~136 MCP tools and ~161 HTTP routes.

The build covers the single-machine Default desktop path (in-session UIA + capture + input), secure-desktop capture (UAC / lock / logon, via a SYSTEM helper), and governance. One thing is capture-only and called out where it appears: driving input on the secure desktop — a hard Windows boundary (capture works; synthetic input is refused even for a SYSTEM, uiAccess process).

Secure-desktop input: Deskhand can view the secure desktop, and ships an opt-in, best-effort path to drive it. Testing showed the UAC consent prompt cannot be driven by synthetic input by any mechanism we can deploy on demand (SYSTEM, a self-granted uiAccess token, or a signed uiAccess binary all hit ACCESS_DENIED) — it’s a deliberate Windows boundary. To automate a UAC approval, move the prompt off the secure desktop with /uac/config {promptOnSecureDesktop:false} (one-click in the dashboard). Full write-up: docs/secure-desktop-input.md.

Deskhand vs. a CUA driver

A CUA driver (computer-using agent — Anthropic Computer Use, OpenAI’s Computer-Using Agent / Operator, trycua/cua, and similar) drives a machine the way a person looks at a screen: the model gets a screenshot, reasons over the pixels, and emits click(x, y) / type(...) actions. It is OS-agnostic and needs no knowledge of the app — but every step costs a vision round-trip, and the click lands wherever the model guessed the pixel was.

Deskhand starts from the opposite end — structure before pixels — and keeps the pixel path as a fallback rather than the whole story. It is not itself an agent loop; it’s the tool surface you point your own agent at (over MCP or HTTP).