🤖 AppMCP

AI-Powered macOS Application Automation via Model Context Protocol

AppMCP bridges the gap between AI models and macOS applications, enabling intelligent automation through visual inspection, UI interpretation, and precise control of native applications.

---

✨ Features

🎯 Visual Intelligence

• Smart Screenshots: Capture high-resolution app windows using ScreenCaptureKit
• OCR Text Recognition: Extract text from screenshots using Apple’s Vision Framework
• UI Tree Analysis: Extract detailed accessibility hierarchies for precise element targeting
• Multi-App Discovery: Identify and monitor multiple running applications simultaneously

🛠 Automation Toolkit

• Precise Interactions: Mouse clicks, keyboard input, and gesture automation
• Smart Waiting: Intelligent delays and condition-based waiting mechanisms
• Error Recovery: Robust fallback strategies for reliable automation

🔒 Privacy & Security

• Permission Management: Seamless TCC (Transparency, Consent, and Control) integration
• Secure Communication: JSON-RPC over STDIO with structured error handling
• Bundle ID Validation: Verified application targeting for enhanced security

---

🚀 Quick Start

Prerequisites

• macOS 15.0+ (Sequoia or later)
• Swift 6.1+
• Xcode 16.0+

Installation

# Clone the repository
git clone https://github.com/your-username/AppMCP.git
cd AppMCP

# Build the project
swift build -c release

# Run the daemon
./.build/release/appmcpd --stdio

Permissions Setup

AppMCP requires the following macOS permissions:

1. 🔓 Accessibility: System Preferences → Privacy & Security → Accessibility
2. 📺 Screen Recording: System Preferences → Privacy & Security → Screen Recording

The application will guide you through the permission setup process.

---

🎮 Usage Examples

Weather App Automation

import json
import subprocess

# Start AppMCP server
process = subprocess.Popen(['./appmcpd', '--stdio'])

# Take screenshot of Weather app
request = {
    "jsonrpc": "2.0",
    "id": 1,
    "method": "resources/read",
    "params": {"uri": "app://app_screenshot"}
}

# Send request and get response
response = send_mcp_request(request)
print(f"📸 Screenshot captured: {response['result']['contents'][0]['text']}")

UI Element Discovery

# Get accessibility tree
request = {
    "jsonrpc": "2.0",
    "id": 2,
    "method": "resources/read",
    "params": {"uri": "app://app_accessibility_tree"}
}

tree = send_mcp_request(request)