通过模型上下文协议实现AI驱动的macOS应用自动化

AppMCP 弥合了 AI 模型与 macOS 应用之间的鸿沟，通过可视化检查、界面解读和对原生应用的精准控制实现智能自动化。

---

✨ 特色

🎯 视觉智能

• 智能截图：使用 ScreenCaptureKit 捕捉高分辨率应用窗口
• OCR 文本识别：使用苹果的 Vision Framework 从截图中提取文本
• UI 树分析：提取详细的无障碍层级结构，以实现精准的元素定位
• 多应用发现：同时识别和监控多个运行中的应用程序

🛠 自动化工具包

• 精准交互：鼠标点击、键盘输入和手势自动化
• 智能等待：智能延迟和基于条件的等待机制
• 错误恢复：可靠自动化的稳健后备策略

🔒 隐私与安全

• 权限管理：无缝的TCC（透明度、同意与控制）集成
• 安全通信：基于STDIO的JSON-RPC带结构化错误处理

🤖 AppMCP

AI-Powered macOS Application Automation via Model Context Protocol

AppMCP bridges the gap between AI models and macOS applications, enabling intelligent automation through visual inspection, UI interpretation, and precise control of native applications.

---

✨ Features

🎯 Visual Intelligence

• Smart Screenshots: Capture high-resolution app windows using ScreenCaptureKit
• OCR Text Recognition: Extract text from screenshots using Apple’s Vision Framework
• UI Tree Analysis: Extract detailed accessibility hierarchies for precise element targeting
• Multi-App Discovery: Identify and monitor multiple running applications simultaneously

🛠 Automation Toolkit

• Precise Interactions: Mouse clicks, keyboard input, and gesture automation
• Smart Waiting: Intelligent delays and condition-based waiting mechanisms
• Error Recovery: Robust fallback strategies for reliable automation

🔒 Privacy & Security

• Permission Management: Seamless TCC (Transparency, Consent, and Control) integration
• Secure Communication: JSON-RPC over STDIO with structured error handling
• Bundle ID Validation: Verified application targeting for enhanced security

---

🚀 Quick Start

Prerequisites

• macOS 15.0+ (Sequoia or later)
• Swift 6.1+
• Xcode 16.0+

Installation

# Clone the repository
git clone https://github.com/your-username/AppMCP.git
cd AppMCP

# Build the project
swift build -c release

# Run the daemon
./.build/release/appmcpd --stdio

Permissions Setup

AppMCP requires the following macOS permissions:

1. 🔓 Accessibility: System Preferences → Privacy & Security → Accessibility
2. 📺 Screen Recording: System Preferences → Privacy & Security → Screen Recording

The application will guide you through the permission setup process.

---

🎮 Usage Examples

Weather App Automation

import json
import subprocess

# Start AppMCP server
process = subprocess.Popen(['./appmcpd', '--stdio'])

# Take screenshot of Weather app
request = {
    "jsonrpc": "2.0",
    "id": 1,
    "method": "resources/read",
    "params": {"uri": "app://app_screenshot"}
}

# Send request and get response
response = send_mcp_request(request)
print(f"📸 Screenshot captured: {response['result']['contents'][0]['text']}")

UI Element Discovery

# Get accessibility tree
request = {
    "jsonrpc": "2.0",
    "id": 2,
    "method": "resources/read",
    "params": {"uri": "app://app_accessibility_tree"}
}

tree = send_mcp_request(request)