[! 중요] Labs, OpenAdapt 제품이 아닙니다. 이 저장소는 인식 계획 사실입니다. 컴퓨터 사용 실험. OpenAdapt 레코드, 컴파일, 또는 작업 흐름을 재생합니다. 이 제품은 openadapt-flow: 컴파일 VERIFIED]를 보고하는 프로그램 pip install openadapt.

OmniMCP는 Model Context Protocol (MCP) 및 Microsoft/OmniParser를 통해 AI 모델에 풍부한 UI 컨텍스트 및 상호 작용 기능을 제공합니다. 시각적 분석, 구조 계획 및 정확한 상호 작용을 통해 사용자 인터페이스의 깊은 이해를 가능하게합니다.

핵심 특징

• 가상 인식: OmniParser를 사용하여 UI 요소를 이해합니다. -LLM 기획: 목표, 역사, 시각 상태에 근거한 다음 행동 계획.
• Agent Executor: 관현악(omnimcp/agent_executor.py).
• Action 실행: pynput]을 통해 마우스/키보드를 제어합니다. (omnimcp/input.py).
• CLI 인터페이스 : 간단한 입력 포인트 (cli.py]) 실행 작업.
• 자동화: 선택 OmniParser 서버 배포 AWS EC2 자동-shutdown.
• ** 디버깅:** 단계별 시각 로그 생성.

개요

OmniMCP

[!IMPORTANT] Labs, not the OpenAdapt product. This repository is a perceive-plan-act computer-use experiment. It is not how OpenAdapt records, compiles, or replays a workflow. The product is openadapt-flow: a compiled program that reports VERIFIED only if an independent check agrees. pip install openadapt.

OmniMCP provides rich UI context and interaction capabilities to AI models through Model Context Protocol (MCP) and microsoft/OmniParser. It focuses on enabling deep understanding of user interfaces through visual analysis, structured planning, and precise interaction execution.

Core Features

• Visual Perception: Understands UI elements using OmniParser.
• LLM Planning: Plans next actions based on goal, history, and visual state.
• Agent Executor: Orchestrates the perceive-plan-act loop (omnimcp/agent_executor.py).
• Action Execution: Controls mouse/keyboard via pynput (omnimcp/input.py).
• CLI Interface: Simple entry point (cli.py) for running tasks.
• Auto-Deployment: Optional OmniParser server deployment to AWS EC2 with auto-shutdown.
• Debugging: Generates timestamped visual logs per step.

Overview

cli.py uses AgentExecutor to run a perceive-plan-act loop. It captures the screen (VisualState), plans using an LLM (core.plan_action_for_ui), and executes actions (InputController).

Demos

• Real Action (Calculator): python cli.py opens Calculator and computes 5*9.
• Synthetic UI (Login): python demo_synthetic.py uses generated images (no real I/O). (Note: Pending refactor to use AgentExecutor).

Prerequisites

• Python >=3.10, <3.13
• uv installed (pip install uv)
• Linux Runtime Requirement: Requires an active graphical session (X11/Wayland) for pynput. May need system libraries (libx11-dev, etc.) - see pynput docs.

(macOS display scaling dependencies are handled automatically during installation).

For AWS Deployment Features

Requires AWS credentials in .env (see .env.example). Warning: Creates AWS resources (EC2, Lambda, etc.) incurring costs. Use python -m omnimcp.omniparser.server stop to clean up.