iphoneclaw

English | 中文

iPhone + AI. Open-source Apple Intelligence: let agents take over your iPhone.

Full demo video: assets/iphoneclaw.mp4

Official site: https://iphoneclaw.com

Feature Cards

macOS-only Python CLI worker that controls the iPhone Mirroring / iPhone镜像 window using a VLM (Vision Language Model) agent loop:

1. Capture window screenshot (Quartz CGWindowList)
2. L0 memoization check — if the screen fingerprint (dHash) matches a previously seen screen with a known-good action, replay it instantly and skip the VLM call
3. Call an OpenAI-compatible vision chat endpoint (skipped on L0 cache hit)
4. Parse Thought: / Action:
5. Execute actions via Quartz CGEvent (mouse / keyboard)
6. Verify & record each step to runs/

It also exposes a local Supervisor API (text-only + SSE) so external agent frameworks can supervise the run: poll the latest conversation (tail N rounds), subscribe to live events, and intervene with pause/resume/stop/inject. This is designed to plug into orchestrators like Claude Code / Codex as a “boss agent” supervising a UI worker.

It can also improve over time: supervisors can record “lessons learned” in WORKER_DIARY.md, and consult it before starting new tasks.

Community diary repo (opt-in PRs): https://github.com/NoEdgeAI/awesome-iphoneclaw-diary

What You Can Automate (User Recording + Replay)

iphoneclaw now supports real user behavior recording and script replay for iPhone automation:

• Record real gestures in iPhone Mirroring window:
  • python -m iphoneclaw script record-user --app "iPhone Mirroring" --out action_scripts/recorded/my_live_flow.txt
• Replay recorded script:
  • python -m iphoneclaw script run --app "iPhone Mirroring" --file action_scripts/recorded/my_live_flow.txt
• Register reusable scripts in action_scripts/registry.json, then let agent run them via run_script(name=...).

Typical use cases:

• Daily repetitive flows (open app, navigation, check-in, fixed click sequences)
• Holiday or campaign interactions (for example, red packet flows / 抢红包-like operations)
• Green app routine tasks (for example, Ant Forest energy collection / 蚂蚁森林能量收取-like operations)

Notes:

• Keep scripts focused and deterministic for better replay stability.
• Please follow each app/platform terms and local regulations when automating interactions.

Prerequisites

• A Mac (Mac mini / MacBook) + an iPhone
• iPhone Mirroring supported:
  • Mac: macOS Sequoia (macOS 15) or newer
  • iPhone: iOS 18 or newer
  • Both devices signed in with the same Apple ID
• Python >= 3.9
• Screen Recording & Accessibility permissions granted to your terminal

Installation

git clone https://github.com/user/iphoneclaw.git
cd iphoneclaw

# pip
pip install -e .

# or uvall -e .

To include dev dependencies (pytest):

pip install -e ".[dev]"
# or
uv pip install -e ".[dev]"

Verify the installation and check macOS permissions:

iphoneclaw doctor

If Screen Recording or Accessibility shows MISSING, go to System Settings > Privacy & Security and grant permissions to your terminal app.

Supported Models

iphoneclaw works with any OpenAI-compatible vision model endpoint. Below are three recommended options.

Option A: UI-TARS via vLLM (Self-hosted)

UI-TARS is a GUI agent model by ByteDance, purpose-built for screen interaction. It outputs structured Thought: / Action: in the format iphoneclaw expects natively.

Available models on HuggingFace:

Deploy with vLLM:

pip install vllm

python -m vllm.entrypoints.openai.api_server \
  --served-model-name ui-tars \
  --model ByteDance-Seed/UI-TARS-1.5-7B \
  --limit-mm-per-prompt image=5 \
  -tp 1

For the 72B model use -tp 4 (4 GPUs with tensor parallelism).

Run iphoneclaw:

python -m iphoneclaw run \
  --instruction "Open Settings and enable Wi-Fi" \
  --base-url http://127.0.0.1:8000/v1 \
  --model ui-tars

Option B: Qwen2.5-VL via vLLM (Self-hosted)

Qwen2.5-VL by Alibaba has strong vision-agent capabilities out of the box, including screen understanding and UI interaction.

Available models on HuggingFace:

Deploy with vLLM:

pip install vllm

vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --served-model-name qwen-vl \
  --limit-mm-per-prompt '{"image":2,"video":0}'

For the 72B model add --tensor-parallel-size 4.

Run iphoneclaw:

python -m iphoneclaw run \
  --instruction "Open Settings and enable Wi-Fi" \
  --base-url http://127.0.0.1:8000/v1 \
  --model qwen-vl

Option C: Volcengine Doubao UI-TARS (Cloud API)

Volcengine Ark (火山引擎方舟) hosts Doubao vision models as a managed cloud API. No GPU required – just an API key.

Available models:

Setup:

1. Register at console.volcengine.com and complete real-name authentication
2. Create an API key in the Ark console

Run iphoneclaw:

export IPHONECLAW_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3"
export IPHONECLAW_MODEL_API_KEY="your-ark-api-key"
export IPHONECLAW_MODEL_NAME="doubao-1-5-ui-tars-250428"

python -m iphoneclaw run \
  --instruction "Open Settings and enable Wi-Fi"

Or pass inline:

python -m iphoneclaw run \
  --instruction "Open Settings and enable Wi-Fi" \
  --base-url "https://ark.cn-beijing.volces.com/api/v3" \
  --api-key "$ARK_API_KEY" \
  --model "doubao-1-5-ui-tars-250428"

Quick Start

# 1. Check permissions
python -m iphoneclaw doctor

# 2. Launch iPhone Mirroring and verify window detection
python -m iphoneclaw launch

# 3. Take a test screenshot
python -m iphoneclaw screenshot --out /tmp/shot.jpg

# 4. Run the agent (pick one of the model options above)
python -m iphoneclaw run \
  --instruction "Open Settings and enable Wi-Fi" \
  --base-url http://127.0.0.1:8000/v1 \
  --model ui-tars

CLI Reference

iphoneclaw doctor          Check macOS permissions
iphoneclaw launch          Launch target app, print window bounds
iphoneclaw bounds          Print window bounds (x y w h)
iphoneclaw screenshot      Capture target window to JPEG
iphoneclaw calibrate       Screenshot + coordinate mapping info
iphoneclaw ocr             Run Apple Vision OCR on current screen
iphoneclaw windows         List visible windows (debug)
iphoneclaw run             Run the agent loop + supervisor API
iphoneclaw serve           Start supervisor API only (no worker)
iphoneclaw ctl             Control a running worker via supervisor
iphoneclaw script          Action scripts (parse/run/record/record-user/export)

Action Scripts (L1)

iphoneclaw supports local action scripts to reduce tokens and make common flows repeatable.

• Registry: action_scripts/registry.json maps a short name to a .txt script file
• Scripts can be executed locally via iphoneclaw script ...
• The agent/model can output a single low-token action:
  • run_script(name='open_app_spotlight', APP='bilibili')
  • This expands into the underlying .txt script and executes the concrete actions

Run A Script Locally

python -m iphoneclaw script run --file action_scripts/common/open_app_spotlight.txt --var APP=bilibili

Run OCR Locally (No Supervisor Needed)

python -m iphoneclaw ocr --app "iPhone Mirroring" --min-confidence 0.2

# Explicit Chinese + English language set
python -m iphoneclaw ocr --app "iPhone Mirroring" --lang zh-Hans --lang zh-Hant --lang en-US

# Save debug artifacts (raw screenshot + overlay with OCR boxes + OCR JSON)
python -m iphoneclaw ocr --app "iPhone Mirroring" --min-confidence 0.2 --debug-draw --debug-dir ./ocr_debug

Record Or Export A Script

# Record real user behavior (mouse/scroll/hotkey) inside target window
# Stop by Ctrl-C, or pass --seconds N for timed recording
python -m iphoneclaw script record-user --app "iPhone Mirroring" --out action_scripts/recorded/my_live_flow.txt

# Record action lines from stdin (Ctrl-D to finish; this is not live mouse/keyboard capture)
python -m iphoneclaw script record --out action_scripts/recorded/my_flow.txt

# Export executed actions from a previous run (runs//events.jsonl)
python -m iphoneclaw script from-run --run-dir runs/ --out action_scripts/recorded/.txt

record-user mapping highlights:

• left click -> click(...)
• left drag -> drag(...)
• right click -> right_single(...)
• wheel scroll -> scroll(...)
• hotkeys -> hotkey(key='...') (cmd 1/cmd 2 become iphone_home()/iphone_app_switcher())

Call Another Script From A Script (Nested)

Inside any .txt action script, you can include another script by short name or file path:

# by registry short name
include open_app_spotlight APP=bilibili

# by path
include action_scripts/common/open_app_spotlight.txt APP=bilibili

Equivalent explicit form also works:

run_script(name='open_app_spotlight', APP='bilibili')
run_script(path='action_scripts/common/open_app_spotlight.txt', APP='bilibili')

Register A Script (Short Name)

Add an entry to action_scripts/registry.json:

{
  "my_flow": "recorded/my_flow.txt"
}

Then the model can call:

Action: run_script(name='my_flow')

Supervisor API

The worker exposes an HTTP API on 127.0.0.1:17334 for monitoring and control:

# View recent conversation context
python -m iphoneclaw ctl context --tail 5

# Pause / resume / stop
python -m iphoneclaw ctl pause
python -m iphoneclaw ctl resume
python -m iphoneclaw ctl stop

# Inject guidance into the agent's context
python -m iphoneclaw ctl inject --text "Only toggle Wi-Fi; do not change other settings." --resume

# Run a registered action script while the worker is paused
python -m iphoneclaw ctl run-script --name open_app_spotlight --var APP=bilibili

# OCR current iPhone screen with Apple Vision (text + boxes)
python -m iphoneclaw ctl ocr --min-confidence 0.2
python -m iphoneclaw ctl ocr --lang zh-Hans --lang zh-Hant --lang en-US

SSE event stream: GET /v1/agent/events

OCR API: GET /v1/agent/ocr
Optional query params: minConfidence (0…1), maxItems (>0), lang (repeatable), autoDetectLanguage (0/1)

Typing on macOS (AppleScript)

If CGEvent/clipboard typing is unreliable, iphoneclaw can type via System Events AppleScript. By default it uses in-process NSAppleScript (permission attribution follows your terminal/python process).

export IPHONECLAW_APPLESCRIPT_MODE=native    # default
export IPHONECLAW_APPLESCRIPT_MODE=osascript # fallback via /usr/bin/osascript

Environment Variables

Claude Code Integration

iphoneclaw ships with a Claude Code skill that lets Claude supervise the worker autonomously. When invoked, Claude:

1. Starts the iphoneclaw worker in the background
2. Polls the Supervisor API every ~10 seconds (text only, no screenshots)
3. Intervenes if the worker goes off-track
4. Returns a concise summary when done

The skill uses context: fork to run in an isolated subagent — polling noise stays out of your main conversation.

Recommended supervisor models (for the boss agent, not the iPhone vision worker):

• Claude Code: prefer sonnet4.5 for fast, frequent polling and interventions.
• Codex: prefer gpt-5.3-codex-low for cheap, fast supervision loops.

Setup: The skill is auto-discovered from .claude/skills/iphoneclaw/SKILL.md when you open this project in Claude Code. For cross-project use, copy to your home directory:

mkdir -p ~/.claude/skills/iphoneclaw
cp .claude/skills/iphoneclaw/SKILL.md ~/.claude/skills/iphoneclaw/SKILL.md

Usage:

/iphoneclaw Open Settings and enable Wi-Fi
/iphoneclaw Check battery percentage and report back
/iphoneclaw Open Safari, go to example.com, and take a screenshot

Ensure model environment variables are set before invoking (IPHONECLAW_MODEL_BASE_URL, IPHONECLAW_MODEL_API_KEY, IPHONECLAW_MODEL_NAME).

Docs

• Architecture / implementation plan: PLAN.md
• Claude Code skill: .claude/skills/iphoneclaw/SKILL.md

Thanks

• UI-TARS

Friends

1. doc2x.noedgeai.com - pdf2md, PDF conversion/translation, NanoBanana image-to-editable export to PPTX, browser translation extension.
2. QuantumNous/new-api - Next-Generation LLM Gateway and AI Asset Management System.
3. teamoteam.com - Zero-deploy cloud ClawDBot, smart agents.

L0 In-Run Memoization

iphoneclaw includes an L0 memoization layer that caches screen fingerprints (64-bit dHash) within a single run. When the same screen reappears (e.g., repeated scrolling, dismissing the same overlay), it replays the known-good action instantly instead of calling the VLM — saving time and tokens.

• Zero new dependencies — dHash is computed via pure PyObjC/Quartz (grayscale CGBitmapContext)
• Safe fallback — if verification fails (screen didn’t change), it falls back to the VLM
• Per-entry reuse limit (default 3) prevents infinite loops
• Status bar masking — top 8% of the screenshot is cropped before hashing to ignore clock/battery changes

Enable via IPHONECLAW_AUTOMATION_ENABLE=1 (default off). CLI output:

[iphoneclaw] L0 cache HIT step=5 hit#1 action=scroll(direction='down')
[iphoneclaw] L0 verify OK step=5 (VLM call skipped)

Action Space

TODO

1. Fine-tune UI-TARS-1.5 7B to better fit iOS interaction patterns.
2. Add deterministic automation scripts to reduce token burn and increase speed/accuracy for known flows. (L0 memoization done; L1 deterministic scripts planned)
3. Build an iPhone-agent data labeling pipeline: use agents to generate high-quality cold-start data (UI-TARS-2 style), with less manual annotation.

License

Apache-2.0