oh-ashen-one/device-takeover
Developer tools이 페이지는 AI로 자동 번역되었습니다. 기술적인 세부 사항은 원문을 확인하세요. 원문 보기
OpenClaw skill: AI agent takes over Android/Linux devices via ADB+SSH — see screen, send inputs, post content, play games autonomously
개요
An OpenClaw skill that lets your AI agent see a device's screen, send inputs, speak through its speakers, and listen through its mic. Over the network. In real time. Your AI agent connects to a device on your network and gets three capabilities: 1. — captures the screen via screenshots, analyzes what's on it with a vision model 2. — sends taps, swipes, button presses, text input, and gamepad commands 3. — TTS through device speakers, mic recording transcribed via Whisper These combine into three operating modes: - running on your machine (Mac, Linux, or Windows WSL) - on the same network (or reachable via SSH/ADB) - adb (Android), sshpass (Linux devices), ffmpeg (audio processing) - Whisper API access (voice input), ElevenLabs or local TTS (voice output) Connecting an Android phone requires a few one-time settings. Full walkthrough in SKILL.md, but the short version: 1. — Settings → About Phone → tap Build Number 7 times 2.
README
원문 README의 일부입니다. 원문 보기.
Device Takeover
Give your AI eyes, hands, and a voice on any device — Android, Steam Deck, Linux PC.
An OpenClaw skill that lets your AI agent see a device’s screen, send inputs, speak through its speakers, and listen through its mic. Over the network. In real time.
What It Does
Your AI agent connects to a device on your network and gets three capabilities:
- See — captures the screen via screenshots, analyzes what’s on it with a vision model
- Act — sends taps, swipes, button presses, text input, and gamepad commands
- Speak & Listen — TTS through device speakers, mic recording transcribed via Whisper
These combine into three operating modes:
| Mode | What Happens |
|---|---|
| See + Act Loop | Screenshot → vision analysis → decide action → send input → repeat |
| Voice Control | Record mic → transcribe → parse intent → execute → speak response |
| Autonomous Gameplay | Full gameplay loop — AI watches the screen, presses buttons, plays the game |
Supported Devices
| Device | Connection | Screenshot Method | Input Method |
|---|---|---|---|
| Android phone/tablet | ADB over WiFi | adb exec-out screencap |
adb shell input tap/swipe/keyevent |
| Linux Handheld (SuiPlay, Steam Deck) | SSH | Gamescope DBus TakeScreenshot |
InputPlumber DBus SendButtonChord |
| Linux PC | SSH | import -window root / grim |
xdotool / ydotool |
What You Need
- OpenClaw running on your machine (Mac, Linux, or Windows WSL)
- Target device on the same network (or reachable via SSH/ADB)
- Dependencies:
adb(Android),sshpass(Linux devices),ffmpeg(audio processing) - Optional: Whisper API access (voice input), ElevenLabs or local TTS (voice output)
Android Setup (Quick Start)
Connecting an Android phone requires a few one-time settings. Full walkthrough in SKILL.md, but the short version:
- Enable Developer Options — Settings → About Phone → tap Build Number 7 times
- Enable Wireless Debugging — Settings → Developer Options → Wireless Debugging ON
- Pair once — tap “Pair device with pairing code”, then run
adb pair IP:PORTon your computer - Connect —
adb connect IP:PORT - Test —
adb exec-out screencap -p > /tmp/screen.png
See SKILL.md for the complete step-by-step including troubleshooting.
Use Cases
1. Hands-Free Phone Control
“Read my notifications. Open the camera. Navigate to that app.” Your AI sees the phone screen, navigates the UI, and performs actions — all from a text or voice command.
2. AI Game Assistant
Point your AI at a Steam Deck or SuiPlay running a game. It watches the screen, identifies what’s happening, and either suggests moves or plays autonomously — pressing buttons through the gamepad API.
3. Remote Linux Automation
추천 도구
다른 키워드를 입력하거나 필터를 제거해 보세요.
설치
npx skillfish add oh-ashen-one/device-takeover