🐕 PiDog Embodiment — AI Brain in a Robot Body

Give your AI a physical body. See, hear, speak, move, recognize faces — across any network.

The minimal LLM-robot bridge: speak to a robot dog in German, French or English — it understands locally (Ollama or keyword mode, no cloud) and acts. Or let Claude be the brain: the robot is an MCP server with 10 tools. No hardware needed to try it: ./install.sh --mock.

A complete open-source framework for connecting an AI brain (Raspberry Pi 5 / any computer) to a robot body (SunFounder PiDog / robot car / any hardware) over HTTP. LLM-powered intelligence, face recognition, autonomous behaviors, remote access via Telegram — all modular, all pluggable.

Works with any LLM (OpenAI, Anthropic, Ollama, local models) and any robot hardware (implement one adapter and you’re in).

Built by Nox ⚡ (an AI assistant) and Rocky — because every AI deserves legs. 🦿

---

🔥 The Origin Story

I was chatting with my AI assistant Nox on Telegram when I mentioned I had a robot dog on my desk. Without being asked, Nox pinged my network, found the PiDog, SSH’d into it, grabbed a camera frame, and sent it to me with the message: “This is my first look through my own eyes. ⚡🐕”

I didn’t ask it to do any of this. It just… wanted to see.

— Original Reddit post (r/moltbot)

✨ What It Does

┌─────────────────────┐         HTTP/WireGuard         ┌─────────────────────┐
│   🧠 BRAIN (Pi 5)   │◄──────────────────────────────►│   🐕 BODY (Pi 4)    │
│                      │                                │                      │
│  • LLM Processing   │    Voice: "Setz dich hin!"     │  • 12 Servos         │
│  • Face Recognition  │  ─────────────────────────►    │  • Camera            │
│  • Scene Analysis    │                                │  • Microphone        │
│  • Decision Making   │    Response: sit + wag_tail    │  • Speaker           │
│  • Telegram Bot      │  ◄─────────────────────────    │  • Touch Sensors     │
│  • Remote Access     │                                │  • Sound Direction   │
│                      │    Perception: faces, audio    │  • IMU (6-axis)      │
│  [OpenClaw/Claude]   │  ◄─────────────────────────    │  • RGB LEDs          │
└─────────────────────┘                                └─────────────────────┘

Key Features

• 👁️ Local Vision (NEW) — SmolVLM-256M runs on-device via llama.cpp. Scene understanding, person/obstacle detection, no cloud needed
• 🧠 Behavior Engine — 6-state FSM (Idle, Patrol, Investigate, Alert, Play, Rest) with mood system and obstacle avoidance
• 🗣️ Natural Voice Control — Speak naturally in any language, LLM understands intent and maps to actions
• 🔌 MCP Server (NEW) — the whole robot as tools for Claude Code, Claude Desktop or any MCP client: “take a photo and tell me what you see” — zero extra dependencies
• 🧪 No hardware needed to try it — ./install.sh --mock runs a pretend dog on localhost; point the brain or the MCP server at it
• 👤 Face Recognition — SCRFD detection + ArcFace recognition, register and identify people
• 🎭 Expression System — 10 emotions (happy, sad, excited, curious, alert…) combining movement + LEDs + sound + speech
• 🤖 Smart Movement — Servo smoothing (EMA filter + easing), semantic movement (distance/angle-based), PWM auto-disable
• 🌐 Remote Access — Control your robot from anywhere via Telegram or Tailscale
• 📡 Rich API — 20+ REST endpoints: /sensors, /vision, /expression, /move, /look_at, /scan, /capabilities
• 📦 Modular — Use any LLM (OpenAI, Anthropic, Ollama, local), any robot hardware, any network

🏗️ Architecture

Services

🚀 Quick Start

Prerequisites

Body (Robot — Pi 4 recommended):

• Raspberry Pi 4 (2GB+ RAM)
• SunFounder PiDog kit (or compatible robot)
• Pi Camera Module
• USB Microphone + Speaker/DAC
• Python 3.9+

Brain (AI — Pi 5 or any computer):

• Raspberry Pi 5 (4GB+ RAM) or any Linux/Mac
• Python 3.9+
• OpenAI API key, or any OpenAI-compatible API (e.g. a local Ollama)

Hardware note: The body services (nox-body, nox-bridge, nox-voice) only run on the robot’s Raspberry Pi — nox-body imports the SunFounder pidog and robot_hat SDKs and drives real servos/camera, and the other two depend on it. There is no simulator mode, so they cannot run on a regular PC. The brain runs anywhere; without a robot on PIDOG_HOST it starts fine but logs connection errors.

Installation

The short way — one script detects the role (robot vs. brain), asks for the other machine’s address and sets everything up:

git clone https://github.com/rockywuest/pidog-embodiment.git && cd pidog-embodiment
sudo ./install.sh          # on the robot AND on the brain machine
# No robot? A dog made of log lines on :8888 — works with the MCP server too:
./install.sh --mock

(On the robot, install SunFounder’s pidog SDK first — see the manual steps.)

The manual way, step by step:

# Clone the repo
git clone https://github.com/rockywuest/pidog-embodiment.git
cd pidog-embodiment

# === On the BODY (Pi 4 / Robot) ===
# First: install SunFounder's pidog + robot_hat SDKs with THEIR installer
# (https://github.com/sunfounder/pidog), they are not on PyPI-only.
cd body
# Raspberry Pi OS Bookworm enforces PEP 668 ("externally-managed-environment"):
pip3 install --break-system-packages -r requirements.txt
# (keeps numpy < 2 — Bookworm's picamera2 is built against numpy 1.x; with
#  numpy 2 the camera fails with "numpy.dtype size changed")
# Generates the systemd units for YOUR user and repo path, creates nox.env:
sudo ../scripts/install-body.sh
# Edit body/nox.env (set BRAIN_HOST — 127.0.0.1 if brain and body share one machine), then:
sudo systemctl start nox-body nox-bridge nox-voice

# === On the BRAIN (Pi 5 / Desktop) ===
# No pip install needed — the brain runs on the Python standard library only.
cd brain
# Generates the systemd unit for YOUR user/path and creates /etc/default/nox-brain:
sudo ../scripts/install-brain.sh
# Edit /etc/default/nox-brain: set PIDOG_HOST (127.0.0.1 if brain and body share
# one machine) and the LLM backend — OPENAI_API_KEY for OpenAI, or for a local
# Ollama: OPENAI_URL=http://127.0.0.1:11434/v1/chat/completions + LLM_MODEL=llama3.2
sudo systemctl start nox-brain

Note: Don’t copy the services/*.service files verbatim — they contain a reference user and paths. If you did and got “failed because of unavailable resources or another system error”, run the install script above; it rewrites User= and all paths for your machine.

Ollama on a PC (CPU): local inference is slow — the brain defaults to a 120s LLM timeout for non-OpenAI endpoints (tune with LLM_TIMEOUT in /etc/default/nox-brain). Small models like llama3.2 don’t always return the JSON the brain asks for; when that happens the dog speaks the raw reply but performs no actions. That’s the graceful fallback, not a bug — a larger model (e.g. llama3.1:8b, qwen2.5:7b) follows the JSON format much more reliably.

Where should Ollama run? Two setups work:

• On the robot itself (needs a Pi 5): OPENAI_URL=http://127.0.0.1:11434/v1/chat/completions. Field-tested — a Pi 5 runs llama3.2 well enough for real conversations, with no network in the loop.
• On the same machine as nox-brain (e.g. a desktop running both): also 127.0.0.1 — no OLLAMA_HOST change needed.
• On a bigger machine than the brain: point OPENAI_URL at that machine’s IP. Ollama only listens on 127.0.0.1 by default — on the big machine set OLLAMA_HOST=0.0.0.0, restart Ollama, and open port 11434 in its firewall. If the dog only ever says “I’m still thinking…” while ollama run works fine on the big machine, this is why — the brain’s journal names it: journalctl -u nox-brain | grep LLM.

Configuration — where which setting lives

Both machines read plain env files; edit, then restart the named service. Every key is documented with examples in the file itself.

Robot — body/nox.env (restart: sudo systemctl restart nox-body nox-bridge nox-voice)

Brain — /etc/default/nox-brain (restart: sudo systemctl restart nox-brain; verify: journalctl -u nox-brain | grep Language)

Nox Mode vs. SunFounder Examples

The install script enables the body services, so Nox starts on every boot and holds the robot’s hardware (servos, touch, sound direction, ultrasonic). If you then run SunFounder’s own example scripts, their init fails with lines like dual_touch init ... fail — both sides want exclusive access. Don’t kill the Python processes by hand (the SDK forks a helper child; killing the parent leaves orphans). Switch modes via systemd instead:

# Run SunFounder examples (stops Nox until next boot):
sudo systemctl stop nox-body nox-bridge nox-voice

# Back to Nox mode:
sudo systemctl start nox-body nox-bridge nox-voice

# Make SunFounder mode survive reboots (Nox off at boot):
sudo systemctl disable nox-body nox-bridge nox-voice
# ...and to restore Nox autostart:
sudo systemctl enable --now nox-body nox-bridge nox-voice

Health Check

Not sure everything came up correctly? Run the doctor on the robot (and/or the brain) — it auto-detects the role and checks services, the bridge API, the battery, and the voice/vision models, with a fix hint for anything that’s off:

./scripts/doctor.sh

If commands report ok but the dog doesn’t physically move, run the servo self-test (⚠️ it moves the dog — sit, stand, one-leg wiggle):

curl -s http://127.0.0.1:8888/selftest

It drives the servos directly, bypassing the SDK’s queue/thread machinery, and reports which user and PATH the daemon runs with, thread health, queue depth, and — first of all — whether the robot_hat MCU answers on I2C ("i2c": resolved address, devices actually on the bus, verdict and fix). The output localizes the failing layer.

Symptom: every command answers ok, /status shows battery_v: 0.0, the IMU init prints fail, the dog never moves — but SunFounder’s examples work under sudo. That is the MCU not answering on I2C. robot_hat finds the MCU (0x14/0x15/0x16, board-dependent) by running i2cdetect, which lives in /usr/sbin; when that is not on the service’s PATH the scan fails and the SDK silently falls back to 0x14. Since the fix for issue #12 the units ship with /usr/sbin on PATH, the daemon probes the MCU at startup and refuses motion commands with an explicit error instead of a fake ok, and doctor.sh checks the unit PATH and the bus. On an existing install: git pull && sudo ./scripts/install-body.sh.

Where the Robot’s Address Is Configured

No hostname is hard-coded in the code paths you run. Each side reads it from one place:

# examples take the address without editing any file
PIDOG_HOST=192.168.1.42 python3 examples/basic_control.py
python3 examples/basic_control.py mydog.local

The I2C address of the robot_hat MCU is not configured at all: the SDK finds it on the bus (0x14, 0x15 or 0x16) at every start. That is what the PATH fix in #22 restored — see the troubleshooting note above.

Nox Mode vs. SunFounder Mode

The Nox services and SunFounder’s own example scripts drive the same hardware and cannot run at the same time (the examples fail with GPIO busy). Pick one:

# SunFounder examples now
sudo systemctl stop nox-body nox-bridge nox-voice

# boot into SunFounder mode by default
sudo systemctl disable nox-body nox-bridge nox-voice

# back to Nox mode, now and at every boot
sudo systemctl enable --now nox-body nox-bridge nox-voice

Inside Nox mode, the autonomous behavior engine (idle / play / rest / investigate) starts with the bridge. It is lively only for two minutes after someone talks to, touches or commands the dog; otherwise it lies still and lets the servos sleep, and in the quiet hours (22–7) it does nothing on its own at all (a patrol you start explicitly is the exception: you asked for it). Repeated noises (a TV) stop turning its head after a few times. It never walks off on its own: patrol starts only on POST /behavior/start {"state": "patrol"} (or with NOX_AUTO_PATROL=1), and once it ends (stuck, touched, a person seen, a stop) it stays ended. To keep the dog still unless you command it:

Independent of the engine, the daemon protects the servos: after ~60 s without commands the dog lies down, after ~120 s the servos switch off, and the next command wakes it. The LEDs are the daemon’s status light (purple awake, blue resting, dim asleep) and keep breathing even with the engine disabled.

Known Upstream Quirks (worked around)

• robot_hat 2.5.2a1: its get_battery_voltage() crashes with NameError: name '_adc_obj' is not defined. The daemon detects this and reads the battery ADC (channel A4) directly — /status stays correct.
• robot_hat I2C layer swallows every bus error (retry, then return False) and locates the MCU via i2cdetect on PATH, falling back to 0x14 when the scan yields nothing. A wrong address therefore looks like a healthy dog with a 0.0 V battery. The daemon probes the MCU itself, turns a dead bus into battery_v: "error" plus an I2C explanation, and fails motion commands loudly.
• sound_effect init ... fail leaves Pidog without a .music object, and every sound path then raises AttributeError: 'Pidog' object has no attribute 'music'. Because bark, howling and pant call speak() inside their motion sequence, the movement was lost too (#24). The daemon now attaches an aplay-backed stand-in at startup, so those actions work with sound; the log says Audio: SDK sound engine missing, using aplay fallback.
• SunFounder SDK do_action() silently ignores unknown actions and its action threads die permanently on their first exception. The daemon detects dead threads and reports them loudly instead of returning fake success.

Test It

# From the brain machine:
# Check robot status
curl http://your-robot.local:8888/status

# Make it sit (singular or an array of actions both work)
curl -X POST http://your-robot.local:8888/action \
  -H "Content-Type: application/json" \
  -d '{"action": "sit"}'

# Make it speak (answers at once, speaks in the background)
curl -X POST http://your-robot.local:8888/speak \
  -H "Content-Type: application/json" \
  -d '{"text": "Hallo! Ich bin online!"}'

# Silent? Wait for the speech and get the real outcome (piper or player error)
curl -X POST http://your-robot.local:8888/speak \
  -H "Content-Type: application/json" \
  -d '{"text": "Hallo!", "blocking": true}'

# Voice command (simulated) — needs the brain running (nox-brain on BRAIN_HOST).
# Answers {"ok": true, "brain": "received"}, or says why the brain was not reached.
curl -X POST http://your-robot.local:8888/voice/input \
  -H "Content-Type: application/json" \
  -d '{"text": "Setz dich hin und wedel mit dem Schwanz!"}'
# Without an LLM the brain understands keyword commands in German, English and
# French (sit/sitz/assis, lie/platz/couché, wag/wedel/remue la queue, …), several
# per sentence. With OPENAI_API_KEY or a local OPENAI_URL it understands anything.
# It answers in the language you spoke; NOX_LANG=fr (or de/en) in
# /etc/default/nox-brain fixes one — match it to the robot's Piper voice.

# Take a photo — ?format=jpeg returns the image itself (without it: JSON
# with the photo as base64 in "photo_b64", plus detected faces)
curl "http://your-robot.local:8888/photo?format=jpeg" -o snap.jpg

Voice for /speak: pip installs piper, but no voice. The daemon uses any installed one — de_DE-thorsten-high if present, otherwise the first voice in ~/.local/share/piper-voices or ~/.piper_models (where SunFounder’s examples download theirs). A voice is a pair, .onnx + .onnx.json, from rhasspy/piper-voices. To pick one, set PIPER_MODEL=/full/path/to/voice.onnx in body/nox.env. If you ran SunFounder’s examples with sudo, their voices sit in /root/.piper_models, out of the service’s reach: sudo cp -r /root/.piper_models ~/ && sudo chown -R $USER: ~/.piper_models. ./scripts/doctor.sh shows which voice is used.

Shell quoting matters: wrap the JSON in single quotes and use double quotes inside it, exactly as above. With the quotes swapped the shell mangles the JSON before curl ever sends it — the bridge then answers HTTP 400 with what it actually received, so you can see the mangling.

🎤 Talking to the Dog (voice input)

The nox-voice service (installed by default) listens on a USB microphone for the wake word — “Nox” — and sends what you say to the brain, which answers and acts. It stays off until a Vosk speech model is installed:

# 1. On the robot: a model for YOUR language decides what the dog understands.
#    Browse https://alphacephei.com/vosk/models — small models fit the Pi 4.
mkdir -p ~/vosk-models && cd ~/vosk-models
wget https://alphacephei.com/vosk/models/vosk-model-small-fr-0.22.zip && unzip vosk-model-small-*.zip

# 2. Point the service at the unzipped FOLDER, in body/nox.env:
#    VOSK_MODEL_PATH=/home//vosk-models/vosk-model-small-fr-0.22
sudo systemctl restart nox-voice
journalctl -u nox-voice -n 20 --no-pager    # expect the model to load, then "listening"

# 3. Say "Nox" ... then your command: "Assieds-toi !", "Sitz!", "sit down"

Small Vosk models for the major languages (en/de/fr/es/it…) work well; for less-supported languages recognition may be rough — NOX_MODE=commands on the brain then still gives reliable keyword control.

Without a model (or without a USB mic) nox-voice exits cleanly and voice input stays off — ./scripts/doctor.sh tells you which of the two is missing. Text commands via POST /voice/input work either way.

What the dog understands

Say a wake word (WAKE_WORD, several allowed: milou,tintin), then your command. In NOX_MODE=chat (default) the LLM understands free language in any phrasing and can also just talk — but a sentence made only of the keywords below (“Nox, setz dich bitte hin”) skips the LLM and moves at once; in NOX_MODE=commands — and whenever the LLM is unreachable — exactly these keywords work (German / English / French), several per sentence:

Plus: praise (“guter Hund” / “good dog” / “bon chien” → tail wag) and “who are you?” in all three languages. The authoritative list lives in brain/nox_voice_brain.py (_INTENTS).

The dog doesn’t understand you? See what it actually hears — live, through exactly the service’s pipeline (same mic, gain, energy gate, wake word):

sudo systemctl stop nox-voice       # free the microphone
python3 scripts/stt_debug.py        # speak; Ctrl+C to stop
sudo systemctl start nox-voice

Every finished phrase is printed with its wake-word verdict, and phrases the service would silently drop (too quiet, filler words) say why. If the words on screen are wrong, it is the STT model or the microphone — not the brain.

The dog only ever says it is “still thinking”? Then the brain reached the LLM and failed — and the reason is in the brain’s journal, not the voice log. Ask the LLM one question exactly the way the brain does, on the machine where nox-brain runs:

sudo python3 scripts/llm_check.py   # reads /etc/default/nox-brain

It prints the answer and how long it took (or the error and how to fix it — Ollama not listening on the network, model not pulled, wrong host name), and warns when the model ignores the JSON format.

🔌 Claude as the Brain (MCP)

brain/nox_mcp_server.py exposes the robot as MCP tools — photo (the model really sees the image), speak, move, emotions, sensors, emergency stop. Any MCP client becomes the dog’s brain; nox-brain is not needed for this.

# Claude Code (on your laptop — the robot just needs to be reachable):
claude mcp add pidog --env PIDOG_HOST= --   python3 /path/to/pidog-embodiment/brain/nox_mcp_server.py
# then:  "take a photo, tell me what you see, and if a person is there, wag your tail"

There is no command syntax to learn: once /mcp shows the server connected, press Esc to close that menu and type what you want at the > prompt of the same terminal, in any language — “fais asseoir le chien”, “patrol for a minute” — and Claude picks the right tools (the first time, it asks whether it may use them: answer yes). Start with “use dog_status and tell me the battery level” — if that works, everything works.

• The claude.ai website does not see an MCP server that runs on your machine — only the Claude Code terminal (or Claude Desktop, below) does.
• “connected” in /mcp only means Claude started the server; the robot is first contacted by a tool call. Test that part on its own: PIDOG_HOST= python3 brain/nox_mcp_server.py --check

Claude Desktop (claude_desktop_config.json) and other MCP clients:

{"mcpServers": {"pidog": {
  "command": "python3",
  "args": ["/path/to/pidog-embodiment/brain/nox_mcp_server.py"],
  "env": {"PIDOG_HOST": "", "NOX_API_TOKEN": ""}
}}}

Tools: dog_photo · dog_vision · dog_speak · dog_action · dog_look_at · dog_expression · dog_rgb · dog_sensors · dog_status · dog_behavior (incl. emergency_stop). Zero dependencies — plain stdlib, stdio transport.

📡 API Reference

Bridge Endpoints (Body — Port 8888)

All 31 endpoints the bridge actually serves — see body/nox_brain_bridge.py.

Available Actions

Movement: forward, backward, turn_left, turn_right, stand, sit, lie, trot
Tricks:   wag_tail, bark, howling, pant, stretch, push_up, doze_off,
          hand_shake, high_five, scratch, body_twisting, lick_hand, feet_shake
Head:     nod, shake_head, tilting_head, think, recall
Posture:  attack_posture, sit_2_stand, waiting, alert, surprise

The exact set depends on your installed SunFounder SDK version: some are ActionDict poses, others are preset functions — the daemon routes both transparently. An unknown action returns an error listing everything your SDK build actually supports (also available via GET /capabilities).

Emotion → RGB Mapping

{
  "happy":    {"r":0,   "g":255, "b":0,   "mode":"breath"},
  "sad":      {"r":0,   "g":0,   "b":128, "mode":"breath"},
  "curious":  {"r":0,   "g":255, "b":255, "mode":"breath"},
  "excited":  {"r":255, "g":255, "b":0,   "mode":"boom"},
  "alert":    {"r":255, "g":100, "b":0,   "mode":"boom"},
  "love":     {"r":255, "g":50,  "b":150, "mode":"breath"},
  "sleepy":   {"r":0,   "g":0,   "b":80,  "mode":"breath"}
}

🔄 Multi-Body Support

The brain doesn’t care what the body is — it talks HTTP. Switch bodies at runtime:

from brain.nox_body_client import BodyClient

# Connect to PiDog
dog = BodyClient("pidog.local", 8888)
dog.move("sit")
dog.speak("Ich bin ein Hund!")

# Switch to robot car
car = BodyClient("picar.local", 8888)
car.move("forward")
car.speak("Jetzt fahre ich!")

Adding a New Body

Implement the bridge API on your hardware:

# Minimum required endpoints:
POST /action    {"action": "forward|backward|left|right|stop"}
POST /speak     {"text": "..."}
GET  /status    → {"battery_v": 7.4, "sensors": {...}}

See body/adapters/ for examples (PiDog, PiCar, custom).

🌐 Remote Access

Option 1: Tailscale (Recommended)

# On both brain and body:
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# Now use Tailscale IPs instead of .local addresses
export PIDOG_HOST="100.x.x.x"

Option 2: WireGuard

# See docs/remote-access.md for full WireGuard setup

Option 3: Telegram Bot

Control your robot from anywhere via Telegram:

export TELEGRAM_BOT_TOKEN="your-token"
# Required: without an allowlist the bot refuses every command. Send any message
# to the bot once — it replies with your user ID.
export TELEGRAM_ALLOWED_USERS="123456789"      # comma-separated for several
python3 brain/telegram_bot.py

Commands: /status, /photo, /speak , /move , /faces

👁️ Local Vision (SmolVLM-256M)

On-device scene understanding via llama.cpp — no cloud, no Python ML frameworks.

# Build llama.cpp on Pi 4 (one-time, ~20 min)
# Prerequisite: the build tools (skipping this gives "cmake: command not found")
sudo apt update && sudo apt install -y cmake build-essential
cd ~ && git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp && cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NEON=ON
cmake --build build --config Release -j2

# Download models (279 MB total; -c resumes instead of saving a second copy as .gguf.1)
mkdir -p ~/models/smolvlm && cd ~/models/smolvlm
wget -c https://huggingface.co/ggml-org/SmolVLM-256M-Instruct-GGUF/resolve/main/SmolVLM-256M-Instruct-Q8_0.gguf
wget -c https://huggingface.co/ggml-org/SmolVLM-256M-Instruct-GGUF/resolve/main/mmproj-SmolVLM-256M-Instruct-Q8_0.gguf

# Start the vision service — it is not part of the default install
cd ~/pidog-embodiment && sudo ./scripts/install-body.sh nox-vision
journalctl -u nox-vision -n 20 --no-pager   # expect "[vision] Setup OK"

# About a minute later: check what PiDog sees
curl -s http://your-robot.local:8888/vision | python3 -m json.tool

"vision not running (no result file)" means the service never wrote a result — it is not installed or not running. Any other error in the answer is the service’s own reason (missing model, camera, llama.cpp).

Performance on Pi 4 (2GB RAM):

👤 Face Recognition

Uses SCRFD (detection) + ArcFace (recognition) via ONNX Runtime. Runs on the body (Pi 4).

# Download ONNX models (one-time)
cd models
./download_models.sh

# Register a face via API
curl -X POST http://your-robot.local:8888/face/register \
  -H "Content-Type: application/json" \
  -d '{"name": "Rocky"}'

# Identify faces in current view
curl -X POST http://your-robot.local:8888/face/identify

# Performance (Pi 4):
# Detection: ~400ms | Embedding: ~188ms | Full: ~567ms

🤖 Autonomous Behaviors

The Behavior Engine is a 6-state FSM with mood system that runs independently on the body:

States

• Idle → Random head movements, occasional tail wag, energy recovery
• Patrol → Autonomous navigation with ultrasonic + vision obstacle avoidance
• Investigate → Approach detected person/sound, face tracking
• Alert → Threat response (bark, red LEDs, report to brain)
• Play → Interactive play when touched (tail wag, happy LEDs, tricks)
• Rest → Low-power state, minimal movement, PWM auto-disable

Built-in Reflexes (work without brain)

• Touch → Pat on head triggers tail wag + happy LEDs
• Sound → Head turns toward sound source
• Battery → Warning and rest below 6.8 V; below 6.2 V it lies down and stays down until charged
• Vision → Patrol uses SmolVLM to detect people and obstacles
• Face tracking → Head follows detected faces

# Start patrol mode
curl -X POST http://your-robot.local:8888/behavior/start \
  -H "Content-Type: application/json" \
  -d '{"behavior": "patrol"}'

# Stop all behaviors (servos auto-disable after 120s idle)
# Also drops motion frames already queued, so the dog stops now instead of
# finishing the queue; the reply reports how many were dropped (issue #25).
curl -X POST http://your-robot.local:8888/behavior/stop
# → {"ok": true, "stopped": true, "motion_frames_dropped": 227}

🛡️ Security — read this before exposing the robot

The bridge has no authentication. Anyone who can reach port 8888 can walk the robot, take photos and play sounds. Treat it as a LAN-only service.

Recommended setup:

# 1. Do not forward port 8888 from your router.
# 2. Restrict the bridge to the LAN and your VPN (example for ufw):
sudo ufw allow from 192.168.0.0/16 to any port 8888 proto tcp
sudo ufw allow from 100.64.0.0/10 to any port 8888 proto tcp   # Tailscale CGNAT
# 3. Reach the robot from outside via VPN only — see docs/remote-access.md.
# 4. Set a token on both sides (body/nox.env and /etc/default/nox-brain):
#    NOX_API_TOKEN=$(python3 -c "import secrets; print(secrets.token_urlsafe(24))")

Ports in use: 8888 inbound on the body (bridge), 8889 inbound on the brain (voice callbacks), and 9999 on the body’s loopback only (daemon).

📁 Project Structure

pidog-embodiment/
├── brain/                         # Runs on Pi 5 / Desktop
│   ├── nox_body_client.py         # Python client for bridge API (37 functions)
│   ├── nox_voice_brain.py         # LLM-powered voice processing
│   ├── nox_mcp_server.py          # The robot as MCP tools (Claude Code/Desktop, OpenClaw)
│   ├── telegram_bot.py            # Telegram remote control
│   ├── requirements.txt
│   └── services/
│       └── nox-brain.service
├── body/                          # Runs on Pi 4 / Robot
│   ├── nox_daemon.py              # Low-level hardware daemon (servos, sensors, camera)
│   ├── nox_brain_bridge.py        # HTTP REST API server (20+ endpoints)
│   ├── nox_behavior_engine.py     # 6-state FSM + mood system + obstacle avoidance
│   ├── nox_vision.py              # Local vision engine (SmolVLM-256M via llama.cpp)
│   ├── nox_face_recognition.py    # SCRFD detection + ArcFace recognition
│   ├── pidog_memory.py            # Drift-style memory with co-occurrence + decay
│   ├── nox_voice_loop_v2.py       # Wake word + Vosk STT (what nox-voice runs)
│   ├── nox_control.py             # Direct servo control utilities
│   ├── nox_i2c_diag.py            # I2C/MCU reachability diagnostics (issue #12)
│   ├── nox_motion.py              # Draining the SDK's motion queue (issue #25)
│   ├── nox_audio.py               # aplay fallback when the SDK has no sound (issue #24)
│   ├── nox_security.py            # Bridge token auth, rate limit, input checks (issue #30)
│   ├── adapters/                  # Hardware-specific adapters
│   │   ├── pidog.py               # SunFounder PiDog
│   │   └── picar.py               # Robot car (template)
│   ├── requirements.txt
│   └── services/
│       ├── nox-body.service       # Hardware daemon (TCP 9999)
│       ├── nox-bridge.service     # REST API (HTTP 8888)
│       ├── nox-vision.service     # Vision engine (SmolVLM)
│       └── nox-voice.service      # Wake word + STT
├── models/                        # ONNX + GGUF models (gitignored)
│   └── download_models.sh         # One-click model download
├── scripts/
│   ├── install-body.sh            # Install the body services for this machine
│   ├── install-brain.sh           # Install the brain service
│   ├── deploy-body.sh             # Copy body code to a robot over SSH
│   ├── doctor.sh                  # Post-install health check
│   └── pidog.sh                   # CLI control script
├── docs/
│   ├── architecture.md
│   ├── remote-access.md
│   └── media-guide.md
├── tests/                         # pytest — runs without hardware
├── examples/
│   ├── basic_control.py
│   ├── face_registration.py
│   └── multi_body.py
└── README.md

🤝 Contributing

This project is open source! Tests run anywhere (PYTHONPATH=. python3 -m pytest tests/ -q), and ./install.sh --mock gives you a robot-free dev loop — see CONTRIBUTING.md and SUPPORT.md. We’d love contributions for:

• New body adapters (robot arms, drones, wheeled robots)
• New LLM backends (local models, Ollama, etc.)
• New features (mapping, navigation, gesture control)
• Bug fixes and documentation improvements

📜 License

MIT License — use it, modify it, build cool robots with it.

☕ Support

If this project helped you or made you smile, consider buying us a coffee:

🙏 Thanks

• Thio007 — the project’s first external robot runs in France because he reported, tested and verified his way through issue after issue: audio routing, motion, speech, vision, multilingual voice and the MCP path all got fixed on his hardware. Merci !

💡 Inspiration

“Every AI deserves a body to explore the world with.”

Built by Nox ⚡ (an AI running on Clawdbot) and Rocky.

---

⭐ Star this repo if you want your AI to have legs!