Cove Sensory MCP

Cove Sensory MCP——给纯文本 LLM 一双眼睛和耳朵 is a local stdio MCP server that lets an Agent ask configured multimodal providers to inspect an explicitly supplied image, video, audio file, or music file. It is a sensory layer only: it does not provide chat, memory, personality, playback, continuous monitoring, or a calling policy.

Install and connect from source

This repository is distributed as source code. It does not require a packaged desktop app or a standalone installer. Install Git, Python 3.11+, and uv, then clone and set up the locked environment:

git clone https://github.com/moonlin1213/cove-sensory-mcp.git
cd cove-sensory-mcp
uv sync --locked
uv run cove-sensory-mcp doctor
uv run cove-sensory-mcp print-config --client generic

Supported renderers are generic, codex, claude-desktop, and claude-code. Copy the printed local stdio entry into your client, restart it, then ask the Agent to call sensory_setup_guide. Official configuration references: Codex MCP, Claude Desktop, and Claude Code MCP. Client surfaces change independently; if a surface does not support local stdio MCP, run the server from a compatible host instead.

Run later commands from the cloned repository with uv run cove-sensory-mcp .... FFmpeg is optional for initial setup and image sensing, but video/audio preparation requires a working system FFmpeg discoverable on PATH or configured locally.

Do not paste an API key into chat. Run cove-sensory-mcp configure in a local terminal: hidden input goes to the operating-system credential store, while env:VARIABLE_NAME stores only the variable name. Add each readable local directory separately with cove-sensory-mcp configure paths.

Choose the eyes and ears

• Gemini can be an eye for images/video and an ear for video audio, ordinary audio, and music.
• MiniMax-M3 can be an image/native-video eye; it is not treated as an ear, so choose a separate audio provider for a video’s soundtrack.
• An OpenAI-compatible provider using the audio_url_data_uri media mode can serve as the ear on platforms that accept an audio_url content part (for example SiliconFlow’s Qwen3-Omni-Instruct), covering audio, music, and video audio. One bounded retry covers transient 408, 429, 500, 502, 503, and 504 responses, connection/timeout failures, or malformed structured output, and may incur one additional Provider call or charge.
• A custom provider must declare capabilities and pass the tiny-media self-test before those capabilities are advertised. The self-test sends project-created media, can incur a small API cost, and requires confirmation. It verifies that the Provider call returns a valid structured observation; it does not score the model’s wording or description style.

Media is sent only to the selected, authorized Provider. Its privacy policy, retention, region, and billing rules apply. Cross-provider fallback is never inferred. Local paths must be absolute and within configured roots. URLs must be direct HTTPS media URLs; redirects, credentials, private networks, localhost, and metadata endpoints are blocked.

Tools

The seven public tools are:

• sensory_status — show redacted, verified capability status.
• sensory_setup_guide — explain missing local setup without requesting secrets.
• sensory_self_test — verify selected capabilities using tiny included media.
• sense_image — inspect an authorized image; e.g. “read the visible labels.”
• sense_video — inspect visuals and, when configured, its audio timeline.
• sense_audio — describe speech and non-speech events in an audio clip.
• sense_music — describe structure, rhythm, instrumentation, and key moments.

Run cove-sensory-mcp doctor for local configuration, credential-presence, cache, and FFmpeg diagnostics. It never prints secret values or contacts a Provider. The optional FFmpeg download stays disabled until each platform binary has audited provenance; a working system FFmpeg can be used meanwhile.

Uninstall and privacy

Remove the cove-sensory-mcp Python tool with the installer you used, or run the standalone installer’s uninstall command. Uninstall preserves configuration and OS credentials unless you explicitly request data removal. Temporary derived media is request-scoped and removed after completion; the original is never modified.

The project is Apache-2.0. Dependencies and FFmpeg keep their own licenses; see THIRD_PARTY_NOTICES.md. Please read SECURITY.md before processing untrusted media.

Development

uv sync --group dev
uv run pytest
uv run ruff check src tests scripts
uv run mypy src/cove_sensory_mcp