walterhu1015/mmskills

Towards Multimodal Skills for General Visual Agents

News | Paper | Website | Skill Library | Demos | Overview | Installation | Quick Start | Citation

If you find this project helpful, please give us a star ⭐ for the latest updates.

📣 Latest News

• 🏆 [May 2026] MMSkills ranked #1 on Hugging Face Daily Papers.
• 🤗 [May 2026] The MMSkills dataset is now available on Hugging Face Datasets.
• 🌐 [May 2026] The project website is live with demo comparisons and a searchable MMSkills Library indexing 515 skills across Ubuntu, macOS, VAB-Minecraft, and Mario.
• 🚀 [May 2026] The public release includes a compact multimodal desktop-skill subset, OSWorld-ready runtime adapters, task mappings, and model-agnostic skill modes.
• 🔌 [May 2026] We are exploring integrations with OpenClaw, Codex, and Claude Code. Stay tuned.

🎬 Demos

Four OSWorld demos compare the same task under no skills, text-only skill guidance, and multimodal MMSkills. To keep GUI text readable in the GitHub README, each case uses three separate 1080p MP4 players instead of a compressed side-by-side composite. The full video layout is also available at deepexperience.github.io/MMSkills/cases.html.

💡 Overview

MMSkills is a framework for representing, loading, and using reusable multimodal procedural knowledge for visual agents. Each skill combines textual procedure guidance, compact state-card metadata, and optional visual references. At inference time, the agent keeps only lightweight skill hints in the main context, then opens a temporary skill branch when task state suggests that a skill may help.

This repository is a focused open-source release. It is not a full OSWorld fork; instead, it provides the MMSkill runtime layer, an install script, OSWorld runner patches, task-to-skill mappings, and a representative public skill library.

Project pages:

• arXiv paper
• MMSkills website
• Searchable Multidomain Skill Library
• Demo video comparisons

Website frontend files are published from the gh-pages branch. The main branch is kept focused on the open-source code, runtime integration, skills, and documentation.

✨ Highlights

🗂️ Repository Layout

MMSkills/
├── mm_agents/                 # MMSkill runtime architecture and model adapters
├── osworld_integration/       # MMSkills-aware OSWorld runner files
├── skills_library/            # Public multimodal skills subset for direct runtime use
├── task_skill_mappings/       # OSWorld task-to-skill mapping for released skills
└── scripts/
    ├── install_into_osworld.py # Install this release into an OSWorld checkout
    └── sync_from_sources.py    # Maintainer sync helper for source checkouts

🧠 Architecture

The public runtime entrypoint is mm_agents/mm_skill_agent.py, exposed in OSWorld as:

--agent_type mm_skill

The architecture is model-agnostic. A main visual agent receives compact skill hints; when a skill may apply, the runtime opens a branch that decides whether visual evidence is needed, requests relevant state views, compares them with the live screenshot, and returns structured guidance for the next grounded action.

The reference integration supports:

• mm_skill: multimodal branch-loaded skill consultation.
• general_text_skill: text-only skill consultation for ablation and lightweight runs.
• general: baseline model-agnostic screenshot-to-pyautogui visual-agent routing.

Legacy gemini, gemini_skill, and gemini_text_skill CLI names are still accepted by the runner as aliases for compatibility, but the public files and recommended commands use the model-agnostic general* names.

Any screenshot-capable VLM served through an OpenAI-compatible chat-completions API can use the same general* and mm_skill interfaces by setting --model, --api_model when needed, --base_url, and --api_key.

🔧 Installation

1. Clone MMSkills

git clone https://github.com/DeepExperience/MMSkills.git
cd towards_mmskills

2. Install Python dependencies

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

3. Install into OSWorld

Clone and install OSWorld following its upstream instructions, then run:

python3 scripts/install_into_osworld.py /path/to/OSWorld --with-runner --with-skills

This copies the MMSkill agent files into OSWorld/mm_agents/, installs the MMSkills-aware runner files, and copies the released skills_library/ plus task_skill_mappings/.

4. Configure model endpoints

For an OpenAI-compatible endpoint:

export OPENAI_BASE_URL="https://your-openai-compatible-endpoint/v1"
export OPENAI_API_KEY="your_api_key"

For native Gemini-compatible routing, pass --api_backend gemini and set:

export GEMINI_BASE_URL="https://your-gemini-compatible-endpoint/v1"
export GEMINI_API_KEY="your_api_key"

🏃 Quick Start

Run commands from the OSWorld checkout after installation.

Baseline Without Skills

python run.py \
  --agent_type general \
  --model gpt-4o \
  --api_backend openai \
  --observation_type screenshot \
  --action_space pyautogui \
  --max_steps 20 \
  --test_all_meta_path evaluation_examples/test_nogdrive.json \
  --domain chrome \
  --result_dir results/no_skills

Text-Only Skills

python run.py \
  --agent_type general_text_skill \
  --model gpt-4o \
  --api_backend openai \
  --observation_type screenshot \
  --action_space pyautogui \
  --max_steps 20 \
  --skills_library_dir skills_library \
  --task_skill_mapping_root task_skill_mappings/task_skill_mapping.json \
  --skill_mode text_only \
  --text_skill_mode branch_planner \
  --test_all_meta_path evaluation_examples/test_nogdrive.json \
  --domain chrome \
  --result_dir results/text_only

Multimodal MMSkill Agent

python run.py \
  --agent_type mm_skill \
  --model gpt-4o \
  --api_backend openai \
  --observation_type screenshot \
  --action_space pyautogui \
  --max_steps 20 \
  --skills_library_dir skills_library \
  --task_skill_mapping_root task_skill_mappings/task_skill_mapping.json \
  --skill_mode multimodal \
  --task_skill_top_k 6 \
  --save_conversation_json \
  --test_all_meta_path evaluation_examples/test_nogdrive.json \
  --domain chrome \
  --result_dir results/mm_skill_multimodal

Use --domain all for the full no-Google-Drive OSWorld split. The runner writes trajectories, screenshots, skill_invocations.json, skill_usage_summary.json, and aggregate metrics under the selected --result_dir.

📚 Skill Library

The website indexes 515 skills from the open-source Ubuntu, macOS, VAB-Minecraft, and Mario skill assets. Each skill card links to a structured view of its SKILL.md, runtime state cards, and ordered visual references.

Browse the live library at deepexperience.github.io/MMSkills/skills.html.

The repository also includes a compact runtime-ready subset under skills_library/ for immediate OSWorld integration.

📦 Skill Package Format

skills_library///
├── SKILL.md                  # Procedure, applicability, transfer limits, checks
├── runtime_state_cards.json  # Compact state/view metadata used at inference time
├── state_cards.json          # Audit-grade state metadata for inspection
├── plan.json                 # Generated plan metadata, when available
└── Images/                   # Full frames, focus crops, before/after references

The main agent sees only concise skill names and state hints. Detailed visual evidence is loaded lazily by the branch planner, which keeps the main context compact while preserving access to state-specific multimodal references.

runtime_state_cards.json is the inference-facing version: it contains compact state descriptions, when-to-use rules, visible cues, verification cues, and selected image views for branch-time loading. state_cards.json is the richer authoring/audit version: it keeps transfer-limit notes, highlight targets, grounding queries, bounding boxes, crop decisions, and evidence-source metadata for inspection and regeneration.

🧪 Outputs

MMSkills adds skill-aware artifacts to OSWorld result directories:

🤝 Contributing

Contributions are welcome for new skills, runtime integrations, documentation, and reproducibility fixes. Please read CONTRIBUTING.md before opening an issue or pull request.

📄 License

This project is released under the Apache License 2.0. Portions of the OSWorld integration are derived from OSWorld; see NOTICE for attribution details.

📝 Citation

If you use MMSkills in your research or applications, please cite our arXiv paper:

@misc{zhang2026mmskills,
  title = {MMSkills: Towards Multimodal Skills for General Visual Agents},
  author = {Kangning Zhang and Shuai Shao and Qingyao Li and Jianghao Lin and Lingyue Fu and Shijian Wang and Wenxiang Jiao and Yuan Lu and Weiwen Liu and Weinan Zhang and Yong Yu},
  year = {2026},
  eprint = {2605.13527},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  url = {https://arxiv.org/abs/2605.13527}
}

You can also use the machine-readable citation metadata in CITATION.cff.