🚀 Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems

---

🥳 News

• [2026-06-17] We release the code and paper: Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems.

---

Table of Contents

• 1. Overview
• 2. Setup
• 3. Demo Inference (Custom Question)
• 4. Running Evolution
• 5. Evaluating a Fixed Skill
• 6. Tips
• 7. BibTeX

---

1. Overview

Skill-MAS evolves a single meta-agent skill file (SKILL.md) that instructs an LLM to design and orchestrate a Multi-Agent System (MAS) in three stages:

Trajectories from each evolution round drive contrastive reflection and skill rewriting; the best skill snapshot is selected across rounds.

Two ways to use this repo

---

Repository Layout

Skill_MAS/
├── assert/               # README figures (background.png, Method.png)
├── core/                 # CLI entry, evolution pipeline, resume, task selection
├── evolution/            # Rollout, contrastive reflection, skill optimizer, bench adapters
├── skill_mas/            # 3-stage MAS builder, async LLM client, model_config.json
├── template/             # Generated MAS code templates and SubAgent runtime
├── utils/                # Paths, cost tracking, logging, redaction
├── init_skill/           # Initial (pre-evolution) SKILL.md
├── optimized_skill/      # Pre-evolved skills per benchmark (ready to test)
├── dataset/              # Bundled benchmark code + data
│   ├── vitabench/        # VitaBench (src/vita, data/)
│   ├── deep_research_bench/
│   ├── hlemath/
│   └── BrowseComp-Plus/
├── run_*.sh              # Evolution wrappers (run from parent of Skill_MAS/)
├── demo_inference.py     # Single-question build + inference demo
├── demo_inference.sh     # Shell wrapper for demo_inference.py
└── results/              # Generated at runtime (not shipped)

How an evolution round works

1. Multi-trajectory rollout (evolution/rollout_multi.py) — run k trajectories per validation task; the agent reads the current round’s SKILL.md, generates MAS code via three build stages (skill_mas/build.py), executes it, and records scores plus phase-level traces.

2. Contrastive reflection (evolution/contrastive_reflect.py) — compare high- vs low-scoring trajectories and synthesize structured improvement signals.

3. Skill bank optimization (evolution/bank_optimizer.py) — an optimizer LLM rewrites the single SKILL.md using reflection reports and round statistics.

4. Round selection (evolution/assemble_select.py) — track per-round scores; after all rounds, select the best skill snapshot.

Benchmark backends

Where results land

Skill_MAS/results/{backend}_{model_tag}/
├── artifacts/
│   ├── skills/{bench_id}/{run_id}/round_XX/SKILL.md   # skill snapshots per round
│   └── runs/{bench_id}/{run_id}/summary_rXX.json      # round metrics
└── log/{bench_id}/{run_id}/round_XX/                  # traces, exports

Each round also saves:

round_r/
  ├── trajectories/     # per-task, per-trajectory records
  ├── aspects/          # phase-level snapshots
  └── contrastive/      # reflection reports

skills/.../round_r/
  ├── SKILL.md          # skill used in this round (rewritten after r-1)
  ├── bank_meta.json    # optimization history
  └── knee_images/      # task-priority elbow plots

---

2. Setup

Repository layout

Check out this repo so that Skill_MAS/ lives under a parent directory (e.g. demo/ or arxiv_code/). All benchmarks are vendored under Skill_MAS/dataset/; you do not need sibling copies of vitabench_single/, hlemath/, etc.

Python environment

Install dependencies for Skill-MAS and the benchmark you use (see dataset/vitabench/requirements.txt and each benchmark’s README).

conda create -n skill_mas python=3.11 -y
conda activate skill_mas
pip install openai pydantic loguru  # plus benchmark-specific deps

Model & API configuration

Edit skill_mas/model_config.json — model ids, pricing, base_url, and ${OPENAI_API_KEY} placeholders. Shell scripts resolve credentials via environment variables:

---

3. Demo Inference (Custom Question)

Use this when you want to try Skill-MAS on your own task without running multi-round evolution.

We ship ready-to-use skill files — no evolution required:

demo_inference.py loads any of these paths, runs the three-stage build, prints Stage-3 MAS code, executes the generated workflow, and prints the answer.

Supported standalone datasets in the demo: hlemath, drb, bcp. VitaBench needs the full simulator (run_vita.sh or dataset/vitabench/run_skill_mas.sh).

Option A — shell script (recommended)

cd /path/to/parent-of-Skill_MAS

# Usage: bash Skill_MAS/demo_inference.sh   ""

bash Skill_MAS/demo_inference.sh qwen3.5-plus \
  Skill_MAS/init_skill/SKILL.md \
  "What is 17 + 28? Give the final answer in \\boxed{...} form."

bash Skill_MAS/demo_inference.sh qwen3.5-plus \
  Skill_MAS/optimized_skill/hlemath.md \
  "Find the number of positive integers n such that n^2 + 3n + 2 is divisible by n + 1."

Option B — Python directly

cd /path/to/parent-of-Skill_MAS
export OPENAI_API_KEY="your-key"
export PYTHONPATH="$(pwd):$(pwd)/Skill_MAS/dataset:$(pwd)/Skill_MAS/dataset/vitabench/src"

python Skill_MAS/demo_inference.py \
  --model qwen3.5-plus \
  --skill Skill_MAS/optimized_skill/drb.md \
  --question "Summarize recent progress in retrieval-augmented generation." \
  --verbose

python Skill_MAS/demo_inference.py \
  --model qwen3.5-plus \
  --skill Skill_MAS/init_skill/SKILL.md \
  --question "Your task prompt here." \
  --dataset hlemath \
  --save-mas-code /tmp/demo_mas.py

CLI flags

BrowseComp-Plus skills (bcp) additionally accept --bcp-index-path, --bcp-retrieval-topk, --bcp-doc-max-tokens, and --bcp-max-retrieval-rounds. The BM25 index under dataset/BrowseComp-Plus/scripts_build_index/indexes/bm25 must be available (see that benchmark’s README).

For BrowseComp-Plus, refer to the official repo for data preprocessing. We do not ship a plain-text data file used in our paper, to reduce data contamination risk.

---

4. Running Evolution

Quick start (shell scripts)

Run from the directory above Skill_MAS/. Each run_*.sh takes two positional arguments: agent model id and max concurrency.

cd /path/to/parent-of-Skill_MAS

# VitaBench (cross-domain: delivery, instore, ota)
bash Skill_MAS/run_vita.sh  

# Deep Research Bench
bash Skill_MAS/run_drb.sh  

# HLEMath
bash Skill_MAS/run_hlemath.sh  

# BrowseComp-Plus
bash Skill_MAS/run_bcp.sh  

Default evolution settings in the scripts:

Tweak the variables at the top of each script to change rounds, task limits, evaluator/judge models, etc.

CLI (direct invocation)

cd /path/to/parent-of-Skill_MAS
export OPENAI_API_KEY="your-key"
export PYTHONPATH="$(pwd):$(pwd)/Skill_MAS/dataset:$(pwd)/Skill_MAS/dataset/vitabench/src"

python -m Skill_MAS evolve \
  --bench-backend vitabench \
  --bench-id skill_mas_agent \
  --run-id exp1 \
  --domain "delivery,instore,ota" \
  --jsonl Skill_MAS/dataset/vitabench/data/vita_validate.json \
  --rounds 10 \
  --k-trajectories 5 \
  --agent-llm  \
  --user-llm  \
  --evaluator-llm  \
  --optimizer-llm  \
  --max-concurrency 16 \
  --max-steps 300 \
  --language chinese

Resume and fresh runs

---

5. Evaluating a Fixed Skill

Run a benchmark with an existing skill — no multi-round evolution. The first argument is a path relative to Skill_MAS/ (e.g. init_skill/SKILL.md or optimized_skill/hlemath.md).

cd /path/to/parent-of-Skill_MAS
export OPENAI_API_KEY="your-key"

bash Skill_MAS/dataset/hlemath/run_skill_mas.sh init_skill/SKILL.md  
bash Skill_MAS/dataset/deep_research_bench/run_skill_mas.sh optimized_skill/drb.md  
bash Skill_MAS/dataset/BrowseComp-Plus/run_skill_mas.sh optimized_skill/bcp.md  
bash Skill_MAS/dataset/vitabench/run_skill_mas.sh optimized_skill/vitabench.md  

---

6. Tips

• Start with demo_inference.sh on one HLEMath question before launching a full evolution run — it is the fastest way to validate API config and skill quality.
• Use --verbose in demo_inference.py to inspect Stage 1–2 JSON; Stage 3 MAS code is always printed before execution.
• Begin with smaller MAX_PROBLEMS, fewer ROUNDS, or lower K_TRAJ in the shell scripts.
• --fresh vs resume: same --run-id resumes; add --fresh when you want a clean experiment directory.
• BrowseComp-Plus requires the BM25 index and official data pipeline — see dataset/BrowseComp-Plus/README.md.
• Cached trajectories and skills live under results/; add that path to .gitignore if you fork the repo.

---

7. BibTeX

@article{lin2026skill,
  title   = {Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems},
  author  = {Lin, Hehai and Yang, Qi and Qin, Chengwei},
  journal = {arXiv preprint arXiv:2606.18837},
  year    = {2026}
}