thu-aicosmos/skillevolver

---

Headline results

---

What you get

SkillEvolver is a single SKILL.md (a meta-skill) that any CLI agent — Claude Code, Codex, or another protocol agent — can activate. The agent reads the meta-skill, runs the loop on a target task, and writes out an optimized domain skill: a tight SKILL.md plus any helper scripts, references, and probes the loop discovered are worth keeping. That artifact then plugs into a fresh agent session on a new instance of the task — no SkillEvolver needed at deploy time.

How it works

Each round r = 0 … R−1 operates on the training task τ_train only:

• 🔍 Strategize. The agent brainstorms K diverse strategies and writes them as separate domain-skill variants s_{r,1} … s_{r,K}.
• 🚀 Explore. It spawns K downstream agents in parallel Harbor trials, each loading one variant of the current skill v_r. Trajectories τ_1 … τ_K come back with reward signals (r⁺, r⁻).
• 🔬 Analyze. The success vs failure contrast → Δ_r: the missing guidance that separated the winners.
• ✏️ Synthesize. A targeted patch is applied to produce a candidate ṽ_{r+1}.
• 🛡️ Audit. An independent fresh-session auditor verifies the candidate before it becomes v_{r+1}.

Refinement is deployment-grounded, not self-reflective: a candidate skill can fail by being silently bypassed, omitting a needed step, or misleading the next agent — none of which show up in the author’s own trace. So each version is tested on a fresh downstream agent before promotion.

---

The same pipeline drives two benchmarks via the Harbor runner:

Common setup (do this once)

git clone  skillevolver
cd skillevolver

# Install
conda env create -f environment.yml
conda activate skillsbench
# Auth
export ANTHROPIC_API_KEY=""

# One-shot setup: clones SkillsBench, installs Harbor, applies the 3 Harbor
# patches required by the runner, runs health checks
bash scripts/setup.sh

# Start a tmux window for Harbor (the agent uses tmux send-keys to dispatch
# Harbor commands so they don't collide with the agent's own shell)
bash scripts/prepare_tmux.sh

After this, both benchmarks work. Then pick one:

SkillsBench quick start (≈ $10, ≈ 30 min per task)

python -m agent.run \
  --task court-form-filling \
  --train-split \
  --model claude-opus-4-6

Validation runs on Benchmarks/skillsbench/tasks// (the canonical task); exploration runs on Benchmarks/skillsbench/tasks-train// (a generated variant with different filenames and values). See docs/skillsbench.md.

KernelBench quick start

# One-time: clone KernelBench upstream
git clone https://github.com/ScalingIntelligence/KernelBench.git "$HOME/KernelBench"

# Convert one problem to a Harbor task
python Benchmarks/kernelbench/harbor_converter.py \
  --kernelbench-root "$HOME/KernelBench" \
  --level 1 --problem-id 1 \
  --output-dir Benchmarks/kernelbench/tasks-train

# Build the task container
TASK_NAME=kb-l1-p1-square-matrix-multiplication \
  bash Benchmarks/kernelbench/build_image.sh

# Run the agent (continuous reward)
python -m agent.run \
  --task kb-l1-p1-square-matrix-multiplication \
  --reward-signal-mode continuous \
  --model claude-opus-4-6

See docs/kernelbench.md for converter details and how to extend the verifier for GPU performance scoring.

What’s inside

SkillsBench coverage

The paper reports headline results on 83 tasks. That number comes from the intersection of (a) the 86 training variants we ship in bench-assets/tasks-train/ and (b) the canonical tasks// set in upstream SkillsBench at the pinned commit. A few numbers that look inconsistent at first glance but aren’t:

• 86 shipped train variants vs. 83 evaluated. Three of the 86 (mhc-layer-impl, scheduling-email-assistant, speaker-diarization-subtitles) live upstream in tasks_excluded/ rather than tasks/ because they require external credentials or have integration constraints. We shipped their training variants for completeness, but didn’t include them in the paper sweep. They are runnable if you provide the required credentials and move them out of tasks_excluded/.
• Upstream has more tasks than we evaluated. SkillsBench has continued to add tasks since our cutoff (94 in tasks/ at the pinned commit, and more on upstream main). Those don’t have training variants in this repo — use tools/generate_train_variant.py to create one if you want to extend the sweep.

Bottom line: 83 tasks were evaluated end-to-end in the paper; that’s also what bash scripts/run_eval.sh runs by default at the pin.

Documentation

• Pipeline overview — how the explore → analyze → update loop works
• SkillsBench — running the discrete benchmark, sweeps, deploy
• KernelBench — converter, image build, continuous reward, GPU notes

Requirements

• Python 3.12
• Docker Desktop (Harbor runs each trial in a container)
• An Anthropic API key with access to Claude Opus 4.6

License

MIT — see LICENSE.