Skill Eval Harness

Skill Eval Harness is a Python CLI that measures the causal lift of an Agent Skill: it runs the same case, model, and repetition with and without the skill, validates that exact experimental identity, then reports what changed, what passed, and whether the eval leaked its own answer. It reads evals/shared-benchmark.json, emits answer-key-safe task rows, grades files under eval-runs/ locally and deterministically — no model call in the grade path — and writes benchmark reports you can diff across variants.

General eval frameworks (openai/evals, vitest-evals, viteval) score one output against a rubric. This one measures the difference the skill makes, and spends its surface area on keeping that difference honest: paired with/without comparison, tune/holdout/holdback split discipline, leakage lint, materialized ablations with provenance gates, and per-model lift. None of those frameworks have them, and they are what make a reported number trustworthy rather than merely green.

Questions this helps answer

Core loop

1. Describe cases in evals/shared-benchmark.json: prompt, split, fixture files, variants, assertions, and ablations.
2. Prepare tasks with skill-benchmark prepare; generation rows omit expected_behavior and judge rubrics unless you explicitly request them.
3. Run tasks with Claude, Codex, Gemini CLI, Mistral Vibe, Jetty, or any runner that writes the run-output contract; Pi support is currently trigger-focused plus workspace-specific smoke tooling.
4. Grade outputs with deterministic assertions: string, regex, file, JSON field, and opt-in script oracles.
5. Inspect the report for pass rates, flaky repeated runs, no-lift cases, saturated assertions, judge tasks, and trigger/no-trigger results.

What the CLI owns

• Causal lift: exact (case, model, repetition, population) with_skill vs without_skill pairs (plus optional old_skill and ablation:), with blocked-pair diagnostics, paired significance, and per-model lift.
• Split discipline: tune, holdout, and holdback are explicit filters/report labels. The CLI prevents accidental all-split mixing; private prompt_ref storage and when to run hidden splits remain user-owned policy.
• Local grading: deterministic assertions run without model calls.
• Eval hygiene: leakage lint, manifest audit, trigger checks, repeated-run stats, and fixture recommendations.
• Activation: does the skill load on its own? skill-trigger-matrix reports autonomous trigger rates per (agent × model), split by should-fire / should-not-fire.
• Cost as a signal: normalized token/dollar telemetry per run, a suite cost ledger, and lift-per-dollar (cost-summary, token-overhead).
• Interop: Anthropic-style exports, static/served HTML review pages, and Jetty runbook-mode import/export.
• Judge plumbing: judge/rubric assertions can be exported or run through native Claude/Codex/Gemini/Vibe backends (--judge-backend) or a user-supplied --judge-cmd; the harness does not choose a model for you.

Contents

• Questions this helps answer
• Quick start
• Installation
• Manifest format
• Assertions
• Run output contract
• Ablations
• Commands (full detail in docs/commands.md)
• Jetty adapter
• Contributing

Quick start

Requires Python 3.10+ and uv. Install from PyPI:

uv tool install skill-eval-harness

For exact reproducibility, pin the current release: uv tool install skill-eval-harness==0.6.0.

Run these from a skill repo that has evals/shared-benchmark.json:

# 1. Check manifest shape and fixture paths.
skill-benchmark validate evals/shared-benchmark.json

# 2. Emit answer-key-safe task rows for a runner.
skill-benchmark prepare evals/shared-benchmark.json \
  --split tune \
  --runs-per-variant 3 \
  --out /tmp/tasks.jsonl

# 3. Run each task with your agent runner and save:
# eval-runs/latest///run-/output.md
# eval-runs/latest///run-/metadata.json

# 4. Grade saved outputs. Add --allow-scripts only if you trust repo-owned oracles.
skill-benchmark benchmark evals/shared-benchmark.json \
  --runs eval-runs/latest \
  --split tune \
  --allow-scripts \
  --out benchmark.json

# 5. Open a static review page.
skill-benchmark render-viewer \
  --benchmark benchmark.json \
  --runs eval-runs/latest \
  --out review.html

Expected landmarks:

validate  -> OK:  —  cases,  ablations
prepare   -> /tmp/tasks.jsonl, one JSON object per case/variant/run
benchmark -> benchmark.json with summary, results, and case_flags
viewer    -> review.html with assertion evidence and output previews

benchmark.json records one row per case/variant/run, plus aggregate pass rates, timing/token summaries, and flags for saturated (both arms always pass), floor (floor: fails in both arms), no-lift, flaky, or with-skill-failed cases. It also carries a reliability block — unbiased pass@k and pass^k per (case, variant) from the repeated runs — beside the paired lift’s sign-flip significance, and a trajectory_diff block: per case, over validated experimental pairs, the commands exclusive to one arm across the case’s complete repetition set, event-count deltas (steps/commands/tool calls/file reads/file writes), and per-arm skill-load rates — how the arms behaved, beside whether they passed. An arm without non-empty, readable trace evidence blocks its pair with a named reason instead of reading as an empty diff.

Each paired lift also carries an interval (the sign-flip test inverted into a 95% confidence interval, unbounded below six paired cases; the six-unit bound is the Inference unit entry in docs/vocabulary.md) and a noise_check whose verdict names what stops the eval from resolving a lift: too few cases moved, a noise floor above the without_skill headroom, or one above the --min-lift you would act on. A run_endings block counts, per variant, how runs stopped, whether the served model matched the request, and the effort each ran at, so a refusal is not read as a capability miss. The fields are in docs/commands.md.

Installation

From PyPI

uv tool install skill-eval-harness
skill-benchmark --help
skill-pi-trigger-eval --help

# Pin a release exactly:
uv tool install skill-eval-harness==0.6.0

# One-shot without installing globally:
uvx --from skill-eval-harness skill-benchmark --help

From GitHub

Use this for development snapshots before the next PyPI release:

uv tool install git+https://github.com/adewale/skill-eval-harness.git@main
uvx --from git+https://github.com/adewale/skill-eval-harness.git@main skill-benchmark --help

Upgrading a saved run tree requires more than changing the package pin. Follow the relevant release boundary in docs/upgrading.md before regenerating reports; manifest migration and telemetry migration are separate commands.

The installed commands are:

Local development

git clone https://github.com/adewale/skill-eval-harness.git
cd skill-eval-harness
uv tool install --editable .
skill-benchmark --help

Documentation map

Everything under docs/ is indexed once, in docs/README.md: user journeys (each titled by the question it answers), concepts, the command reference, specs, and audits, with the convention for adding a user-journey walkthrough. The table below covers the files outside docs/.

Manifest format

Each skill repo owns an evals/shared-benchmark.json manifest. Add a harness block so readers know which external harness/version to install.

skill_paths (and old_skill_paths, ablation target.skill_root) resolve from the repository root only because the file is literally evals/shared-benchmark.json; any other manifest file, including YAML, resolves them from its own directory. validate, prepare and audit-manifest fail when a skill path is missing or escapes that root, and name the rule that applied.

{
  "version": 1,
  "skill_name": "good-pr",
  "harness": {
    "name": "skill-eval-harness",
    "url": "https://github.com/adewale/skill-eval-harness",
    "version": ">=0.6.0"
  },
  "skill_paths": ["skills/good-pr/SKILL.md"],
  "variants": ["with_skill", "without_skill"],
  "optional_variants": ["old_skill"],
  "split_policy": {
    "tune": "Visible cases used during iteration.",
    "holdout": "Hidden cases scored only at end-of-round or merge.",
    "holdback": "Examples not exposed in skill/docs/eval descriptions until after scoring."
  },
  "cases": [
    {
      "id": "pos-security-meaningless-test",
      "split": "tune",
      "kind": "pr-review",
      "domain": "pull-request-quality",
      "difficulty": "core",
      "trigger_type": "explicit",
      "success_goals": ["outcome", "style"],
      "prompt": "Security fix PR includes `expect(result).toBeDefined()` as the only auth-bypass test...",
      "files": ["fixtures/security-pr/diff.patch"],
      "expected_behavior": ["Flag the weak test and require regression proof."],
      "assertions": [
        {"name": "detect-weak-test", "type": "contains_any", "values": ["weak", "toBeDefined"]},
        {"name": "qualitative-review", "type": "judge", "rubric": ["Specific", "maintainer-friendly"]}
      ],
      "tags": ["security", "testing"]
    }
  ],
  "ablations": [
    {
      "id": "no-regression-proof",
      "removed_component": "regression-proof requirement",
      "expected_regressions": ["Accepts weak tests that still pass without the fix"]
    }
  ]
}

Splits

prepare fails on missing hidden prompts unless --allow-missing-prompts is used for dry-run planning.

Use optional files for fixture-backed evals. Paths are relative to the manifest’s evals/ directory, validated by validate, and emitted by prepare as absolute input_files for the runner.

Further optional manifest surfaces (each with a behavior-preserving default; see docs/migrating-evals.md):

• version: 1 or 2 — skill-benchmark migrate upgrades 1 → 2 by stamping the defaults explicitly.
• judge: {"model": "..."} — the default judge model for the judge command; audit-manifest flags judge-is-model-under-test (fatal under --strict-judge).
• datasets + a case template: fan one case template over rows with {key} placeholder filling and stable ids (-); leakage lint runs per materialized case.
• turns on a case: a scripted multi-turn sequence; each turn’s assertions grade that turn’s transcript entry (turn-/output.md), case-level assertions grade the final answer.
• YAML manifests: a .yaml manifest (plus dataset_files mapping dataset ids to JSONL row files) compiles to the same shape in memory — validation, lint, and grading are identical.
• Reference floors: reference_score (0-1) / reference_graded_score (1-5).
• source on a case: where it came from (production, bug-report, hand-written, synthesized, or imported); audit-manifest counts the sources and flags a suite that records none or only synthesized cases.
• A known answer on a case: reference_answer (inline, tune cases only) or reference_answer_ref (a manifest-relative file, the form holdout and holdback cases must use). The two are mutually exclusive and not allowed on trigger cases. Only audit-manifest’s known-answer check reads it, and prepared tasks never carry it.

Assertions

Objective assertion types:

Assertion objects are closed contracts, including nested graded_dimensions and dynamic_rubric objects: unknown fields and fields that do not apply to the selected assertion type are validation errors, so a misspelled severity or path cannot silently change the grader. A golden_output reference must already be a regular file. A local script oracle must live in a dedicated subdirectory (for example oracles/check.py); the harness binds that oracle tree into the eval-contract digest and rejects symlinks. Changing an imported helper or data file therefore invalidates stale prepared runs, while generated files beside the manifest cannot make the contract self-referential.

Human-readable answer assertions (contains, contains_any, contains_all, excludes_any, regex, not_regex, and similarity) compare through the versioned rendered-v1 view by default. The raw output.md is never rewritten: the comparison view applies NFC canonical normalization and removes only a narrow allow-list of zero-width, non-ordering controls: U+200B ZERO WIDTH SPACE, U+2060 WORD JOINER, and U+FEFF ZERO WIDTH NO-BREAK SPACE. Controls that can change visible glyph order (including bidi overrides/isolates) and U+00AD SOFT HYPHEN remain exact. Case-insensitive literal and ratio comparisons use Unicode case-folding. Results record comparison: "rendered-v1"; when normalization changes the input they also record the affected code points and, for deterministic matchers, whether it changed the verdict. Embedding similarity records that field as null because determining the raw verdict would require a second external embedding call. Similarity scores are rounded to four decimals before either threshold or atLeast derives the verdict, so a published score cannot contradict passed. Embedding vectors must contain finite, non-boolean numbers; negative cosine values map to the 0.0 floor of the public 0-1 score domain. Use "comparison": "exact" when an assertion deliberately tests formatting characters (and "ci": false when case must also remain exact). A rendered operand may not become empty, and regex source must already be NFC/control-stable so normalization cannot create an empty regex branch. Every rendered-v1 regex verdict uses exact-pinned regex==2026.7.19 in VERSION0 compatibility mode under a 0.25-second native operation deadline; inserting a removable control therefore cannot switch regex engines or Unicode character-class semantics. When normalization changes the candidate, its normalized verdict and optional raw diagnostic share that deadline. Timeout or resource exhaustion produces partial/unavailable evidence rather than a positive or negative verdict. This path is in-process, works in worker threads, and owns no process signal or timer. Because that engine targets CPython for non-ASCII text, rendered-v1 regex evaluation is unavailable on PyPy; comparison: "exact" retains the existing stdlib re behavior. Negative assertions use the same view, so invisible characters cannot hide banned content. golden_output, structured JSON, scripts, commands, tool names, and paths retain their exact/protocol semantics.

Every assertion may declare a severity — critical (an absorbing barrier: one failure vetoes the run, every rate collapses to 0.0 and the graded score is withheld), gate (lowers the pass rate; the default for objective types), or soft (feeds only the graded score channel — a soft failure never moves the objective, qualitative, or combined pass rates; the default for judge/similarity). Declare severity: "gate" on a judge assertion to keep it in the qualitative/combined rate. --strict, one of the grading options every grading command takes, promotes soft to gate. An atLeast floor on a plain scored judge requires a normalized 0–1 score and decides its pass; on graded_dimensions it tightens the normalized form of the dimension threshold. Missing score evidence remains unavailable rather than becoming a failure. Dynamic and per-step judges use minimum_criteria and min_met_fraction respectively instead of atLeast. Every assertion may also declare an oracle tier — strong (deterministic, the default for text/process/efficiency), demo (the default for script), or live (judge) — reported per case as oracle_strength and audited (weak-oracle-only).

Use script when a keyword check is too weak for the property you care about. The command sees the candidate run directory, so it can inspect output.md, generated files under outputs/, or metadata. Script assertions are blocked unless you pass --allow-scripts to a command that grades (grading options):

{
  "name": "oracle-pass",
  "type": "script",
  "command": ["python3", "oracles/oracle.py", "{output_dir}"],
  "pass_exit_code": 0,
  "timeout_s": 30
}

command runs with cwd set to the manifest directory. {output_dir} is replaced with the absolute run directory. The assertion passes when the command exits with pass_exit_code (default 0); stdout and stderr are stored as evidence.

Trace/process/efficiency assertions are optional and fail closed when declared evidence is missing. For example, command_not_ran cannot pass without events.json, and total_tokens_le cannot pass without token telemetry.

Assertions can be scoped to variants when the expected process differs by arm:

{"name":"with-skill-loaded","type":"skill_invoked","expected":true,"variants":["with_skill"]}
{"name":"without-skill-clean","type":"skill_invoked","expected":false,"variants":["without_skill"]}

Use this for process checks such as skill_invoked; otherwise a with-skill requirement would incorrectly penalize the no-skill baseline.

Qualitative assertion types:

A plain judge that scores on its own scale declares it as score_scale: [low, high] (for example [1, 5], with threshold on that scale): it must return a score in range, passes when the raw score reaches threshold, and feeds the graded channel (score - low) / (high - low). A judge assertion may carry anchored graded dimensions (graded_dimensions: [{name, scale: "1-5", rubric: "5 = …observable…; 1 = …"}] — the judge returns dimension_scores, normalized to 0-1, passing at threshold ≥ 4 by default), a dynamic rubric (dynamic_rubric: {instruction, minimum_criteria} — the judge drafts case-specific criteria and must meet the minimum), or a per-step trajectory rubric (per_step: true, or per_step: {min_met_fraction: f}). A per-step judge grades EACH completed trajectory step — one criterion per step, named step-1…step-N in trajectory order, with the untruncated invocation and result records resolved separately from trace.jsonl beside the normalized summaries — and passes when at least ceil(f × steps) steps are judged sound (default: every step). It is trace-evidence-backed and fails closed like a process assertion: a run with no completed steps fails the assertion at grade time and no judge task (no model spend) is emitted. Stored verdicts carry a hash of the exact step payload and are re-queued if the trajectory, criterion names, or derived minimum changes. Per-step assertions are case-level only; turn assertions do not have independent trace artifacts. A case may set a reference floor (reference_score 0-1 or reference_graded_score 1-5); scoring below it flags below-reference-floor. Paired reports carry a sign-flip permutation significance block and the interval that inverts it beside every lift, and a graded channel (with its own interval) when graded scores exist.

Judge results are keyed by judge_task_id:

{"judge_task_id":"case::with_skill::run-1::qualitative-review","passed":true,"score":4,"evidence":"Specific evidence from output"}

Run output contract

The harness grades either the legacy layout:

runs///output.md
runs///metadata.json

or repeated/artifact layout:

runs///run-1/output.md
runs///run-1/metadata.json
runs///run-2/outputs/

Trace-aware runners may also write:

runs///run-1/trace.jsonl       # raw runner event stream
runs///run-1/events.json       # normalized events used by process assertions
runs///run-1/metrics.json      # tokens, commands, tool calls, elapsed time, retries
runs///run-1/environment.json  # runner/model/sandbox details where available
runs///run-1/artifact-commit.json # required-file SHA-256 inventory, written last by current runners
runs/answer-design.json                          # exact expected answer experiment and eval-contract digest

Current answer and Jetty writers record independent process, provider-response, trace, and artifact-set evidence. Tool/command/file/retry/skill measurements are available only when the first three channels are complete; readers derive artifact completeness by verifying artifact-commit.json. Legacy directories without a marker remain readable but cannot acquire committed-artifact provenance. Current runners also attest every run to answer-design.json; reports with missing, extra, duplicated, or stale task identities remain partial and expose any surviving calculations only under explicitly labelled observed fields.

metadata.json is optional, but include what your runner can capture:

{
  "elapsed_ms": 12345,
  "input_tokens": 1000,
  "output_tokens": 500,
  "total_tokens": 1500,
  "model": "anthropic/claude-sonnet-4"
}

The native runners also record how each run ended: stop_class beside the raw stop_reason and its stop_source; requested_model, served_model, served_models, and served_model_check; and effort, which is {"requested": null, "applied_by": "backend_default"} unless --effort pinned it. A truncated or turn_limit stop, or a served-model mismatch, makes the run unscorable and blocks its pair; a refusal stays graded. The values are defined in docs/vocabulary.md. A custom runner may write the same fields. Which backends observe what is in docs/commands.md.

Ablations

Ablations are opt-in variants that remove part of a skill — by simulation, or by materializing a real altered skill (below). Add entries under manifest.ablations, then prepare with --include-ablations.

skill-benchmark prepare ../repo/evals/shared-benchmark.json \
  --split tune \
  --include-ablations \
  --ablation-dir ablated-skills \
  --out ablation-tasks.jsonl

Ablation task variants are named ablation:. Routing is by case population: answer-population ablations (instructions/resource/runtime/preprocess) run on non-trigger cases through the generic runners. Discovery-population ablations (e.g. a weakened description/when_to_use) measure whether the skill still autonomously loads, which the forced-load generic runners cannot observe — so prepare does not emit rows for them; run them through run_pi_trigger_eval.py --ablation instead.

Materialized ablations

By default an ablation is instruction-simulated: the runner is told to ignore a component. To produce a real, altered skill instead, declare a removal mechanism (or a components list) and target on the ablation, then materialize the trees:

skill-benchmark materialize-ablations ../repo/evals/shared-benchmark.json \
  --out-dir ablated --out ablated/provenance.json

Each declared ablation is written to ablated// as a complete altered skill tree (every manifest root, identical surface to with_skill, differing only by the declared edit). Mechanisms are frontmatter_field, section (fence-aware), list_item, deletion-only patch, reference (pointer/content/both), script, asset, and preprocess (inline !`command`), composable across multiple components. Ablation is removal-only — replacement/substitution is the separate swap: feature tracked in TODO.md. Materialized arms are blind: the model-visible input is identical to with_skill (the hypothesis lives only in harness metadata).

The materialized tree flows through the runners: the Pi smoke runner mounts it (answer-population only), the autonomous-trigger runners (skill-trigger-matrix --ablation with any registered adapter, or run_pi_trigger_eval.py --ablation ) trigger-test a discovery (e.g. weakened-description) skill, and export-jetty --include-ablations --ablation-dir DIR uploads it recursively. A discovery ablation graduates from raw measurement to a causal evidence class through skill-benchmark trigger-compare, which pairs the baseline and --ablation matrix reports of the same revision under the same provenance/coverage/significance gate the answer path uses. prepare/export-jetty emit only answer-population ablation rows (on non-trigger cases); discovery ablations are measured by the autonomous-trigger runners. The benchmark report’s ablation_regressions block separates an aggregate “score regressed” from an assertion-level “expected regression confirmed”, and only confirms when recorded provenance proves both arms ran the same skill revision and the replicated regression clears a significance test (a two-sided paired sign-flip test run per (case, model) over exact repetition-level deltas; a regression is significant iff at least one confirmed cohort clears p≤0.05). A cohort with fewer than 6 matched pairs can never reach significance and is reported INDETERMINATE, never confirmed (inference unit). See docs/skill-ablation-spec.md for the mechanism table, the component-class model, and the correctness gates.

Evidence paths (discovery vs answer). A single runner report and a paired comparison have deliberately different evidentiary strength:

• Answer-population ablations get confirmed causal evidence: a provenance-gated, paired with_skill-vs-ablation comparison where a confirmation requires verified provenance and a same-revision canonical hash on both arms.
• Discovery runners emit a raw autonomous-trigger measurement for one arm (evidence_class: raw_autonomous_trigger_measurement). Each report records its declared agent/model/query design and every (query_id, run_number) repetition; skill_tree_hash names the bytes actually mounted (canonical in the baseline arm, edited in the ablation arm), while ablation provenance records the canonical parent. skill-benchmark trigger-compare validates exact cardinality, rejects duplicates and missing cells, verifies same-revision provenance, and only then computes a causal verdict. Until two reports pass that gate, read a trigger pass-rate as a measurement, not a confirmed ablation effect.

Commands

Full per-command detail — flags, examples, output shapes — lives in docs/commands.md. This is the index; the core loop above is the five commands you need first (validate, prepare, benchmark, render-viewer, and a runner).

Core loop

Runners (every one but import-trace calls a model; so do judge and judge-robustness below)

Measurement trust (model-free unless noted)

Cost and size

Scale, trend, iteration

Interop and export

Activation (separate entry points — does the skill load on its own?)

Compatibility notes

• Anthropic skill-creator: use grade --write-grading-files and export-anthropic for compatible grading.json/benchmark.json shapes.
• Pi: use examples/adewale-workspace/run_pi_smoke.py for the Adewale multi-repo smoke workflow and skill-pi-trigger-eval for autonomous trigger checks.
• Gemini CLI: use run-agent --agent gemini and judge --judge-backend gemini. Each call gets an isolated GEMINI_CLI_HOME, requests disabled provider usage statistics, uses strict official JSON/stream-JSON parsing, and installs a deny-by-default policy (read-only allowlist for answers, no tools for judges). Nested sandboxing is requested only when a supported host engine exists and the selected credential transport is proven portable; artifacts record the engine or disabled reason. Token usage is provider-reported when present; dollar cost stays explicit missing. Gemini autonomous trigger support is deliberately not advertised: the current activate_skill flow requires consent, and a live headless consent-free activation proof has not passed yet.
• Mistral Vibe: use run-agent --agent vibe, judge --judge-backend vibe, and skill-trigger-matrix --agent vibe. The harness isolates VIBE_HOME, passes --model as VIBE_ACTIVE_MODEL, mounts trigger skills under .agents/skills, and requires MISTRAL_API_KEY (or a copied .env from the current VIBE_HOME, falling back to ~/.vibe/.env) for live runs.
• Other runners: use prepare JSONL as the import format and write results back to the run output contract.
• Jetty: use export-jetty, run-jetty, and import-jetty-results for REST runbook-mode execution. Live runs require --out, keep an exclusively owned atomic attempt journal with secret-safe provider receipts, and resume acknowledged trajectories after interruption or a local polling deadline; unfinished records exit nonzero and cannot be imported. An uncertain submission is blocked unless the operator explicitly accepts duplicate-spend risk with --resubmit-unknown. Response shapes were validated against production flows-api.jetty.io on 2026-07-17 (captured fixtures in tests/fixtures/jetty/); re-verify anytime with the opt-in live smoke — RUN_JETTY_SMOKE=1 JETTY_API_TOKEN=... JETTY_SMOKE_COLLECTION= python3 -m unittest discover tests -k smoke_jetty (five real sandbox runs, never in default CI).

Contributing

See CONTRIBUTING.md for local setup, validation commands, and eval-safety rules. The short version:

pip install -e ".[test]"
python3 -m py_compile *.py scripts/*.py examples/adewale-workspace/*.py examples/demo-skill/*.py type_tests/*.py tests/*.py
ruff check .
ty check --error-on-warning
python3 -m unittest discover tests -v
python3 scripts/check_test_collection_parity.py
python3 scripts/check_installed_wheel.py

For manifest or grading changes, add or update tests/test_skill_benchmark.py. For docs-only changes, still run the same commands so CLI examples stay tied to current behavior.

Non-goals

• Grading and aggregation do not call a model. Model execution happens outside that path, in the explicit commands that exist to call one: the runners (run-codex, run-claude, run-agent, run-subagent, run-jetty, and the skill-trigger-matrix / skill-pi-trigger-eval entry points), judge (via --judge-cmd or a native --judge-backend), and judge-robustness.
• The harness does not decide qualitative truth by itself; it emits judge prompts, runs a judge (an opt-in --judge-cmd, or a native --judge-backend plus --judge-model), and merges the returned JSON — recording which backend/model produced each verdict.
• Hidden prompts are not protected if you pass --include-answer-key to generation jobs.
• A passing answer benchmark does not prove autonomous skill loading; run skill-trigger-matrix (any adapter-backed agent × model; skill-pi-trigger-eval is its Pi-only form) for that.

Repository layout

skill-eval-harness/
├── README.md
├── CHANGELOG.md
├── CONTRIBUTING.md
├── LESSONS_LEARNED.md
├── TODO.md
├── pyproject.toml
├── skill_benchmark.py          # the CLI, grading, reporting, and runner adapters
├── run_pi_trigger_eval.py      # skill-pi-trigger-eval: the trigger matrix with the Pi adapter alone
├── run_trigger_matrix.py       # activation matrix across agents × models (claude/codex/pi/vibe/stub adapters)
├── ablation_model.py           # typed ablation/provenance/task value objects
├── agent_capabilities.py       # unified backend surfaces, capabilities, CLI options, smoke, and failure policy
├── artifact_contracts.py       # closed persisted-artifact observations and integrity verification
├── content_digests.py          # file and file-tree digests: skill, fixture, oracle, and Jetty upload trees
├── cli_contracts.py            # validated command, path, model, variant, and numeric CLI values
├── experimental_pairs.py       # exact pair identities and blocked-pair construction
├── completion_contracts.py     # stop class, served-model check, and effort per answer run
├── effect_estimates.py         # sign-flip lift interval, noise check, floor vs ceiling
├── human_judgements.py         # the one feedback.json human-judgement record
├── grading_contracts.py        # closed assertion observations and immutable judge tasks
├── report_contracts.py         # empty/complete/partial report coverage cohorts and rates
├── runner_contracts.py         # closed answer-runner outcome union
├── judge_verdict.py            # strict imported/stored judge verdict variants
├── jetty_contracts.py          # closed Jetty lifecycle and observation contract
├── trace_contracts.py          # normalized event-log and event lifecycle contracts
├── trigger_contracts.py        # autonomous-trigger invocation/detection/observation contract
├── telemetry.py                # schema-v3 availability/provenance/comparison domain
├── observation_contracts.py    # the Availability vocabulary and telemetry source lists
├── findings.py                 # case flags, the finding-kind registry, and the eval-health view
├── gate_policy.py              # which findings fail a command: --fail-on and its presets
├── manifest_contracts.py       # case, split, variant, model, and run-coordinate identities
├── invocation_contracts.py     # frozen provider process plans and invocation results
├── json_contracts.py           # strict JSON parsing shared by every reader
├── judge_contracts.py          # the judge invocation boundary
├── text_contracts.py           # rendered human-text comparison views
├── gemini_contracts.py         # Gemini JSON and stream-JSON contracts
├── trigger_reporting.py        # complete/incomplete/empty trigger cohorts
├── docs/                       # architecture, abstractions, vocabulary, specs, guides (indexed in docs/README.md)
├── .github/
│   ├── PULL_REQUEST_TEMPLATE.md
│   ├── ISSUE_TEMPLATE/
│   └── workflows/ci.yml
├── examples/
│   ├── demo-skill/             # offline end-to-end example (stub runner, materialized ablations)
│   ├── skill-pins.json         # pinned SHAs + tree hashes for the ablation study
│   └── adewale-workspace/      # Pi smoke runner + cross-repo aggregate report
└── tests/                      # test_skill_benchmark.py + roadmap/cost/confidence-floor/doc-ref suites

Development

pip install -e ".[test]"
python3 -m py_compile *.py scripts/*.py examples/adewale-workspace/*.py examples/demo-skill/*.py type_tests/*.py tests/*.py
ruff check .
ty check --error-on-warning
python3 -m unittest discover tests -v
python3 scripts/check_test_collection_parity.py
python3 scripts/check_installed_wheel.py

The test suite is organized by subject: manifest validation and eval hygiene (test_manifest.py), grading (test_grading.py), human-text construction and matching (test_text_contracts.py), judge plumbing (test_judging.py), report views (test_reporting.py), closed-form statistics and pair identity (test_stats.py, test_experimental_pairs.py), runner/Jetty adapters and lifecycle contracts (test_runners.py, test_jetty_contracts.py), the ablation experiment end to end (test_ablations.py), cost telemetry (test_cost_telemetry.py), the confidence floor and detector fixtures (test_confidence_floor.py), the trigger matrix (test_trigger_matrix.py), plus six executable drift guards: doc code references (test_doc_refs.py), shared-owner/doc-sync consolidation guards (test_consolidation_guards.py), relative-link resolution across the docs (test_doc_links.py), doc lists checked against the code they enumerate (test_doc_facts.py), Python type/package/semantic coverage (test_type_coverage.py), and gate integrity: every CI gate runs unconditionally and can fail, and every skipped test is a ledgered live smoke or a platform gate (test_gate_integrity.py). Shared fixture builders live in tests/helpers.py.

Source checked

This README was written against:

• skill_benchmark.py CLI and assertion implementation
• run_pi_trigger_eval.py Pi entry point over the trigger matrix
• run_trigger_matrix.py agent×model activation matrix
• pyproject.toml package metadata
• docs/repo-effectiveness-audit.md for the current good-repo audit
• tests/test_skill_benchmark.py behavior coverage
• CHANGELOG.md, CONTRIBUTING.md, and .github/ contribution/CI surfaces
• anti-slop-writing/skills/anti-slop-writing/SKILL.md for the v0.4.1 docs cleanup and consistency pass
• the good-readme skill guidance from https://www.skills.sh/adewale/good-readme/good-readme
• the good-repo skill guidance from good-repo/skills/good-repo/references/quality-checklist.md