Dokimi — A/B testing for agent context

Proof, not popularity: find out whether your agent’s skills, docs and AGENTS.md actually work.

**Documentation: ** — the full tour, with every figure.

What it is

Skills, instructions, docs, MCP servers and a repository’s AGENTS.md are cheap to write and easy to over-claim. Dokimi runs the same questions or coding tasks with and without one change, grades both the same way, and reports the difference with its p-value, per model. Then it shows which parts of a skill the model already knew, which it added and which it broke, and proposes a shorter version that keeps what matters.

dokimi — Greek δοκιμή, “test, proof”. An artifact gets trusted because it passed a test that could have failed, not because it is popular.

Which parts help: novelty. Dokimi asks the model about each claim in a skill twice, without the skill and with it, and a different model grades both answers. Every claim lands in one of four outcomes (more):

What measuring shows so far (evidence): under lenient grading, 70% of what a skill says a frontier model already produces unprompted; under strict grading, which also checks exact values, 38% is still missing even with the skill in context; a plausible wrong value planted in a skill is repeated in 85–94% of cases in a small study; and what a small model needs, a large one may not, so every verdict is per model.

Start here

cd dokimi && pip install -e .     # installs the `dokimi` command