sx4im/skillcheck

Controlled A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts — measure whether agent instructions actually improve model performance or just waste context tokens.

Not a syntax linter or basic assertion check — a controlled scientific experiment that measures statistical effect size.

Most published SKILL.md files, .cursorrules, and agent prompts have never been tested against an unprompted baseline. You cannot tell whether they help your model, make it worse, or are purely decorative prompt bloat. Skillcheck replaces vibe checks with empirical AI evaluation: paired control vs. treatment trials, blind grading, and bootstrap confidence intervals.

Point it at any Markdown skill or rule file and it runs an automated A/B experiment: it generates fresh domain-specific tasks, has the model solve each task with and without the skill injected, grades both arms blind, and reports the measured effect with a 95% bootstrap confidence interval and a 0–100 satisfaction score.

[!NOTE] Empirical AI Evaluation with Controlled A/B Testing

Evaluate your own agent skills, prompt instructions, and rule files against an unprompted baseline:

• Paired evaluation arms: Runs identical tasks with and without the skill injected.
• Blind grading: Evaluates outputs blind (the grader is blind to which arm produced an output, the subject model cannot be) with candidate order shuffled using a seeded hash.