sx4im/skillcheck

针对AI代理技能、.cursorrules、CLAUDE.md和系统提示的受控A/B测试——衡量代理指令是否真正提升模型性能，还是浪费上下文标记。

不是语法小板或基本断言检查——而是测量统计效应大小的受控科学实验。

大多数已发表的SKILL.md文件、.cursorrules和代理提示从未在无提示基线上进行过测试。你无法判断它们是帮助模型、使其更糟，还是纯粹是装饰性的提示膨胀。Skillcheck用实证AI评估替代了氛围检验：配对对照与治疗试验、盲评和自助置信区间。

指向任意Markdown技能或规则文件，它会运行自动化A/B实验：生成新的领域特定任务，让模型解决每个任务，无论是否注入该技能，双臂都进行盲评，并以95%自助置信区间和0–100满意度评分报告测量效果。

[!注] 带受控A/B测试的实证AI评估

Controlled A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts — measure whether agent instructions actually improve model performance or just waste context tokens.

Not a syntax linter or basic assertion check — a controlled scientific experiment that measures statistical effect size.

Most published SKILL.md files, .cursorrules, and agent prompts have never been tested against an unprompted baseline. You cannot tell whether they help your model, make it worse, or are purely decorative prompt bloat. Skillcheck replaces vibe checks with empirical AI evaluation: paired control vs. treatment trials, blind grading, and bootstrap confidence intervals.

Point it at any Markdown skill or rule file and it runs an automated A/B experiment: it generates fresh domain-specific tasks, has the model solve each task with and without the skill injected, grades both arms blind, and reports the measured effect with a 95% bootstrap confidence interval and a 0–100 satisfaction score.

[!NOTE] Empirical AI Evaluation with Controlled A/B Testing

Evaluate your own agent skills, prompt instructions, and rule files against an unprompted baseline:

• Paired evaluation arms: Runs identical tasks with and without the skill injected.
• Blind grading: Evaluates outputs blind (the grader is blind to which arm produced an output, the subject model cannot be) with candidate order shuffled using a seeded hash.