fullsend-ai/agent-eval-harness

以   制成

探员Eval Harness

使代理业绩能够衡量和即兴发挥

** Eval Harness代理商** 以一个申报方式评价技能和代理能力 eval.yaml:分析 生成案件 运行 法官 跟踪 ML流量 优化。 你的笔记本电脑 港湾集装箱 或EvalHub上也有同样的配置

医生 开始… 快开始… 执行模式

□为什么埃瓦尔·哈尼斯探员

-** 到处都有一个配置。** eval.yaml 驱动本地跑道、港站和EvalHub。

• ** 技能或即时模式。 ** 直接测试包装技能或代理能力 (包括代理文件检查).
• ** Robust评分。 ** LLM + 代码评分,双向A/B、阈值和HTML报告。
• ** 流动-内在痕迹。 ** 外接实验,数据集,以及分级的GenAI痕迹.
• ** 关闭循环。 ** /eval-optimize 提议通过失败和再运行实现技能修正。

□ 执行模式

吊带将引用([T3])和执行([T4]或[T5])分开:

执行模式

• 案件:每个试验案件一次援引(违约)。 绳子会绕开案子.

Made at 

Agent Eval Harness

Make agent performance measurable — and improvable

Agent Eval Harness evaluates skills and agent capabilities with one declarative eval.yaml: analyze → generate cases → run → judge → trace in MLflow → optimize. Same config on your laptop, Harbor containers, or EvalHub.

Docs · Get started · Quick start · Execution model

Why Agent Eval Harness

• One config everywhere. eval.yaml drives local runs, Harbor, and EvalHub.
• Skill or prompt mode. Test packaged skills, or agent capabilities directly (including agentic documentation checks).
• Robust scoring. LLM + code judges, pairwise A/B, thresholds, and HTML reports.
• MLflow-native traces. Opt-in experiments, datasets, and hierarchical GenAI traces.
• Close the loop. /eval-optimize proposes skill fixes from failures and re-runs.

Execution Model

The harness separates how many invocations (execution.mode) from what to execute (execution.skill or execution.prompt):

Execution Mode

• case: One invocation per test case (default). The harness loops over cases.
• batch: One invocation for all cases via batch.yaml. The skill/agent loops internally.

What to Execute

• Skill mode (execution.skill): Test predefined skill implementations (/my-skill --args). Evaluates skill correctness, quality, and cost efficiency.
• Prompt mode (execution.prompt) ✨ NEW: Test agent capabilities directly by sending prompts without a skill wrapper. Extensible to any agent evaluation scenario.

Implemented flavor - Agentic Documentation Testing (see examples/openshift-agentic-docs.md):

• Documentation effectiveness: Can agents navigate and use your docs?