Reforge AI Evaluation Course

This repository contains everything built during the live demo sessions of the Reforge AI Evaluation course. It is a general-purpose framework for building a repeatable, measurable eval loop for any AI feature or agent — from a single-turn prompt to a multi-step pipeline.

The goal is to go from “I have a prompt that works” to “I have an eval loop I trust” — and eventually to online monitoring of the agent in production.

We use a Market Map agent as the shared demo project across all four sessions. It is a concrete example you can follow along with, but the frameworks, skills, and methodology apply to any AI product.

---

Example Agent: Market Map

The Market Map agent takes a market research query and returns a ranked list of the top 3 players in that market, with supporting metrics and citations.

Example queries:

• “team chat” → ranks Microsoft Teams, Slack, Google Chat
• “what AI security startups could Okta acquire in 2026 with its cash?” → scoped acquisition target analysis
• “search in 2003” → period-accurate historical market map

It is a single-turn prompt with no tools or retrieval — all knowledge is in the model’s weights, which makes it a great eval target: the failure modes are predictable and the output format is fixed.

Prompt: prompts/market-map-prompt.md

---

What’s in This Repo

prompts/

datasets/

These datasets are specific to the Market Map agent and serve as examples of how to build and expand an eval dataset.

All datasets are in Braintrust CSV format and can be imported directly.

Metadata tags (in the metadata column as JSON):

{
  "query_type": "direct_category | competitive_comps | acquisition_targets | historical_snapshot | future_speculative | segment_specific | validation | trend_evolution | edge_out_of_scope",
  "domain": "tech_saas | healthcare | financial | consumer_brand | industrial_other",
  "style": "well_specified | under_specified | multi_constraint | jargon_heavy | edge_out_of_scope",
  "temporal": "current | historical | future",
  "edge_case": true | false
}

evaluators/

What’s inside:

Code-based evaluators (deterministic, no LLM call):

• company_count — checks that exactly 3 ranked companies appear in the output
• has_sources — checks that 3–4 source citations are present
• has_metrics — checks that each company has at least 2 supporting data points
• has_category — checks that a market category is identified before ranking

LLM judge evaluators (semantic, calls Claude):

• ranking_quality_judge — are the 3 companies the right ones in the right order?
• edge_case_handling_judge — for vague or out-of-scope queries, did the agent handle ambiguity gracefully rather than hallucinating confidently?
• reference_judge — compares the output to a gold-standard expected response (requires the expected column to be populated)

Each judge prompt includes 3 few-shot examples (clear PASS, clear FAIL, BORDERLINE) and returns structured JSON with a critique before the score.

skills/

Reusable frameworks and methodology — not Market Map-specific. These apply to any AI product eval project.

rubric-templates/

Worked examples of the AI PRD’s rubric section. Corner and Ledger were built live during the Eval Rubrics webinar — a consumer shopping agent and an enterprise finance agent, each taken from v1 rubric through a real trace to v2. Support Triage is a lighter-weight sample PRD for a support-ticket classification agent, useful as a quick reference alongside the full worked examples.

Corner and Ledger sit on opposite ends of the same framework: a consumer product where asking the user is expensive and mistakes are small and recoverable, and an enterprise product where asking is free and a mistake compounds. See the webinar recording and slides for the full walkthrough.

---

Session Overview

---

Getting Started

Prerequisites:

• A Braintrust account
• An Anthropic API key (for the LLM judge evaluators)
• Python 3.10+ with anthropic installed: pip install anthropic

Install Claude Code skills (one-time):

bash install-skills.sh

This copies all skills from skills/ into ~/.claude/commands/ so they’re available as slash commands in any Claude Code session: /uig, /eval-code, /eval-llm-judge, /llm-align, /ticket-to-eval.

Optional: braintrust-write MCP server

Enables native dataset row inserts from Claude (used by /ticket-to-eval). The install script prints the exact config snippet to add to ~/.claude/settings.json.

pip install -r mcp/requirements.txt
# then follow the config instructions printed by install-skills.sh

Session 2 setup:

1. Import week2-dataset-30.csv into Braintrust as a new dataset
2. Copy market-map-prompt.md into the Braintrust playground as your system prompt
3. Run an experiment against the dataset
4. Add evaluators.py as scorer functions in your experiment config

Running evaluators locally:

from evaluators import company_count, has_sources, ranking_quality_judge

output = "your model output here"
input_query = "team chat"

print(company_count(output, input_query))          # 0.0, 0.5, or 1.0
print(ranking_quality_judge(output, input_query))  # {"score": 1.0, "metadata": {...}}

---

The User Input Grid

The User Input Grid (UIG) is the framework we use to ensure an eval dataset covers the real diversity of inputs an agent will face in production. It applies to any AI product — the Market Map UIG below is one example.

See skills/uig-skill.md for the general methodology, or skills/uig-market-map.md for the Market Map-specific analysis.

Market Map UIG — 4 dimensions:

The Week 1 dataset (10 rows) had zero coverage of the Financial domain and Segment-Specific query type. The Week 2 dataset was designed to fill those gaps deliberately.

---

Key Ideas from the Course

On evaluator design:

• Code-based evaluators check structure. LLM judges check semantics. Use both.
• Write one criterion per judge. Compound judges are uninterpretable when they fail.
• The borderline few-shot example is more valuable than the pass example — it calibrates the gray zone.
• Validate judges with TPR/TNR against human labels before trusting them in production.

On dataset design:

• Most eval datasets are built from engineer intuition (overfits to what engineers know) or real user logs (overfits to easy common cases). A UIG forces deliberate coverage.
• 30–80 queries hitting each dimension value 3+ times outperforms 500 random queries.
• Tag every row with its dimension values. You need to slice by dimension to diagnose failures.

On the eval loop:

• Evals are only useful if they run on every prompt change. Wire them into your workflow, not just ad-hoc.
• A score going up is only meaningful if you know what changed and which eval category improved.

On judge calibration (TPR/TNR):

• Before trusting an LLM judge in production, validate it against human labels using TPR (does it catch true positives?) and TNR (does it reject true negatives?).
• False negatives (judge too strict) and false positives (judge too lenient) have different root causes and need different fixes. Investigate the scorer’s reasoning for each disagreement before changing the prompt.
• Use the llm-align-skill.md Claude Code skill to automate this workflow end-to-end for any Braintrust experiment.