Q Evaluation Harness

An open-source framework by KX for evaluating Large Language Models on Q/kdb+ code generation tasks.

This project introduces the first standardized evaluation benchmark for Q/kdb+, addressing a critical gap in assessing Large Language Models (LLMs) for this specialized programming language. The lack of such a benchmark has limited meaningful measurement and progress in Q code generation.

Our evaluation harness provides a robust and rigorous framework, beginning with a Q-language adaptation of OpenAI’s HumanEval. Our roadmap includes integrating additional benchmarks, such as a Q-language port of MBPP, empowering the community to effectively evaluate and advance LLM capabilities for Q.

Model Leaderboard

Track the performance of Large Language Models on Q/kdb+ code generation tasks using our standardized evaluation framework. Scored with the current grader (PR #7).

Agent mode — one task attempt with iterative tool use (top results; q-kdb skill ✓ installed, ✗ clean room):

New: cost per task. Top models now score within a couple of points of each other, so the leaderboard also reports what each one costs to run: total API-equivalent spend divided by the 164 tasks attempted. We use the same measure Databricks uses in its coding-agent benchmark. Rankings are still by Pass@1. Read Cost per task on q-humaneval for what it shows.

One-shot mode — 50 independent samples, no tool use:

📈 View Complete Leaderboard →
Full agent skill A/B, one-shot, and historical (pre-#7 grader) results, with methodology.

---

Features

• 🚀 Simple CLI: One-command evaluation with qeval run .
• 🤖 Agent Evaluation: Evaluate coding agents (Claude Code, Codex) that iteratively write and test Q code.
• 📊 Q-HumanEval Dataset: 164 hand-crafted Q programming problems.
• 🔧 Multi-Model Support: Supports both closed-source APIs and open-source Hugging Face models.
• 📈 Standard Metrics: Pass@1, Pass@5, Pass@10 with isolated execution.
• ⏱️ Timeout Protection: Code execution with configurable timeout limits.

---

Installation

# Clone the repository
git clone https://github.com/KxSystems/q-evaluation-harness.git
cd q-evaluation-harness

# Install dependencies with Poetry
poetry install

# Activate the Poetry environment (Poetry 2.0+)
eval $(poetry env activate)

Requirements: Python 3.10+, Poetry, and a kdb+ license.

⚠️ Security Warning: This tool executes generated code in your local environment. While we provide timeout protection, the execution is not sandboxed. Only run evaluations with trusted models and in isolated environments. Sandboxed execution is planned for future releases.

---

Setup & Configuration

Mandatory: KDB Setup

1. Install KDB with PyKX license: Standard install KDB or follow KDB-X
2. For multithreaded execution: Set PYKX_THREADING=1 in your environment

🚧 Future: We plan to support MCP/REST API for Q execution to remove the PyKX dependency requirement.

Optional: API Keys

💡 Note: API keys are only needed for proprietary models (e.g., from OpenAI, Anthropic). You can skip this if you are using open-source Hugging Face models.

export OPENAI_API_KEY="your-key-here"
export ANTHROPIC_API_KEY="your-key-here"

---

Quick Start

Verify your installation by running an evaluation on an open-source model. This command should work without any API keys configured.

# Make sure you're in the Poetry environment (Poetry 2.0+)
eval $(poetry env activate)

# Run evaluation on an open-source model from Hugging Face
qeval run q-humaneval Qwen/Qwen2-1.5B-Instruct

---

Usage Guide: Running Evaluations

Use the run command to evaluate models using our standardized framework. This generates and executes Q code solutions in one step.

# Evaluate GPT-4o on Q-HumanEval (default: 50 samples for statistical significance)
qeval run q-humaneval gpt-4.1

# Evaluate an open-source model
qeval run q-humaneval google/gemma-3-4b-it

# Specify custom sample size (50 samples recommended for leaderboard submissions)
qeval run q-humaneval your-model --num-samples 50

📊 Evaluation Standard: Use 50 samples per problem for statistically significant results and leaderboard submissions.

---

Agent Evaluation

Evaluate coding agents that can iteratively write, test, and fix Q solutions. Unlike standard evaluation (which generates code in a single pass), agent mode gives models access to a Q interpreter and lets them iterate on their solutions.

Supported Backends

Running Agent Evaluations

# Evaluate Claude Code with Opus 4.7 (use --timeout 600 — opus iterates a lot)
qeval agent-run q-humaneval --backend claude-code --model claude-opus-4-7 \
  --timeout 600

# Evaluate Codex with GPT-5.5
qeval agent-run q-humaneval --backend codex --model gpt-5.5

# Install a skill (e.g. q-kdb) into agent workspaces — strongly recommended
qeval agent-run q-humaneval --backend claude-code --model claude-opus-4-7 \
  --skill-dirs path/to/claude-skills/skills/q-kdb

# Capture the agent's full event stream for auditing skill activation,
# tool calls, and reasoning. Saved as /events.jsonl.
qeval agent-run q-humaneval --backend claude-code --model claude-opus-4-7 \
  --save-events --keep-workspaces

# Override the built-in workflow instructions with your own (the default
# instructions already cover skill loading, parse-checking, and verification).
qeval agent-run q-humaneval --backend claude-code --model claude-opus-4-7 \
  --agent-instructions path/to/custom-instructions.md

# Customize concurrency and timeout
qeval agent-run q-humaneval --backend claude-code --model claude-opus-4-7 \
  --concurrency 10 --timeout 600

# Run specific problems only
qeval agent-run q-humaneval --backend codex --model gpt-5.5 \
  --problem-ids 0 1 2 3

# Keep workspaces for debugging
qeval agent-run q-humaneval --backend claude-code --model claude-opus-4-7 \
  --keep-workspaces

Agent Skills

The --skill-dirs option installs Agent Skills into each agent workspace and instructs the agent to read them. Each path should point to a skill directory containing a SKILL.md file. Skills are installed into both .claude/skills/ (for Claude Code) and .agents/skills/ (for Codex), so the same --skill-dirs argument works with any backend.

Note: In headless / non-interactive contexts, agents do not reliably auto-load skills based on description matching alone. The harness’s built-in workflow instructions (CLAUDE.md / AGENTS.md) explicitly tell the agent to read q-kdb/SKILL.md before writing any Q code. If you install skills under a different name, update the instructions accordingly with --agent-instructions.

For Q evaluations, point --skill-dirs at any skill directory whose SKILL.md covers q syntax, common errors, shell-running idioms, and Python→Q translations. The qdex plugin previously bundled similar guidance under /qdex:code.

Agent Leaderboard

Scored with the current grader (PR #7). The Skill column marks whether the q-kdb skill was installed (✓ via --skill-dirs) or the clean-room --no-skills baseline (✗).

Cost/task is total API-equivalent spend (list prices, including prompt caching) ÷ 164 tasks attempted, counting each task’s final attempt only. It is computed from each run’s token counts at published prices, not taken from the CLI (see ‖). Haiku 5.5 rows show four decimals because its cost per task is under a cent. See Cost per task on q-humaneval.

The skill no longer pays for itself on frontier Claude models. Fable 5 was the first to show it: 158 clean room against 155 skilled in June 2026 (exact McNemar p = 0.45). On Opus 5 it lost 1.9 points while raising cost per task 1.53x; on Opus 4.8 it gained 1.2 points for 2.1x the cost. At n=164 the Opus 5 accuracy gap is not statistically significant, so read it as no demonstrated benefit rather than active harm. The cost increase is not in doubt. The GPT models are different: the skill lifts GPT-6 Sol by 3.7 points (95.1% vs 91.5%) for only 11% more cost per task, and GPT-5.5 by a similar margin. Haiku 5.5 sits between the two cases: the skill added 3 to 5 tasks in each of its three skilled arms (default, medium and high effort) for 6% to 21% more cost, but no single gain is statistically significant (best case +5 tasks, exact McNemar p = 0.18). Sonnet 5.5 gained nothing from the skill (157 vs 158, at 2.6x the cost per task, because the skill’s roughly 15k extra context tokens are written to the cache on every task), so the Haiku 5.5 versus Sonnet 5.5 comparison holds with or without it. Opus 5.5 has no skilled run. Explicit effort made no detectable difference: the three skilled arms span 157 to 159 of 164. Tasks 92 and 116 fail in every Haiku 5.5 arm.

Across six months of OpenAI flagships (GPT-5.5 in April, GPT-5.6 Sol in July, GPT-6 Sol in September), clean-room scores held at about 90% while cost per task fell 65%, from $0.225 to $0.080. Almost all of that drop is lower token prices; each model uses about as many tokens per task as the last.

* Reference rows — prior overnight runs re-graded with the new grader (different machine, ~600s timeout with cap-resume → cost/turns omitted; pre-date --no-skills, so reported as skill ✓, except Fable 5, which was run both ways in June 2026 and is the first model on the board to show no benefit from the skill; earlier versions of this board listed its clean-room run’s first pass, 156/164, as skilled). † The April GPT-5.5 skilled run pre-dates Codex cost capture. ¶ OpenAI runs, Oct 2 2026: Codex 0.160.0, --reasoning-effort high, --timeout 600; cost computed from token counts at OpenAI list prices; the October GPT-5.5 clean-room run replaces April’s (145/164, no cost). Mean turns are not comparable across backends: Codex counts every action, Claude Code counts model round trips (see the full leaderboard). ‡ Opus 5 rows required manual re-runs after a harness issue silently terminated some agent processes mid-task; affected tasks were re-run and every task in both arms is verified attempted — see OPUS-5-NOTES.md. § Fall 2026 runs (Sept 30 sweep; Fable 5.1 on Oct 1): clean room, --timeout 600, Claude Code 2.1.286, except the skilled Sonnet 5.5 row (Oct 8, Claude Code 2.1.287, otherwise the same setup); Claude Code applies per-turn effort to the 5.5 models. Fable 5.1 hit the 5-hour usage cap mid-run; 31 tasks were deferred and run after the reset, and cost counts final attempts only. Sonnet 5.5’s cost was corrected from $7.82 ($0.048 per task) to $6.78 ($0.041): Claude Code 2.1.286 billed its cache reads at twice the published rate. ‖ Haiku 5.5 runs, Oct 8 2026: Claude Code 2.1.287, --timeout 600, concurrency 4, all 164 tasks attempted. Rows without an effort level use Claude Code’s default; “medium” and “high” pass an explicit --effort. The CLI does not recognize Haiku 5.5 and reported about 32x the real cost, so cost is computed from token counts at its published prices (up to 100k-token prompts: $0.10 input, $0.50 output, $0.01 cache read per million; longer prompts cost more, and no final-attempt request reached that length). See the full leaderboard for caveats, the GPT-5.5 cross-check, and what’s next (a skilled GPT-5.6 Sol run).

Agent mode results are not directly comparable to the one-shot leaderboard — agents get a single task attempt but can iterate with tool use, while one-shot evaluation generates 50 independent samples per problem.

How Agent Evaluation Works

Each problem is evaluated in an isolated workspace containing:

• The problem prompt and function signature in problem.md
• A starter solution.q with the function stub
• A CLAUDE.md (Claude Code) or AGENTS.md (Codex) with the default workflow instructions, optionally augmented or replaced via --agent-instructions
• Agent skills in .claude/skills/ and .agents/skills/ (if --skill-dirs is provided)
• Access to a Q interpreter for testing

The agent writes its final implementation to solution.q, iterating until it considers the work done or the per-task --timeout is hit. Note: neither the Claude Code CLI nor the Codex CLI enforces a turn cap, so --max-turns is a soft warning threshold only — --timeout is the enforced backstop. Results include wall time, token usage, cost, and the actual turn count per problem.

When --save-events is set, each workspace also gets an events.jsonl file containing the agent’s full event stream (tool calls, model thinking, command results), plus an events.stderr.log for the CLI’s debug output. These are useful for verifying which skills the agent actually loaded and where it spent its turns.

---

Submission Guidelines

Help us grow the leaderboard! Submit your model evaluation results to contribute to the Q/kdb+ AI development community.

📋 Complete Submission Guide →

💬 Questions? Use GitHub Issues →

---

Project Reference

Dataset: Q-HumanEval

Our primary benchmark adapts HumanEval to Q/kdb+, featuring 164 problems with hand-verified solutions and comprehensive test cases.

Contributing

We welcome contributions! Key areas include new datasets, model integrations, and evaluation metrics.

# Development setup
poetry install --with dev
pre-commit install
# Run tests
poetry run pytest

Roadmap

• [x] Agent Evaluation: Evaluate coding agents (Claude Code, Codex) with iterative tool use.
• [ ] Q-MBPP: Basic programming problems in Q.
• [ ] Custom Metrics: Flexible framework for adding custom evaluation metrics beyond Pass@k.
• [ ] Sandboxed Execution: Secure, isolated code evaluation environment.
• [ ] MCP Server for Q: Support MCP/REST API for Q execution to remove the PyKX dependency requirement.
• [ ] Native Q Execution: Remove the PyKX dependency.

---

Troubleshooting

vLLM Issues

Problem: vLLM (NCCL library) fails with NVLS-related errors on single-node setups.

Solution: Disable NVLS to force fallback:

export NCCL_NVLS_ENABLE=0

---

Citation

@software{q_evaluation_harness,
  title={Q Evaluation Harness: Benchmarking LLMs on Q/kdb+ Code Generation},
  author={Miahi, Erfan and Morrison, Andrew},
  year={2025},
  url={https://github.com/kxsystems/q-evaluation-harness}
}

---

License

MIT License - see LICENSE for details.