Q Evaluation Harness

An open-source framework by KX for evaluating Large Language Models on Q/kdb+ code generation tasks.

This project introduces the first standardized evaluation benchmark for Q/kdb+, addressing a critical gap in assessing Large Language Models (LLMs) for this specialized programming language. The lack of such a benchmark has limited meaningful measurement and progress in Q code generation.

Our evaluation harness provides a robust and rigorous framework, beginning with a Q-language adaptation of OpenAI’s HumanEval. Our roadmap includes integrating additional benchmarks, such as a Q-language port of MBPP, empowering the community to effectively evaluate and advance LLM capabilities for Q.

Model Leaderboard

Track the performance of Large Language Models on Q/kdb+ code generation tasks using our standardized evaluation framework. Scored with the current grader (PR #7).

Agent mode — one task attempt with iterative tool use (top results; q-kdb skill ✓ installed, ✗ clean room):

New: cost per task. Top models now score within a couple of points of each other, so the leaderboard also reports what each one costs to run: total API-equivalent spend divided by the 164 tasks attempted. We use the same measure Databricks uses in its coding-agent benchmark. Rankings are still by Pass@1. Read Cost per task on q-humaneval for what it shows.

One-shot mode — 50 independent samples, no tool use:

📈 View Complete Leaderboard →
Full agent skill A/B, one-shot, and historical (pre-#7 grader) results, with methodology.

---

Features

• 🚀 Simple CLI: One-command evaluation with qeval run .
• 🤖 Agent Evaluation: Evaluate coding agents (Claude Code, Codex) that iteratively write and test Q code.
• 📊 Q-HumanEval Dataset: 164 hand-crafted Q programming problems.