Cromford

Cromford is the open-source software factory (cromford.dev). It takes tickets off a product board, decides which are actually ready to build, plans them, builds them on a remote machine, reviews the result adversarially, opens a PR, QAs the preview, merges, confirms the deploy rolled, and then checks in production that the thing it merged is really there.

It runs unattended. The design problem it exists to solve is not “can a model write code” — it is what has to be true around the model so that a loop nobody is watching does not open the wrong PR, spend money twice, or tell a customer something shipped when it did not.

The pipeline

One ticket moves through fixed stages. Each has its own runner, its own model, its own budget, and its own idea of what “done” means.

Stages talk through files and a database, not through one long conversation. A stage that dies is restartable because its inputs are on disk.

Merge tiers. A build lands in tier 1 or tier 2. Tier 1 is eligible for automatic merge once review approves and CI is green. Tier 2 waits for a human. flags/automerge-disabled forces every build to tier 2 — bin/run-build.sh:1592.

What holds it together

Deterministic before LLM. Anything a script can decide, a script decides. The gate that says “this ticket has an unanswered question on it” is a status field and a comparison, not a judgement call — so it costs nothing, never drifts, and is testable (bin/triage-gate.sh; on GitHub-Issues projects its fingerprint also carries the open harness PRs, so a merge or a check flip wakes the tick). The model is asked only the questions that genuinely need reading comprehension.

Configuration is data. Which model a stage uses, what a lane may spend, which projects are enrolled, what may run concurrently: rows and registry files, not constants. fleet.conf is the project registry and no script keeps its own copy — tests/fleet-projects-not-hardcoded-test.sh fails if one tries.

A fleet of boxes, plus an optional pool. Each project’s watcher runs on a box; fleet.conf column 6 says which. One box is the hub and runs the cross-project lanes. Builds can optionally be pushed to an ephemeral Fly machine per build instead of running on the watcher — opt-in, and nothing breaks if you never enable it (bin/fly-pool.sh).

Multi-vendor lanes. Every stage names a runner. When the primary is missing, unauthenticated, or fails, bin/agent-call.sh walks a fixed ladder — claude → grok → codex-configured openrouter → fireworks — alerting as it degrades, and refusing to walk it at all when the failure was a budget stop or a kill. The order is pinned by tests/agent-call-provider-order-test.sh.

Every call is priced. bin/agent-call.sh appends a 16-column row to logs/cost.csv for each call. bin/cost-rows-push.sh ships rows to D1 every 10 minutes; bin/cost-rollup.sh builds the per-PR view nightly. A model with no price in catalog/model-prices.json raises a daily alert rather than quietly costing $0.

It learns from its own PRs. Review findings are mined for repeats (bin/rule-miner.sh), and a family of repeats becomes a guardrail under guardrails/ — a rule that fails next time in CI instead of being found again by a reviewer. Nothing in that lane merges or loosens a gate; it only tightens.

Run it on your own accounts

You can run Cromford on your own Cloudflare, GitHub, Fly (or VM) and model accounts. Fork it, copy the settings template, fill it in, and check it:

gh repo fork / --clone
cp env-templates/selfhost.conf.example selfhost.conf
bin/selfhost-install.sh --config selfhost.conf --dry-run

The dry-run prints every resource, env line, spend cap and secret name, and creates nothing. The installer’s --apply step is not built yet, so resources are still built by hand. docs/SELF-HOST.md walks through every setting, the caps, upgrades and the common failures.

Read next

Repeated procedures are repo-managed skills, one SKILL.md per competency under skills/, deployed to every box by bin/sync-skills.sh. A session loads a tested procedure on demand instead of carrying it in every prompt.

Verification contract

Run the suite:

for t in tests/*.sh tests/mc/*.sh; do bash "$t"; done

No network, no secrets, no fixtures to generate. Some suites need jq, semgrep or codegraph and skip cleanly without them; CI sets CODEGRAPH_REQUIRED=1 and SEMGREP_REQUIRED=1 to turn those skips into failures (.github/workflows/harness-ci.yml).

Load-bearing suites — these encode decisions rather than mechanics, and a failure in one is a design regression, not a flake:

Platform. The suite is written for Linux. On anything else, run it in the container. The runner lives in bin/, not tests/, so the loop above never re-enters it:

bin/run-tests-in-docker.sh

That builds a small Debian image with the tools the suite needs and runs the whole thing inside it. It is close to CI but not identical — CI additionally installs semgrep and codegraph and sets *_REQUIRED=1, which turns two clean skips into real checks. Measured on the tree this was written against:

Most of the macOS gap is mapfile, bash 3.2 syntax and GNU-coreutils flags — the platform, not the code. Three suites fail in the container but pass in CI; they are named in the script’s header, and that header says plainly that their cause is not yet understood rather than guessing at one.

The container needs Docker and network for the first build, so the “no network, no secrets” claim above is about the suite itself, not about fetching an image.

Before you push, if you are contributing to this repository, also run bash tests/no-client-identifiers-test.sh. See CONTRIBUTING.md.