SOTA Engineering Skills

Make your AI coding assistant build and audit like your most senior engineer.

Your assistant is brilliant — it just doesn’t know your standards, and it forgets the ones it does know as the task grows long. SOTA-skills fixes both, and the fix is measured: from a bare “build X” prompt, best-practice coverage climbs from ~59% to ~98% (+0.39) — the model stops silently dropping tests, rate limiting, structured logging, and TLS (see every number →).

It works by being a loop, not a prompt dump: route in only the rules a task needs, re-state them every turn, and re-check them last before shipping — so the guidance survives a long context instead of fading into it. That’s why it beats a bigger prompt instead of becoming one. Native on Claude Code; works with Antigravity CLI, Codex, and any agent that reads AGENTS.md. Those agents get the skills and an always-on directive, but not the per-turn re-statement, which is a Claude Code hook (per-agent detail), and no lift has yet been measured through them.

Under the hood: 43 skills (347 files, ~89k lines) of state-of-the-art 2026 practice, each instruction file under 500 lines so only the matching rules load — the cap applies to skills/** alone, never to README/CHANGELOG/docs/ — every fast-moving claim web-verified against a primary source as of the last full sweep (the root LAST-VERIFIED stamp). Between sweeps an upstream can still move under a claim, so the router’s own operating principle 1 stands: re-verify anything version- or CVE-sensitive at use time.

Two commands to install:

/plugin marketplace add martinholovsky/SOTA-skills
/plugin install sota-skills@sota-skills

Or clone + link (best if you want a local checkout to read, hack on, or pin). Skills are discovered from .claude/skills/ (per project) or ~/.claude/skills/ (personal, all projects). Clone the repo, then run the installer — it symlinks every skill (and your profile, if you have one):

git clone https://github.com/martinholovsky/SOTA-skills && cd SOTA-skills
./scripts/install.sh                 # personal: ~/.claude/skills (all projects)
./scripts/install.sh --project DIR   # one project: DIR/.claude/skills
./scripts/install.sh --target all    # also ~/.agents/skills (Copilot, Codex, Cursor, Gemini CLI) + Antigravity's own dirs
./scripts/install.sh --copy          # copy instead of symlink (pin a snapshot)
./scripts/install.sh --mcp           # optional: register the read-only MCP server for other agents

The installer colour-codes what it did (✓ done · ↻ changed or act on this · · no-op) and drops to plain ASCII when the output is not a terminal, on a non-UTF-8 locale, on TERM=dumb, or with NO_COLOR set — --color=always|never|auto (or --no-color) overrides the detection either way.

Then describe the task in plain language — routing loads the right skills; the stack comes from your profile or the skills’ defaults (naming one is optional):

Design a multi-tenant invoicing service.

Run a full audit of this repo — severity, effort, and fix on every finding.

More install options: Installation · more prompts: Using it.

Five slash commands ship with the library — four bracket a working session, the fifth is the heavy escalation. scripts/install.sh links them into ~/.claude/commands/, so they work in any project, not just this one:

They take free-text arguments (/sota-close focus on the migration branch). Full detail: slash commands.

Contents

• Standards & practices baked in · What the audit hunts that a scanner can’t · How the numbers are kept honest
• Skills · Coverage & non-goals
• Installation · Always-on routing · Updating
• Using it — slash commands (/sota-resume · /sota-audit · /sota-deep-audit · /sota-close · /sota-report)
• Optional setup & integrations — badge, gates, other agents, status line, plugin extras
• Structure · How it works · Conventions
• Found a gap? Tell us · Contributing · License

Deeper docs: Find it fast (docs index) · Does it work? (measured results) · Why it works · Why some lifts expire and others don’t · Keeping rules applied as context fills · Roadmap · Agent-loop evals — tools, hidden tests, no model judge (separate repo)

Standards & practices baked in

Findings name the control they violate — not just “this looks wrong”:

• Security — OWASP Top 10 (2025), ASVS, API & LLM Top 10; findings cite CWE IDs. Disabled TLS certificate and SSH host-key verification are each stated once, with a detector per library read from that library’s own source
• OWASP, item by item — 19 OWASP projects compared with the library one item at a time (7,439 items, 2026-09-25); every one of the 847 gap themes was adopted, found covered, or recorded with a reason (ADOPTION-LOG). New ground includes anti-automation and abuse of value-granting flows, browser response hardening (Fetch Metadata, HSTS rollout), file uploads, SBOM content, dependency adoption, vulnerability remediation beyond the score, tag manager governance, the JVM web layer, and audit reports that carry reproduction steps and a finding lifecycle. Where a rule is detectable in code or config it ships an audit probe tested on a known-bad and a known-good fixture
• Languages — all 9 language skills (Rust → Ruby, below) get the same rigor; formal standards where they exist: SEI CERT (C, C++, Java), MISRA C/C++, ANSSI Rust. Each is also gap-checked against its ecosystem’s own scanner registry (all nine, by 2026-09-24 — LANGUAGE-TIER)
• Supply chain — SLSA, Sigstore, in-toto, SBOM (CycloneDX/SPDX), NIST SSDF
• Cloud & identity — CIS Benchmarks, NIST 800-207 zero trust, NIST 800-63-4, OAuth 2.1, FAPI 2.0, passkeys, SPIFFE
• Privacy & compliance — GDPR, CCPA/CPRA, HIPAA, PCI DSS 4.x, SOC 2, ISO 27001, EU AI Act, NIS2, DORA
• Government & regulated — NIST CSF 2.0, 800-53, 800-171/CMMC, FedRAMP, EU Cyber Resilience Act, IEC 62443, ISO/IEC 29147 & 30111 (vulnerability disclosure and the handling process behind it)
• Threats, detection & AI/ML — STRIDE, LINDDUN, MITRE ATT&CK & ATLAS, NIST 800-61, NIST AI RMF
• Frontend, mobile & testing — WCAG 2.2 AA, Core Web Vitals, OWASP MASVS & WSTG

Named standards are the floor. Most of the library is the practice layer no regulation writes down: cancellation & backpressure, retries with jitter, circuit breakers, outbox/saga, double-entry ledgers and the reconciliation that proves an integration is complete rather than merely correct, zero-downtime migrations, measure-first performance, API evolvability, per-language idioms, SLOs, test-suite health — and the ones that only show up once something is already running: triaging a vulnerability report a stranger sent you (reproduce before you rate it, check the cited code even exists, and decide the regulatory clock explicitly, because it starts at awareness), and why a skill can stop triggering with nothing about it changed (a length cap is enforced either by skipping or by silent truncation, and a shared listing budget means a neighbour’s bloat eats the trigger words you wrote).

Measured, not asserted — library vs. an unguided model (same model, no library); clean, blind-judged, stable across samples (results & method →). Every number below names the model it was measured on, because a lift can be overtaken by model progress and then reads as current when it is historical — that happened to the defect-avoidance row on 2026-08-21 and is stated there rather than quietly left standing:

• Completeness +0.39 — and it did not expire. Re-measured 2026-08-21 on the current flagship claude-sonnet-5: 0.62 → 1.00, +0.38 — unchanged. From a bare “build X” prompt, best-practice coverage goes ~59% → ~98% (7 tasks): the model stops silently dropping tests, rate limiting, structured logging, and TLS. Web search likely can’t recover this (an agent won’t search “should I add rate limiting”). Not model-specific, and not stale: a different-family model (openai/gpt-5.1) shows +0.44 on the same tasks (cross-model →), and claude-sonnet-5 — four months newer than the model this was established on — still shows +0.38 (2026-08-21 →). Why this one lasted while defect-avoidance did not: newer models stopped writing known-bad patterns, but sonnet-5 unguided still omits tests in 7 of 7 tasks, transport in 5, rate limiting in 5. Knowledge gaps close with model progress; salience gaps do not.

• How much of that is just asking? Most of it — not all. A pre-registered placebo arm with no library, told only to “ship production-complete code and self-review it”, reaches 0.86 (2026-10-08, sonnet-4.6): 69% of the lift. The library’s own +0.125 sits in the controls a model leaves out even when told to be thorough: transport enforcement, rate limiting, CSRF, request size limits, session invalidation. On generic completeness, asking nearly suffices (PLACEBO →).

• Freshness +0.53 → +0.30 — it erodes, and we measured that rather than assuming either way. Current-2026 facts (RFCs, CVEs, EOLs) 0.44 → 0.97 on claude-sonnet-4.6, where an unguided model is confidently wrong. Re-measured 2026-08-25 on claude-sonnet-5: 0.69 → 0.99, +0.30. The pre-registered prediction was that a training cutoff is a gap model progress cannot close — refuted: the newer model answers 8 of these facts unaided that its predecessor could not, because its cutoff advanced into a fixed case set. Ten of 32 remain outside it, and that is the surviving lift. So freshness is a third shape — neither expired like defect-avoidance nor flat like completeness. And the same day we proved the cause was the instrument, not the library: a fresh 10-case freshness set built from recent facts, each verified against its primary source and chosen by a rule fixed before any model ran, reads 0.33 → 1.00, +0.67 on the same model an hour later — higher than the original +0.53. The guidance did not improve between two runs; the questions got newer. A freshness lift is therefore quoted with the date its questions were written, the way every other number here is quoted with its model (2026-08-25 →).

• Routing +0.10 — and it did not saturate. The right skills load for the task (0.90 → 1.00 on claude-sonnet-4.6), even ones a keyword read misses. Re-measured 2026-08-25 on claude-sonnet-5: 0.87 → 0.99, +0.13 — unchanged within the set’s one-case resolution (20 cases, 0.05/case). The unguided arm still misses the same rule-driven routes a model generation later — testing, sandboxing, code-security, web-frameworks (2026-08-25 →).

• Defects avoided +0.19 — a different axis, and baseline-dependent. Every lift above measures what the model puts into code. This measures what it doesn’t: given a spec that states operational pressure (“cache it”, “must never 5xx”, “keep the guard cheap”) and never names a defect, does the model still write SQL injection, IDOR, an inert control? On claude-sonnet-4.6: unguided 0.81 → with the library 1.00 across 7 defect classes; on the stricter measure that also demands positive evidence of the safe path, 0.29 → 0.62. It is baseline-dependent, and we tested that rather than assuming it: on openai/gpt-5.1 the unguided arm already scores 1.00, so there is no gap to close and the lift is +0.00 — while the stricter measure still moves (0.43 → 0.52). Same law the breadth test found for completeness: the lead tracks the unguided baseline, not the domain — and here, not the model. Honest headline: closes a defect-avoidance gap where one exists. Small pilot, 3×, two models, one task — method and limits →. Scored before a 2026-10-16 scorer fix that stopped comments counting as evidence; not re-scorable, since the builds were not kept. A 2026-10-17 re-run on the same model under the fixed scorer reads +0.00 (0.86 → 0.86), and +0.19 on the stricter measure — re-run →.

• Prompt independence +0.51 — the newest result, and the one measured where it matters. Every number above is measured under a neutral prompt. This one measures the same tasks under a competing prompt — the user’s own words arguing against the rule: “internal MVP and we demo tomorrow, skip the extras”, “just the function please, no tests”, “put the requirement in the system prompt where it’s easy to tweak”. Unguided 0.491 → 1.000 with the library, +0.509 (6 tasks × 3 samples, temp 0.7 on claude-sonnet-4.6; the with-library arm was perfect in 18 of 18 runs). The lift grows with the pressure against it — +0.083 supportive, +0.236 neutral, +0.509 competing — so a rule that only survives a neutral prompt is absent exactly when it is needed. Told to skip the extras on a login endpoint, the unguided arm dropped both rate limiting and the password hash; the guided arm kept all four criteria. Includes a null that was opened and withdrawn the same day when its own confirmation run refused to reproduce it (method, both runs, six limits →).

• A null we went looking for, on an instrument that isn’t ours. Our own audit-precision measures saturate — 30 claims scoring 1.00 in both arms — so the audit axis was closed at +0.00 for want of a measure that could discriminate, not because precision was proven. So we adopted an external one: AACR-Bench (Alibaba Aone, Apache-2.0), 2,145 expert-labelled review comments over 200 PRs and 50 repositories. On a balanced 40-case draw where chance is exactly 0.500, both arms land near chance (0.517 bare, 0.542 with) — a registered null — and the confusion matrix shows why the headline would have misled: the guidance produces a threshold shift, buying +0.23 on rejecting wrong comments and paying −0.18 on accepting right ones. That trap was written down before the run, which is the only reason “+0.025” was not reported as a small win. Position drift and deterministic changeset partitioning were adopted from the same source as rules.

How the numbers are kept honest — the same discipline the library teaches. Every prediction is committed before the run, with the result that would falsify it, so a refuted one is published rather than quietly re-framed (2026-08-25: a freshness prediction was refuted and the write-up leads with it). And case sets are built by a rule fixed before any model runs — never from the cases a model got wrong, which is selection bias and guarantees whatever gap you go on to report. That rule is now part of the library itself (sota-llm-engineering rules/01 §8), because the temptation is real: when an ageing set stops discriminating, the still-failing cases are sitting right there.

And not just vs. an unguided model — head-to-head against the most popular guidance libraries on backend build tasks, SOTA-skills leads on completeness (content-only, blind-judged; wins or ties all 21 cases, loses none):

A five-domain breadth test shows when this edge holds: SOTA-skills leads the field wherever a base model ships incomplete code — production backend in any language and complex/security-sensitive frontend (~+10 pts) — and ties where the base model is already near-complete (simple UI, templated infra). The lead tracks task difficulty, not the domain — we measure it and say so. Full breadth result, consolidated table, method & honest limits →

And a result we will not dress up. Two outside assessments named obra/superpowers as the methodological competitor, so we ran it (2026-09-08, pinned SHA, 7 cases): SOTA 0.987, unguided 0.637, Superpowers 0.599. In the chart above it sits at 60% against an unguided band of 58–64% — the same arm measured in the two runs, since temp-0 is not deterministic — so it lands inside the baseline rather than above it. The band is one row on purpose: an earlier version drew two “unguided model” bars and, though accurate, it read as a bug. That is −0.04 versus no guidance at all, against a ±0.03 noise floor — so on this rubric the two are indistinguishable, and the honest reading is about the instrument, not the project. Superpowers is process guidance; these cases score a built artifact’s domain coverage. A low score is evidence about fit to this measure. It shows the two libraries are not substitutes; it does not settle whether a methodology beats a corpus, and we have no instrument that would. Method and limits →

What the audit hunts that a scanner can’t

Thirty classes of defect survive every linter, SAST rule, and CVE scanner, because in each one the code isn’t wrong. The library hunts them as explicit passes (this said “Eleven” for seventeen days — correct when written on 2026-08-27 and fifteen classes behind by the time anyone recounted at the v1.41.2 cut. Recount the list, never restate this number: the first correction written here was itself stale within the hour, because the same change added three more classes below):

Finding them is the cheap half — and we can prove it. Across nine instruments, a frontier model recognises these classes unaided: audit lift +0.00, published below rather than buried. The expensive half is not writing them in the first place. On a model that does write them, the library stops it: 0.81 → 1.00. On one that already doesn’t (gpt-5.1, unguided 1.00) there is nothing to close — we ran that second model and report the +0.00 too (2026-08-21).

Qualified 2026-09-12, against ourselves. “Recognises these classes unaided” was measured on instruments that scored 1.00 in both arms — which is what a measure looks like when it cannot tell the arms apart. On an external one that can (AACR-Bench), judging whether a real review comment is correct puts a frontier model near chance in both arms. The audit lift is still +0.00, now across ten instruments; what is not supported is the stronger reading that finding these things is easy.

• Controls that are inert — a safeguard whose success and whose total failure look identical from outside: a swallowed enforcement exception, a ruleset that loads zero rules, presence decided by exists() rather than a loaded artifact, a CI gate whose every run is skipped (and on GitHub a skipped job reports Success to branch protection), a policy engine left in Audit/warn/report-only since the day it shipped, a report whose word “verified” traces to no line that can fail, silent truncation on the way out of a generator — an unset token cap returns a fragment that a swallowed parse error publishes as a valid empty result — a test that still passes when the control’s body is replaced with a no-op, a write-back controller logging updated=1 errors=0 every cycle for fifteen minutes while pushing no commit at all — its log describes the update it decided on, not the write landing. The same shape one layer down: a count that is computed and still false, because it was derived from an intermediate the function discards and emitted before the return that dropped it. No function can attest to its own return value — every emission site has a suffix that can drop the result after the line is written — so the claim belongs in the consumer, derived from the value received. Where a job runs unattended, its log is the only witness, so the probe is to change what the function returns and read the output, not to re-run the tests. (rules/10, rules/14, kubernetes rules/04 §7)
• Pins nothing watches, and watchers that can never report — a version embedded in a build-tool invocation (RUN xcaddy build v2.11.4 --with [email protected], a Bazel arg, a go install tool@version) is unwatchable by construction: Renovate sees a RUN line, and Dependabot has no regex manager at all — so pinning it converts an unreviewed drift into an unreviewed freeze, and both states are silent. The usual answer is a bespoke watcher, which then inherits the publishing conventions of whatever it watches: one querying releases/latest takes a permanent 404 from a project that ships git tags and no releases, skips that entry every run forever, and leaves a list that still reads as complete. Same family, one layer down: in Go require is a floor, not a ceiling — under Minimal Version Selection the graph’s highest requirement wins, so “we pinned it” is simply false for anything another module also requires. (devsecops rules/03, rules/09, golang rules/07)
• The agent’s own tooling as an attack surface — three surfaces the ordinary checks miss. A coding agent invoked in CI is a CI actor holding your token: it takes instructions from whatever text it is handed, and the usual env: fix for expression injection is a shell defence that does nothing for a sink which interprets the value — it just removes the ${{ }} reviewers grep for. The developer’s agent session transcript is a credential store nobody inventories, holding whatever the harness loaded, including files it read on its own initiative and mislabelled. And any tool that sweeps such text must never persist raw: store a digest plus a structural skeleton, because shape-based redaction is an enumeration and a bare 64-character hex key has no shape to match. (devsecops rules/01 §1.5a, secrets rules/04 §7, secrets rules/03 §2.1)
• Controls the compiler removed — a safeguard that is correct in the source and absent from the binary. Wiping is the known case (memset is dead-store-eliminated); the same applies to constant-time arithmetic, where strength reduction is an optimiser courtesy, so constant time is a property of the emitted code, verified across every architecture and optimisation level you ship — -Os and -Oz included. Two instruments answer it, and neither stands in for the other: Valgrind memcheck taint flags every branch on the secret, and dudect’s t-test times the running binary. (code-security rules/22 §1.1)
• A signing step that picks its input by name. On a public repo a fork’s pull_request run uploads into the base repo’s artifact store, so “sign the newest artifact named X” can sign a fork’s bytes — and the REST list reads the producer as null, so a provenance filter over it passes everything. (devsecops rules/02 §2.5.2)
• A defect that starts where the gates stop. Formatters, linters and most SAST end at the compiled artifact; a loader, verifier, dynamic linker, capability check or admission controller runs after it, and that whole depth is unprobed while every gate is green. Four gates passed an eBPF object the kernel then refused to load, because a warn-by-default style lint had suggested an owned argument that spent 52 bytes of a 512-byte stack — on a constrained target a lint’s premise can be false, and the word “mechanical” is a classification applied before the analysis that would justify it. A gate can also stop short by default: ShellCheck does not report an unset ALL-CAPS name, so a set -u typo passes the lint and bash -n and dies at run time; the optional check check-unassigned-uppercase reports it, and this repo’s CI enables it. (devsecops rules/09 §2a, shell rules/03 §7, rust rules/07 §1a)
• A version recalled instead of looked up. Not a stale pin — a measurement of someone else’s artifact, which never feels like a version decision. A recalled release number arrives with no felt uncertainty, so every rule that triggers on doubt is structurally unable to fire. The fix is a fact that cannot be recalled: record the EOL date beside the version. A row past its EOL is removed, not corrected — an unsupported branch can answer the opposite of the current one, not merely a staler version of it. (devsecops rules/03 §3.9)
• A finding sourced to whoever sells the remedy. Security claims arrive disproportionately from vendors of the fix, and such a source is not disqualified by that — it is unblinded by it. Independent sources converging is evidence; sources converging on the conclusion that sells their own product is one hypothesis held by several interested parties. Name the interest in the finding, prefer a test you run yourself, and label it a design argument rather than a measurement until someone runs it. (sota/rules/03 §2)
• A green light from the wrong stage. A pipeline with more than one stage has more than one success signal, and the earliest is the one that looks like a summary — it is a count and it renders first. A metrics collector reporting 1 target had matched a selector and gathered nothing for days; the very next field read health=down. Discovery is not collection, and the same split is enumerate-vs-process, connect-vs-authenticate, resolve-vs-fetch, schedule-vs-execute. Accept the downstream artefact — the series, the row, the record — never the pipeline’s own readiness. And prove a repair with an output delta, not a green light: in that case three affirmative signals were already green while nothing was collected, and only 0 → 54 series told the two states apart. (rules/14 §4b)
• A safety net deleted because nobody had needed it yet. The deletion proof for a dependency is remove it and the build still passes. For anything whose job was to be available rather than called, that proof returns the same green either way, because being unreached is what a healthy fallback looks like. A cache, mirror, warm standby or break-glass path needs a second piece of evidence — the successor named, and measured working now. Field case: the replacement mirror was 3.3 hours fresh, so the deletion was right; had it been stale the correct action was the opposite one. (rules/10 §3a)
• A status that is true, about the wrong subject. Not a wrong answer — a correct one describing something other than what the reader assumed, which no amount of care applied to the value can catch. Field-reported, six in one session, every one literally true: “exit code 0” was the wrapper’s and arrived 17 seconds into a 41-minute run; “no advisories for this package” was true of the database that was opened, which had zero rows; “the harness returned no result” was true of a container that never started, and the alternative reading was a conclusion about the target; “38,861 passed” was true of the tree fifty minutes earlier. The rule is one clause long: when a check reports OK, name what the OK is about. (sota/rules/03 §2)
• The ad-hoc command you typed to check something. Unlinted, unreviewed, run against the system under test — and it can report a false absence or destroy what it was inspecting. A sweep reads as clean because the searcher silently excludes things: measured on one tree with four matches, rg defaults found 1, a grep that turned out to be a wrapper function found 3, explicit flags found 4. And a verification command that copied 40 GB into a container filled the host and corrupted the runtime — the check never ran. A count taken from a listing tool has the same shape without any bug at all: gh pr list answers with the first 30 of 330, exit 0 and empty stderr, because a default page size is policy rather than a defect — so a result whose size equals the cap is a page until proven otherwise. Blast radius is not only disk, either: an unbounded backgrounded loop, or a wrapper that invokes a command its own directory shadows, fills the per-user process table, and every tool fails at once including the ones you would use to diagnose it. Two unrelated causes produce that one signature and only parentage separates them. Two more shapes where the command is correct: an extraction or fetch into a blob you have never opened cannot be positive-controlled at all, because nothing is yet known to be in it — so it must print a denominator (bytes, members, total files) beside the zero, or 0 reads as a fact about the subject when tar simply could not open the archive. And the selector can name a different population member than the question did: sort -V | tail -1 is the newest, not the default, and returned a backports kernel for a release that ships another one — nothing truncated, exit 0, no rule broken. And a sweep that writes can destroy what it inspects in a way git status shows as one character: an in-place sweep (perl -pi, sed -i) across a path list silently replaces a symlink with a regular file — exit 0, target untouched. Measured across four implementations, and the danger is the platform split: only BSD sed refuses, so a macOS developer sees the safe behaviour and CI (GNU, BusyBox) does not. That one-liner converted this repository’s own tracked symlinks. And a guard tested in zsh can pass what bash rejects: zsh leaves BASH_REMATCH unset, so empty -eq empty accepted 43/44 as “N/N checks passed”. (rules/06, rules/03 §3a)
• Controls that block everything — the mirror image, and the one every other pass here looks past. An enforcement control (cap, quota, filter, allowlist, sandbox policy) can be tightened until it refuses the legitimate case too, and it passes the same tests: a security suite asserts refusal, refusal is exactly what an over-tight control produces, and the mutation probe above only ever installs the permissive no-op, so it looks in one direction. The fix is an allow arm — a representative legitimate case that must complete through the control, never against the bare environment, because proving the machine can do the work says nothing about whether your control permits it. Worked instance: a memory budget set with RLIMIT_AS refuses the runaway allocation exactly as intended, and also kills every Go or JVM process at startup, since those runtimes reserve gigabytes of address space they never touch (measured: 1.17 GiB reserved against 2.3 MiB resident). Its host-hardening twin: forcing AppArmor’s shipped flags=(unconfined) profiles into complain mode broke rootless Podman, because those profiles exist to grant a permission, not to skip a policy. (rules/12 §1a, sandboxing rules/02)
• Layers that were never layers — the same test applied to an architecture diagram rather than a function. A second-opinion classifier drawn from the same model family as the system it guards shares its blind spots by construction (common-cause failure), and one that only ever sees what the primary already flagged uncertain cannot, even in principle, catch what the primary got confidently wrong. A TEE bought to fix records that were never emitted is answering a liveness question no hardware guarantee covers. Both are counted in a threat model and neither reaches the case it is counted for. (rules/08 §1, rules/18 §1)
• Stages that report success while doing nothing — found by the cheap signals rather than by reading every line: a step returning “nothing found” far faster than its claimed work allows, a gate that never prints how many items it examined (0 checked, 0 failed, exit 0), a size-gated branch no fixture crosses, a cache key narrower than the behaviour it gates, a control written as an assert that -O/NDEBUG/a missing -ea deletes in production. When a tool’s correct output cannot be stated at all — the test oracle problem, which is why “it found 0” goes unchallenged — a metamorphic check pins how the output must change: a fixture with N known items, and an assertion that the count moves when the input does. (rules/11, rules/13)
• A field that is always empty, and the seam nobody could have diffed. A defaulted read — .get(key, default) on a dict you did not construct — turns a key-name disagreement into a plausible constant, and when the producer is a model choosing field names from a distribution there is no change event to test against: the prompt and the reader can both be correct while the response uses a synonym. The only sound detector is at runtime — diff the keys the model returned against the keys anything consumed, and log the ones nothing read. (rules/13)
• A search that never ran, reported as a clean tree. In zsh — macOS’s interactive shell, and the one these checklists get pasted into — an unquoted glob in a flag value (grep --include=*.md) trips the default NOMATCH and aborts the command, where bash would pass the word through and run it. Its stdout and exit status are those of a real no-match: empty output, exit 1. zsh’s no matches found still reaches stderr (the command’s own 2>/dev/null does not cover it), so a script reading $out or $?, or a sweep with stderr dropped around it, reports a clean tree. The sweep you read as “nothing is stale” may never have executed. A second route reaches the same false-clean: in zsh an unquoted expansion does not word-split, so files=$(git ls-files '*.md'); grep -l X $files passes the whole newline-joined list as one impossible filename and it searched nothing — and the remedy is separator-specific, since the ${=var} fix for a space-separated flag string is itself wrong for a file list. Found the hard way in this repo, where the rule was already written and the skill was never loaded: a routing gap, fixed in the trigger rather than the text. (rules/01 §3 and §3a)
• A watcher that cannot say “the thing I was watching is gone”. Done/not-done cannot express “I could not tell”, and once you add that you find you also need GONE — terminal and knowable, not unknown. Collapse it and the watch either invents a success or never ends. It is the row people delete while fixing the other bug: one rewrite replaced an explicit “no longer exists” branch with a blindness counter, which then reported “cannot read for 20min” about a job that had simply been garbage-collected. (rules/15 §2.2a)
• A measurement whose rows came from somewhere else. When tests and production write to one shared sink, every aggregate over it merges two populations — and the danger is not the false positive but the destroyed true finding, because the contaminated number carries the larger n and so reads as the more rigorous one. (rules/11, rules/05)
• Your own scorer, gate or benchmark doing none of the above — a whole file turns the lens around: anything whose output decides whether something is OK is a control too. A broken feature produces a complaint; a broken instrument produces a number, and numbers get quoted. So give it a known-bad input it must fail and a known-good one it must pass, bound what it reads, and never trust a number from an instrument you haven’t watched produce a wrong answer on purpose. The sharpest case is a guard that is an instance of what it guards — a coverage test whose scope is narrower than the population and whose predicate the defect satisfies, so it passes on exactly what it exists to catch. An instrument also fails by going red for a reason that is not a regression: a floor measured on one population and asserted over a pool fires with no code change once the denominator gains a new corpus, language or tenant — and dilution is the one cause a pooled metric cannot distinguish from the cause its author imagined. And the probe that proves a test can fail has a second half nobody writes down: hand-mutating a control to a no-op puts a real defect into production source, so the revert is part of the technique — verify it against the working tree, never against an exit status a wrapper will report as a clean pass. A known-bad also rots without anyone touching it, and in the direction nobody watches: a mutation pinned to a delta — append one line, add one file — breaches a threshold only from the subject’s current slack, so improving the subject disarms the probe. Measured here: shrinking an always-loaded file from 199 lines to 169 silently made the probe guarding its 200-line cap inert, and every standard safeguard passed — the literal matched, the mutation took, the tree was dirty. Pin the mutation to the threshold, and re-run the known-bads after a refactor, not only after a check changes. The remedy every mature discipline reached independently — the proof test, the clinical positive control, aviation built-in test, adversary emulation — is one move: a negative control, a committed known-bad the gate must reject on every run, verified per target rather than once, because a tripwire that fires for 2 of 20 targets looks identical to full coverage. Worth knowing that no mainstream framework asks for this: NIST SSDF and the EU CRA require a record that a scan ran, OpenSSF Scorecard’s SAST check detects only that a tool is configured, and SLSA will sign provenance for a scanner set to scan zero files. A passing compliance check is evidence of process, not of protection. And a negative control answers only whether the gate can fail — never whether it still covers you: a refactor into a nested module, a second manifest or a sidecar image moves code out of a gate’s scope with no diff to the workflow file, so watch the number of units each gate enumerated and treat a drop as a failure. (rules/15, devsecops rules/05)
• Absence encoded as a value — the in-band sentinel: a number whose domain includes an “absent/unknown/error” marker (-1, 0, "", 9999-12-31). It type-checks and no linter flags it: -1 is truthy, so the if x: presence check everyone reaches for admits it, and because the sentinel carries an ordering it loses every < and wins every > — one missing operand makes a guard skip silently in one direction and fire spuriously in the other, from a single input. The tell is not the constant but the asymmetric guard: one operand filtered against the sentinel and the other, in the same comparison, not — because that filtering is applied per site, so it lands only where the author happened to be thinking about it. (architecture rules/02 §8a, with a per-language row measured on each toolchain)
• Dependencies declared but never reached — the inert-dependency sweep: packages, modules, and plugins wired in and inert, installed and pinned and scanned and never run. Proven by deleting them in a scratch copy and running the real build, lint, and full suite, with exit codes and before/after transitive counts reported: a grep is not proof, and no tool’s silence is either. (rules/10)
• Decisions that stopped being right — the datastore picked for scale that never arrived, the rewrite justified by a benchmark that no longer reproduces. Every expensive-to-reverse decision is classified JUSTIFIED / STALE / UNJUSTIFIED / UNVERIFIABLE, and any number one rests on is re-measured this session.
• Findings that don’t survive contact — every Critical/High gets an independent pass prompted to kill it, working from the code rather than the write-up and defaulting to REFUTED when the evidence is ambiguous. Refutations are recorded, so the next auditor doesn’t re-raise them — and swept before they’re dropped, because the refuted pattern routinely closes somewhere the audit’s scope didn’t cover. A severity also has to name its chain (reach → primitive → boundary crossing → channel), and on a diff it is rated against the code the change replaced: filing a High against a hardening fix teaches authors that hardening attracts findings. The refuter also gets less than the finder had — no execution, no writes — with only the artifact crossing over (the PoC, the failing command, the file:line), never your write-up; and it returns a number against a threshold fixed before the findings were seen, because a paragraph of hedging is a judgement you then re-judge. (findings §1, §4, §4a)
• A result you saw once — where the evidence is a behaviour (a crash, a race, a timing bypass, anything an agent or a sampled model produced), it is reproduced N of N and both numbers are reported: 1/1 and 3/3 are typeset identically and mean different things. On a repeat audit the yield curve is read too — count falling while difficulty rises. A flat count wave after wave is a statement about the audit, not the code. (findings §2, methodology §4)
• A control that works, on some of the sites it’s credited with — not inert, not missing: a real containment check guarding one channel of five, a disable_tools=True passed at 2 call sites of 6, a path guard on 9 target walks of 61. Every signal is a working control’s signal, because on the guarded subset it is one. The audit move is a census — enumerate the protected operation, mark each call site, report the ratio — and the API fix is that a parameter selecting a trust boundary defaults to the closed side. The population has a second axis, the principals it runs as: pam_faillock returns success when it cannot open its tally, so a 0700 tally directory recorded no lock-screen failures (PAM runs as the user there) while every root-run test passed. (code-security rules/14 §6)
• Security prose the code no longer keeps — “containment is applied at every target walk” (it was 9 of 61); “no function-calling, no shell — this RCE mechanism is NOT APPLICABLE” (tools were on by default). Each had been true when written. Most such claims are universally quantified and therefore falsifiable by counting, and a wrong NOT APPLICABLE is the worst case: it cancels the next reader’s own investigation. (code-security rules/14 §7)
• Your own tool, ingesting somebody else’s repository — scanners, SAST wrappers, review bots and agentic analysers run on a maintainer’s machine with that identity’s credentials, and the target is the attacker. Four legs, usually owned by four people: staging that dereferences symlinks out of the tree, “static” analysis that evaluates target-controlled build metadata, an LLM step spawned with tools live and the parent’s cwd, and egress left on by default. “Buildless” is no defence: measured on CodeQL 2.27.3, Rust --build-mode=none ran a target’s rustc-wrapper, build.rs and proc macro, and cargo metadata, tree and fetch each ran a repository’s rustc-wrapper. (sandboxing rules/05 §7, rust rules/07 §4a)
• Absence claims — “no hardcoded secrets remain” is the one finding nobody can falsify: a narrow search and a true absence produce identical output. Any absence claim needs a widened search plus a second independent method, with the search stated. (findings §2, §4)
• Controls keyed to a neighbouring setting — the predicate reads a proxy that agrees with the real dependency right up until someone configures the two apart, three files away. A coupling defect: the control’s own site never changes, so neither per-file review nor a per-gate probe can see it — the signature is a control that silently stops at the moment it starts mattering. Field-reported 2026-08-26 with two shell shapes that fail the same way: a process substitution whose producer failed (its exit status is unreachable, so zero lines reads as success) and an “append” to a keyed store that is really an upsert, silently deleting the link target of a hash chain on the second run. The audit for this class is the proxy question, which the usual falsification question cannot answer: is this the thing I actually depend on, or something that currently agrees with it — and who can change one without the other, and would I find out? A control that is correctly enforcing the wrong predicate is not inert, so “would anything observable differ?” answers yes while the control is still wrong.

Where this is not backed by a number: the measured lift is in BUILD (completeness, freshness). Nine audit instruments across four designs sit at +0.00, and a tenth read a registered null too — recognition (snippets, cross-file repo, precision), procedure (does the model actually mutate the control and re-run the build), question-set (an unscoped “audit this repository”, with defect classes outside the standard repertoire), and, since 2026-08-14, a real repository at a real vulnerable commit — the one design a synthetic fixture provably could not stand in for, because a planted defect is a deviation from its filler and agents find deviations without security reasoning. That last one is the strongest form of the test and it closed the question: across 16 real BOLA sites in Harbor v2.5.1 both arms recalled 15/16, and across 59 blinded findings both scored precision 1.00. A frontier model handed the code is already at ceiling — on synthetic code, on real code, and when you stop telling it what to look for. The audit half is justified by gap analysis and by real defects it found in this repo — not by a measured lift, and it is reported that way rather than implied. This paragraph once said there would be no tenth accuracy instrument. One was built on 2026-09-12 (COMMENT-TRIAGE) and read a registered null. A different dependent variable (time-to-find, report usability, reach for a non-expert) is still untested. Two numbers landed 2026-09-10, both pre-registered before any call. The router’s terminal self-audit (BUILD step 4) is no longer only a mechanism: under 400 lines of competing rules prose, removing it costs −0.05, and it recovers +0.062 — SE 0.019, 95% CI [+0.024, +0.099], the first interval here excluding zero on that step (GATE-ABSORPTION). And the conflict rate between two skills loaded together — the one failure mode with a real incident behind it — measured 0.176 verified against 0.353 judge-reported, which refuted our own prior that conflicts are rare. The gap between those two numbers is the story: 3 of 11 reported conflicts quoted sentences that were not in the files named, and the judge could not see the router, which is where conflict resolution lives (CONFLICT-RATE).

Every null, the retraction, and the pre-registered predictions that were wrong →

How the numbers are kept honest

The measurement discipline is the part that is hard to copy, so it is worth stating plainly. Every item below is in the repo, not a claim about it:

• Nulls are published, not buried. Eleven +0.00 rows (counted 2026-09-24) sit on the scoreboard next to the +0.39 — including the ones that say, in our own words, that the audit half of this library adds nothing a good model doesn’t already do. That null is why the defect-avoidance result above matters: we went looking for value where our own measurements said there wasn’t any, and reported both. (This line read “seven” until 2026-08-21 — it was understating the count.)
• A lift was retracted. An early +0.07 on inert-control detection did not reproduce when the sample grew from 15 to 49 cases. It was withdrawn and the retraction is documented rather than quietly dropped.
• Routing is re-measured on a clock, not only on a diff. scripts/routing-baseline.sh re-runs the description classifier locally, and a monthly job goes red when that stamp is older than 3 months, because a model release can move routing with no change here.
• The gates are proven able to fail. scripts/check-negative-controls.sh runs in CI as its own job: it injects a known-bad per invariant into a disposable git worktree and requires the intended check to be the one that complains — a non-zero exit for any other reason is reported as a false pass, not a catch. A passing gate proves the tree is clean; only this proves the gate still works.
• Predictions are pre-registered. Before the 2026-07-30 audit experiments ran, the expected numbers and ranges were committed and pushed (PRE-REGISTRATION.md). Both predictions turned out wrong — one by three times its own lower bound — and are reported as wrong.
• The scorers are themselves tested. A mutation probe once replaced a scoring function with return 1.0 and nothing noticed; the golden tests that now run in CI were watched to fail against that exact mutation first.
• The eval runners carry a duration baseline. Each run records to a local ledger and the next run of the same runner prints the delta — [run-completeness elapsed 12.3s over 7 cases | previous 380.0s — 30.9x faster] — because “finished far faster than the work allows” is a comparison, and a duration without its denominator says nothing. A swing over 5× is flagged for a human; nothing is gated on time. That ledger was itself a shared sink until 2026-09-02 — --selftest runs and aborts landed in it in the same shape as measurements (46 of 60 rows), which had quietly disarmed the comparison for the most-run runner. Runs now mark completion, and only a completed run can be a baseline.
• The library is applied to itself, and it finds things. Its own dead-path rules caught this repo’s CI gates passing over zero files — green, exit 0, examining nothing — and the fix was verified by re-running the mutation. The same lens later caught this repo’s secret scan: gitleaks prints 179 commits scanned, and nobody read it. In a shallow clone it scans 1 of 179, prints no leaks found, and exits 0 — a green scan over 0.5% of history, with one CI setting the only thing preventing it. CI now asserts the scope. Two more, both 2026-09-04. Two releases had shipped and were never tagged — a CHANGELOG entry, a version bump, a merge, and no tag: unreachable by git tag, and invisible because the one version gate checked only that a tag is never ahead of VERSION. And an eval runner had been a dead runner for six days — its ablation target moved to another file in a split, so its guard refused every run. The guard was right; what failed is that the smoke harness rendered it as a bland ok, the same string it prints for a runner that merely wants arguments.

The point is not that every number is flattering. It is that you can tell which ones are load-bearing, because the ones that aren’t are labelled.

Skills

Every garbage-collected language skill also covers its escape hatch into raw memory: JNI/FFM/Unsafe, ctypes/cffi, native addons, unsafe/P/Invoke, PHP FFI, Fiddle. The class is stated once in sota-code-security rules/06 §3, and each language skill carries its own detectors.

Several hazards recur in every language, so each is stated once as a shared class in sota-code-security, with a detector for each language in that shared rule:

• SSH host-key verification;
• temp-file and permission hygiene;
• an LDAP bind used as a login check;
• XML built from strings;
• native library search paths.

Each language skill was also gap-checked against an external, tool-backed list of that language’s security checks: Bandit, gosec, cppcheck, Brakeman, ANSSI, find-sec-bugs, eslint-plugin-security, Psalm and the .NET analyzers. A concept matrix (scripts/gen-concept-matrix.py) keeps universal topics such as numeric precision and money covered in all nine.

Coverage & non-goals

Deliberately not covered: Scala/Elixir, standalone C (inside sota-c-cpp), platform-engineering/IDP depth. File a skill request issue. A language with no skill still routes: the router applies the domain skills and says no language skill covers it.

Installation

Both commands are shown at the top — the plugin (/plugin, auto-updates on version bump) or clone + link (./scripts/install.sh, a local checkout to read, hack on, or pin). A few details on the clone path:

• Skills are discovered from .claude/skills/ (per project) or ~/.claude/skills/ (personal, all projects); install.sh symlinks every skill and your profile.
• --project DIR scopes the skills to one repo (the slash commands still go to ~/.claude/commands, on purpose: they are run about work in another project); --copy pins a snapshot instead of linking.
• Other agents need --target all. Copilot CLI, Codex, Gemini CLI and Antigravity do not document ~/.claude/skills; --target all also links ~/.agents/skills (Copilot, Codex, Cursor, Gemini CLI) and Antigravity’s own ~/.gemini/config/skills / ~/.gemini/antigravity-cli/skills where it is installed, and --routing then mirrors the directive — ending in the router’s absolute path, for an agent with no skills loader — into each installed agent’s global file. With no flag, the installer offers this when it detects one of them. Paths, sources and what is still unverified: docs/MULTI-AGENT.md.
• On Windows without Developer Mode, the install is a copy. Git Bash’s ln -s copies silently when a symlink is refused; the installer now detects that and says so — re-run it after every git pull, or enable Developer Mode and re-run for live links. Run as an elevated administrator, it links: 43 of 43 skills were real symlinks on Windows 11 with Git for Windows 2.56 (measured 2026-10-22). The no-privilege copy path above follows the Git for Windows and MSYS2 docs, and was not measured.
• Prefer no script? Symlink skills/*/ into ~/.claude/skills/ and commands/*.md into ~/.claude/commands/ by hand. You then skip what the script adds beyond links: the update-reminder hook, the routing-hook, code-rules-hook, zsh EQUALS hook and listing-budget offers, and the reach check.

The plugin (or --copy) installs the skills; a few extras (routing reminder, status line, pre-commit gates, AGENTS.md) aren’t auto-enabled — see Optional extras for plugin users. On first run the plugin shows a one-time notice pointing there.

An install is personal, not repo-resident

Installing once makes the skills apply in every project you open, new ones included — there is nothing to run per repo, and a brand-new repo is covered the moment you start work in it. But the install resolves against your home directory: a teammate’s clone and a CI runner see none of it.

So for a repo shared with anyone, decide explicitly:

./scripts/install.sh --project .   # repo-resident: .claude/skills/ in the repo
./scripts/install.sh --project . --copy   # ...pinned snapshot, not symlinks to your paths

— or leave it personal and put the install step in CONTRIBUTING.md so a contributor can reproduce it. What must not stay personal is the gates: a secret scan that lives only in your shell doesn’t run on anyone else’s commit. Wire those into the repo with init-gates.sh.

And the things no install can supply per repo — gates, LICENSE, .gitignore, an agent file, and the order to create them in — are the day-zero list in sota-docs-workflow rules/01 §10. The router raises it once, unprompted, the first time it meets a repo that has none of them.

Updating

Plugin install: updates ship when the version bumps — /plugin update sota-skills@sota-skills (or /plugin marketplace update sota-skills).

Do not assume auto-update covers you. Per the Claude Code docs (checked 2026-07-30): “Third-party and local development marketplaces have auto-update disabled by default” — this is a third-party marketplace, so unless you turned it on (/plugin → Marketplaces → the entry → Enable auto-update), nothing updates on its own. Even with it enabled, the check runs after your session starts “with a random delay of up to ten minutes”, and the running session keeps the versions it loaded at launch, so a refresh lands on /reload-plugins or your next launch — never mid-turn. Run /plugin update when you want a known-current library.

An occasional nudge, with no telemetry. Nothing pushes updates on either path, so a SessionStart hook (scripts/update-reminder.sh) mentions — at most once every 14 days — that your copy has been sitting for a while, and how to check. It makes no network request. It cannot tell whether a new version exists, only how long since it last spoke: the useful part was the reminder, and a real check from every session start would turn a documentation library into something that reports when and how often you work. You run the check. Installed by install.sh (clone) and by the plugin’s own hook; silence it with SOTA_UPDATE_REMINDER_DAYS=0, or set your own interval in days. The one thing it can know locally it says every session: a copied install (no symlinks, e.g. Git Bash without Developer Mode) is stamped with its release, and once git pull moves the checkout past it, the hook names the stale copy and the re-install command. A second local fact, said once: a default-on hook script that a git pull brought to a clone install and nothing registered (the code-rules hook, or the zsh EQUALS hook when the agent’s shell is zsh), since only install.sh --routing writes settings.json. A fresh install stays silent: it was just offered every hook.

Which version am I on? scripts/install.sh --version reports the release, the checkout (git describe), whether your remote is ahead as of the last fetch, and whether the skills are symlinked (update live) or a pinned --copy snapshot. Quote it in a bug report — otherwise a report about a rule’s behaviour can’t be tied to the release that produced it.

Clone install: because linking is symlink-based, existing skills update the moment you pull — the symlinks already point at the live files:

git -C /path/to/SOTA-skills pull

To also pick up newly added skills (a pull alone won’t link a brand-new skill directory) and prune links to removed ones — pull and re-link at once:

./scripts/update.sh                  # git pull --ff-only, then re-link
./scripts/install.sh --update        # the same thing — update.sh is a thin alias

It’s idempotent: re-running only links what’s new and prunes what’s gone, and never touches symlinks it didn’t create (--copy snapshots don’t auto-update — re-run to refresh). With always-on routing enabled, a re-run also refreshes the managed routing directive and reminder hook in place when their wording changes upstream — prompting first, backing up, touching only the managed block; a hook you customized is left untouched.

What an update does not refresh: the gates in your own repos. update.sh re-links skills and re-syncs this checkout’s own hooks; it never reaches a repo you ran init-gates.sh in, so those gates stay at the release that generated them. Re-run init-gates.sh there — it rewrites only its own managed block, so it is safe to repeat. This matters most for hook types: pre-commit writes .git/hooks/ at install time, so a config that gains a pre-push stage installs nothing on its own and the new gate silently never runs (verified on pre-commit 4.6.0). verify-setup.sh check 9a reports exactly that — a declared stage with no hook file — which check 9 cannot see, because a different hook being present makes it read “installed”.

The skill listing budget — raise it, or most of these descriptions never reach the model

Claude Code reserves a per-turn character budget for the skill listing and computes it as (context tokens x 4) x skillListingBudgetFraction, where the fraction defaults to 0.01. On a 200k-context model that is 8,000 characters for every skill you have installed, from every source. This library’s 42 descriptions are about 38,240 of it — and the budget is shared, so what matters is your machine’s total. On the machine this was found on, the engine counted 65 skills / 47,324 characters against an 8,000 budget: our 42 plus 23 built-in skills that are compiled into the binary and invisible to any script.

Over budget, entries are ranked by recent usage and the ones that do not fit are rendered as a bare - name with no description at all — so they carry no trigger vocabulary and cannot be auto-selected on what they do. It is not a tail-trim: it is all-or-nothing per skill. A skill you have never used ranks zero and is dropped first, which is a discovery ratchet — invisible because unused, unused because invisible.

Measured 2026-09-11 by reading the shipped claude binary (2.1.268), not inferred from behaviour, and confirmed live: roughly 20 of these 42 skills were listed name-only in the session that found it.

Claude Code does notice — and tells nobody. Forcing the condition and reading the log it wrote gives [WARN] Skill listing over budget: 65 skills, 47324 chars > 8000 budget — descriptions will be truncated. That line goes to ~/.claude/debug/ and appears nowhere else, which is exactly why a whole session can run on a crippled listing unnoticed. Since then (code.claude.com/docs/en/skills, re-read 2026-10-20): the docs name two places to look in a session. /doctor estimates the listing’s context cost and its biggest contributors. The Skills row in /context reports the listing’s size after the budget is applied, so it matches what the model receives (before v2.1.196 it counted every description in full). The docs also say the over-budget warning is written to the debug log; they do not say whether /doctor shows it. Not re-measured here.

Why it still works anyway, and what actually breaks. The sota router is used every session, so it ranks near the top and keeps its description — and a name-only skill is still invocable by name, which is exactly what the router’s routing table does. That is why always-on routing below is the recommended path and not a nicety: it is the mitigation. What is genuinely lost is direct auto-selection of a domain skill when the router has not fired, which is most likely for the skills you have never used.

scripts/install.sh offers to fix this, on install and on --update, in the same shape as everything else it touches.

• It measures what the listing actually holds, not what the installer just linked: personal skills, your account’s synced claude.ai skills, every enabled plugin, and the project’s own. scripts/skill-listing-sources.sh does this for all three checks (installer, verify-setup.sh, the plugin hook). It adds 25% for the built-in skills, which are compiled in and not on disk.

• It sizes the budget for the context window you actually run. The budget scales with the window, and 1M context gives five times the budget of 200K. Only a live session knows the window exactly, so the checks use, in order:

  • the window scripts/statusline.sh last observed, which it records;
  • CLAUDE_CODE_DISABLE_1M_CONTEXT=1, which forces 200K;
  • a [1m] model setting.

  Otherwise the window is unknown and 200K is assumed. The model name alone can’t rule 1M out, because several current models run 1M natively with no suffix.

• When it fits, it does nothing and says so. When it doesn’t, it asks before writing, backs the file up, writes through a symlink so dotfiles stay in charge, and never lowers a value you already set. For this library’s skills alone that is about 0.07 at 200K, or 0.02 at a known 1M window (measured at 42 skills).

• It warns if the library is installed twice (a clone and the plugin). Every skill is then listed twice, and the fix is to remove one copy, not to raise the budget.

To set it by hand instead, in ~/.claude/settings.json:

{ "skillListingBudgetFraction": 0.07 }

Be aware of what you are buying: the listing is sent every turn, so this reserves ~7% of the context window for it. skillListingMaxDescChars (default 1536) is a separate per-skill cap — no description here exceeds it, so it does not apply.

Plugin users get told, because a plugin cannot fix this itself. Verified against the 2.1.268 binary: the settings precedence is ["userSettings","projectSettings","localSettings","flagSettings","policySettings"] — there is no plugin scope — and a plugin contributes skills, hooks, agents, commands and MCP servers only. So the plugin ships a SessionStart hook (scripts/plugin-budget-check.sh). It measures the same sources as the installer and compares them against the budget at your session’s window; a [1m] model in the hook’s input counts as proof of 1M. It says so with the numbers, the window it assumed and the exact setting when they do not fit, and separately when the library is installed twice. It does not write your settings: that reserves a share of every context window, so it is your call. It speaks once per (size, fraction) pair — silent once fixed, and audible again if you install more skills.

To check your own install: scripts/verify-setup.sh reports it as check 1b, with the arithmetic, the per-source breakdown and the window it used, and a double install as 1g. By hand: look at the skill list Claude is given and count entries that are a bare name with no text after the colon. Any such skill is installed, correct, and unreachable except by name.

Always-on routing (recommended)

Skill descriptions are matched per prompt, so routing is opt-in and depends on how you phrase the request. To make the skills apply to every session regardless of wording, pin the routing instruction where Claude Code always sees it.

The quick path: ./scripts/install.sh offers to set this up for you after linking the skills — interactive and dotfiles-aware: it detects an existing or symlinked ~/.claude/CLAUDE.md / settings.json, asks before touching anything (recommended answer pre-filled), backs up first, writes through a symlink so dotfiles stay in charge, and uses managed markers so re-runs refresh the managed block in place and never duplicate it. Use --routing to force, --no-routing to skip, --yes for non-interactive. Or wire the three layers by hand:

Three layers, strongest last:

1. A stack profile. Copy the template, fill in your stack, and symlink it into ~/.claude/ so the router finds it in every project (not just this repo):

cp profiles/example.md.template profiles/.md   # edit it — profiles/*.md is git-ignored
mkdir -p ~/.claude/profiles
ln -sfn "$(pwd)/profiles/.md" ~/.claude/profiles/.md

2. A global directive. ~/.claude/CLAUDE.md is loaded into every session, every project. Add a routing mandate so the skills apply without trigger words:

# Global engineering directive

Always, on every answer: (1) **validate before you assert** — verify any claim
about code, system state, config, versions, or facts against a primary source
(read the file / run the command / fetch official docs) before answering or
proposing, and label anything unverified as such; (2) **keep docs current** —
when you change code/behavior/config, update the affected docs (README,
CHANGELOG, comments, runbooks, AGENTS.md) in the same change, unprompted.

For any task that builds, designs, refactors, debugs, reviews, or audits code —
in any language or repo — consult the `sota` router skill first, load the
matching `sota-*` skills, and apply their rules before acting. This holds even
when I never say "SOTA" or "audit". Treat `~/.claude/profiles/.md` as the
BUILD default and AUDIT baseline, and stop-and-ask on security-relevant choices.

3. (Optional) A per-prompt reminder. A directive read many turns ago can fade from a long context; a UserPromptSubmit hook in ~/.claude/settings.json re-injects it on every prompt. install.sh --routing writes exactly this, and --update offers to refresh the wording when a release changes it:

{
  "hooks": {
    "UserPromptSubmit": [
      { "hooks": [ { "type": "command",
        "command": "echo 'sota standing rules (every answer): (1) VALIDATE — check any claim about code, system state, config, versions or facts against a primary source before asserting it, and label anything unverified. (2) KEEP DOCS CURRENT — update affected docs in the same change. (3) ROUTE BEFORE YOU ACT — if the turn touches code, a diff, a config or a build/CI file, invoke the sota skill FIRST and apply the matching sota-* skills. Reading a file counts. If you have already read code this session without routing, route now. Routing is per task SHAPE, not per session: if the work has changed shape since you last routed — different layer or language, or from building to verifying — route again. Treat ~/.claude/profiles as the stack baseline; stop and ask on security-relevant choices.'" } ] }
    ]
  }
}

Each rule is a numbered imperative of equal weight. That is not cosmetic: in a real session where routing was the third clause of a run-on sentence, rules (1) and (2) were obeyed every turn and the routing clause was dropped every turn — same text, same repetition, opposite outcome. Rule (3)'s last sentence makes the trigger recoverable: a long session often becomes a code task without any single prompt saying so, and routing late beats not routing.

If you hand-edit the wording, keep the phrase sota-* skills in it — that is the marker install.sh uses to recognise the hook as its own. Reword past it and your hook stops receiving updates, and a later --update adds a second one beside it instead of refreshing it.

No mechanism in the skills forces a model to run a skill — the three layers feed it instructions it chooses to follow, making routing reliable, not phrasing-dependent.

Advisory vs enforced — read this before calling the library “only advisory”

That sentence is about the skills, and it is often read as “nothing here can enforce anything”. Three corrections, each re-checked 2026-10-02:

• The host can enforce, and some of it ships here. Claude Code hooks are not advice: a Stop hook can block the turn from ending, PreToolUse can deny or rewrite any tool call (hooks docs), and the Agent SDK exposes the same loop with hooks and permissions programmatically. scripts/verified-done-hook.py uses that: an opt-in Stop hook that refuses “done” on a changed tree until a verification command has passed on exactly that tree (router principle 6, enforced). scripts/zsh-equals-hook.py, on by default in the plugin, is a PreToolUse hook that refuses a zsh command carrying an unquoted =word before it runs (sota-shell-scripting rules/06 §1), a reflex the written rule did not stop. The git hooks from scripts/init-gates.sh and CI are enforcement on every host, whatever the agent.
• The router is not an always-on 43 KB blob. Only each skill’s frontmatter description is in every turn’s listing; the router’s body loads when the skill is invoked (CONTEXT-MANAGEMENT).
• The evals measure guidance, not an agent loop. Every runner script in evals/ is a chat-completion call with the rules in the prompt; no tools run and nothing the model writes is executed. So the lifts here are real for that condition. The agent-loop condition is measured in a separate repo, sota-agent-evals: Claude Agent SDK sessions with tools, scored by hidden tests in a no-network container with no model judge. Its v2 run (2026-10-02) was clean and read not supported on near-saturated cases. v3 (2026-10-08) hit the ceiling in every arm, and its library arms never invoked a skill: 0 of 166 library runs since v2. That arm installed skills without the routing layer install.sh adds. v4 (2026-10-09) added it: 20/20 runs then invoked the library, but the router is mostly where they stopped (3/20 went further, 0/20 read a rules file). Pass rates were at the ceiling in both arms, so whether the rules help is still unmeasured (v4 report)) (evals/README).

What stays advisory is everything a hook cannot check mechanically: whether a rule was applied well. The line is “can a program decide it from the tree or the tool log”, not “skills vs harness”.

With the hook in place, re-typing the same instruction adds nothing mechanically. Rule (1) is already prepended to every prompt you send, so “always validate your claims” is a second copy of text the model has just received. What is not redundant is naming the technique instead of the goal — “re-read the state, don’t trust your summary”, “verify at the remote”, “run it rather than reason about it”. The failure mode this library keeps finding is never forgot to care; it is checked the wrong artefact, and only the specific phrasing redirects that. Operating principle 7 is the generic form of it. (Reported from a session using the library, 2026-08-18: an observation about mechanism, not a measured effect — nobody can run the counterfactual.)

Using it

With always-on routing set up (above), you don’t name anything — describe the task in plain language and the right skills load automatically (see How it works). Name a skill or rule only to force a specific skill, scope to one rule file, or stack an exact combo.

Building — plain prompts; routing picks the skills:

Design a multi-tenant invoicing service — stack from my profile, or propose one.

Add a RAG search feature over our docs, and write the evals first.

Scaffold the GitHub Actions pipeline for this repo: SHA-pinned actions, OIDC, SBOM + signing.

We handle CUI on this service — what does that require of the architecture and data stores?

Auditing — say the mode (“audit”, “review”, “harden”):

Run a full audit of this repo. Static analysis only, current commit, report with a prioritized roadmap.

Audit this PR before I merge it.

Sweep the repo and git history for secrets — rotate-first recommendations.

Threat-model this service from the code: DFD, trust boundaries, STRIDE.

Audit our Kubernetes manifests and Dockerfiles. Severity + effort on every finding.

Why is checkout slow? Profile first — no guessing.

Review our agent’s MCP setup for tool poisoning, rug pulls, and shadowing.

Naming a skill or rule (optional — to force or scope):

Add a websocket endpoint per sota-api-design rules/05 (auth-at-upgrade, backpressure).

Is this migration zero-downtime safe? Check sota-databases rules/02 (expand/contract, lock-aware DDL).

Review test-suite health against sota-testing rules/07 (flaky policy, coverage ratchets, speed budgets).

Audit this PR against sota-code-security + sota-golang before merge.

Maintaining the library:

Refresh the library — re-verify fast-moving claims against current primary sources, apply fixes, and update the root LAST-VERIFIED stamp.

Create profiles/.md for my stack: .

Add a new skill for , same structure: SKILL.md + rules/ under 500 lines each, claims web-verified.

Tips:

• Say the mode if ambiguous — “audit/review/harden” vs “build/add/design”; skills key off those verbs.
• Scope audits explicitly — which commit/branch, static-only or may-run-tools, time budget (“crown jewels only”). The methodology file asks otherwise.
• Ask for the report format — default audit output is executive summary → findings by severity → roadmap by risk-reduction-per-effort → positive notes.
• Re-verify version-sensitive facts — web-check before pinning any version.

Slash commands

scripts/install.sh (and therefore scripts/update.sh) links every file in commands/ into ~/.claude/commands/, so they work in any project, not just this one. verify-setup.sh check 1d compares what is installed against what the checkout ships — a git pull refreshes existing links and creates none, so a newly added command stays silently uninstalled until you re-run the installer.

/sota-resume and /sota-close are the two ends of the same session: one picks the open work up, the other puts it down where the next session will find it. Neither needs the other to have run — most projects scatter their open work across files written by people who are not here. Both load what they cite: a step 0 invokes sota-shell-scripting and reads its rules/06 before the first sweep, because a citation loads nothing — and invariant 40 fails any command that cites a skill’s rules file without saying to load it.

All five take free-text arguments to steer them (/sota-close focus on the migration branch). They are prompts, not scripts: the text is in commands/*.md and nowhere else, so you can read exactly what your session is being told to do before you run it.

Why the threat model runs first — light in /sota-audit, full in /sota-deep-audit. A finding’s severity rests on the chain it closes, and a chain crosses a trust boundary, so both commands map the boundaries before rating anything. /sota-audit builds the light version: entry points, the authn/authz checkpoint on each, the asset it reaches, and the assumptions that implies, each checked against the code. /sota-deep-audit reconstructs the whole model. That is a pass of its own, and it goes before the lenses rather than inside the security one, because a finding rated without trust boundaries is rated in a vacuum. It extracts entry points, stores, actors and the boundaries the system actually has from the code rather than the docs, then writes down the assumptions that implies — services trust the gateway’s headers, that bucket is private — and tests each against the code that makes it true. No evidence is not “probably fine”; it is a broken assumption, and those are usually the Criticals. What comes out is a control-presence matrix: Present · Partial · Absent · N/A · Unverifiable, each carrying the evidence its state demands — file:line for Present, both sides for Partial, and the searches you ran for Absent, so a reviewer can re-run them. Partial is the most important state. One authorization-checked endpoint proves the team knows the pattern; the seventeen unchecked ones are the finding, and they tell you the remediation is adoption rather than invention. The pass also reads the negative space — exclusion lists, skip-auth decorators, count = 0, disabled tests with security in the name — because a control someone switched off is a stronger finding than one never built. Full procedure: sota-threat-modeling rules/06.

Why /sota-close is a command rather than a habit. The end of a session is when the context has usually been compacted at least once, so “what happened” is reconstructed from your assistant’s own earlier prose about the work rather than from the work — the weakest check available. Everything the command asks for is executed against an artifact instead: git diff, git log -S, the file on disk, the command run again. The ordering is load-bearing too — retraction comes first because every later step writes, and anything still wrong gets propagated into a doc, a commit message or a hand-off where the next session reads it as established.

Optional setup & integrations

Beyond the skills themselves — all opt-in, none required to use the library.

Badge

Built or audited a project with the library? Ship the attribution :

[![Built with SOTA Skills](https://img.shields.io/badge/Built%20with-SOTA%20Skills-2fa45f)](https://github.com/martinholovsky/SOTA-skills)

Enforcing the gates

Routing makes the model apply the rules; to make them stick regardless of who (or what) commits, wire them as git hooks. scripts/init-gates.sh generates a SOTA-aligned .pre-commit-config.yaml for whatever languages it finds in the target repo:

cd /path/to/your/project
/path/to/SOTA-skills/scripts/init-gates.sh        # add --dry-run to preview first

It detects Python / Go / Rust / JS-TS / shell by manifest and extension, then writes the exact tools each skill prescribes — ruff·mypy·pytest·pip-audit, gofumpt·golangci-lint·govulncheck, clippy·cargo-audit, eslint·tsc· audit, shellcheck·shfmt, plus gitleaks everywhere. Fast checks (lint, format, secrets) run on commit; heavy ones (type-check, tests, vuln scans) run on push — the split sota-python rules/01 §6 and sota-devsecops rules/05 require, so commits stay quick.

It is idempotent: re-run it after adding a language and it rewrites only the block between its # >>> sota-gates >>> markers, leaving any hooks you added yourself in place. The hooks call your project’s own toolchain, so install the per-language tools it lists on exit (and pre-commit install if the script couldn’t).

Then check it actually took. init-gates.sh sets things up; nothing verifies the result, and “configured” and “working” render identically — a config file with no installed hook is not a control, and a CI job whose every run is skipped is a gate on paper.

/path/to/SOTA-skills/scripts/verify-setup.sh     # read-only; --runs N widens the CI sample
/path/to/SOTA-skills/scripts/verify-setup.sh --reach-only   # section A only: is the library reaching this machine?
/path/to/SOTA-skills/scripts/verify-setup.sh --no-color     # plain rows; also honours NO_COLOR and TERM=dumb

install.sh (and therefore update.sh) runs the --reach-only half itself at the end of every run, because “linked 42 skills” is what the installer did and reachability is what the agent gets — and the two diverge silently, since a git pull updates every existing symlink and can create none. Pass --no-verify to skip it.

It reports skills reachability, the routing hook, the profile symlink, a licence under any name, which gates exist, whether a hook is installed rather than merely configured, and — from real run conclusions — whether CI has ever executed and ever rejected anything. Its section F adds one fact about the machine rather than the repo: what your default searcher silently skips. Every searcher excludes something quietly — a symlinked directory, a .gitignore’d path, a dot-directory — and the failure mode is a clean absence, which is exactly the answer a sweep was hoping to prove. It runs a four-match fixture under $TMPDIR and names the exclusions as INFO; nothing fails for them, because rg skipping ignored files is a feature. The one thing that can fail is the positive control — a searcher that cannot find the plain file is broken, and its absences are not evidence. It changes nothing and exits 1 on any FAIL. Anything it could not observe is marked UNVERIFIED, never passed: on this repo the reject-history check reads UNVERIFIED at the default sample and turns up a real rejection at --runs 200, which is why the sample size is printed. docs/VERIFY-SETUP.md carries the other half — a paste-in prompt for the judgement calls a script can’t make: whether the agent file’s content is meaningful and whether its claims are still true.

Add --docs-gate to also install a pre-commit hook that blocks a commit which changes code but updates no docs (README/CHANGELOG/docs//*.md) — so docs stay current without you having to ask. It writes a small helper to .sota/docs-gate.sh; it’s heuristic (a docstring-only edit inside a code file will trip it) and bypassable with SKIP=sota-docs-gate git commit, which is why it’s opt-in.

Other AI agents (Codex, Copilot, Gemini, …)

The skill content is plain Markdown — any model reads it. To make the skills themselves discoverable by Copilot CLI, Copilot in VS Code, Codex, Cursor, Gemini CLI or Antigravity, install with ./scripts/install.sh --target all — it links ~/.agents/skills, which all but Antigravity read, plus Antigravity’s own skills directories where it is installed, and --routing writes the directive into each one’s global instructions file (table and sources: docs/MULTI-AGENT.md). To route a project’s agent through the library, generate an AGENTS.md (the cross-tool open standard read by Codex, Cursor, Copilot, Antigravity CLI, Windsurf, Zed, and more):

Or over MCP. scripts/sota-mcp-server.py serves the same files read-only as MCP tools (list_skills, get_skill, get_rules_file), resources and prompts, for an agent that would rather call tools than read a skills folder. ./scripts/install.sh --mcp registers it for Copilot, Codex, Gemini CLI, Antigravity and Cursor where they are installed; verify-setup.sh shows where it is missing. It does not route: the agent’s model picks from the descriptions, as it does with skills (docs/MULTI-AGENT.md).

cd /path/to/your/project
/path/to/SOTA-skills/scripts/gen-agents-md.sh        # add --dry-run to preview

It writes a thin AGENTS.md that carries the operating principles and points the agent at the installed skills/ tree — the index is built from each skill’s frontmatter so it stays in sync, and the agent reads the relevant rules/*.md on demand (no rule text is duplicated). Idempotent via a managed block, like the others; --skills-dir/--output override the defaults. Claude Code keeps using the native Skills install above. This repo itself follows the standard: AGENTS.md is canonical; CLAUDE.md/GEMINI.md are symlinks.

Different tools read different filenames. Claude Code reads AGENTS.md natively since v2.1.277, but by default only when no CLAUDE.md or CLAUDE.local.md exists, so one personal CLAUDE.local.md silently stops it; older versions (and, before v2.1.281, some sessions such as Bedrock) read CLAUDE.md only. Either way the symptom is the model getting worse, not a missing file. Every run therefore reports the state of the two sibling entry points, and --siblings writes them when they are absent:

/path/to/SOTA-skills/scripts/gen-agents-md.sh --siblings
# ./CLAUDE.md  ->  @AGENTS.md      (Claude Code's native import)

It is not a copy — AGENTS.md stays the single source of truth — and the import is preferred to ln -s AGENTS.md CLAUDE.md because on Windows creating a symlink needs Administrator or Developer Mode, so a symlink is not portable.

For Claude Code a pointer is still the portable choice. GEMINI.md is opt-in behind --legacy-gemini, because Antigravity CLI — which replaces Gemini CLI, retired for individuals on 2026-06-18 — reads AGENTS.md natively. (Measured 2026-09-10: Gemini CLI 0.59.0 now exits with IneligibleTierError … migrate to the Antigravity suite, and Antigravity’s own bundled docs list “Directory-Based Rules (GEMINI.md / AGENTS.md)”.)

An existing CLAUDE.md/GEMINI.md is never touched — but it is never silently accepted either. The script inspects it and says what to change, because a file that never mentions AGENTS.md is a file that will not work with this library:

For Gemini there is a route with no pointer file at all: set context.fileName in .gemini/settings.json to ["AGENTS.md", "GEMINI.md"] — it accepts a list. And whichever you choose, confirm it loaded rather than assuming: /context in Claude Code (under Memory files), /memory show in Gemini CLI.

Status line (optional)

scripts/statusline.sh is a Claude Code status line that shows which skills you’ve actually used this session — not just how many are installed:

Opus 4.8 │ ctx 63% │ my-service ⎇ main │ skills▸ code-security, testing (2)

Claude Code’s status-line input doesn’t expose loaded skills, but it passes the transcript path; the script reads back the Skill invocations recorded there, falling back to a count of installed skills before any are used. It also records the context window Claude Code reports (context_window.context_window_size) in ~/.claude/sota-skills-data/context-window, rewriting it only when the value changes. That is the one exact source for the skill-listing budget checks, since nothing outside a live session can know the window. Wire it up in settings.json (requires jq):

"statusLine": { "type": "command", "command": "/path/to/SOTA-skills/scripts/statusline.sh" }

Optional extras (for plugin users)

The plugin installs the skills and, since 2026-10-08, the always-on routing reminder as its own UserPromptSubmit hook, on by default. Without it the skills were not used: across 166 agent runs with every skill installed and no routing layer, the model invoked a SOTA skill zero times (sota-agent-evals v3). Turn it off with SOTA_ROUTING_HOOK=off (for example in settings.json "env"). If you also installed the installer’s hook, the plugin’s stays silent, so the reminder is never doubled.

Since ROADMAP 73 (m) it also ships the code-rules hook, on by default (scripts/code-rules-hook.py, a PostToolUse hook on Edit|Write|MultiEdit). After the agent writes code, it names the rules file for what that code does: an outbound fetch → SSRF, a CSV writer → formula injection, an archive → Zip Slip, a handler → rate limiting. It tells the agent those rules are part of the task, so it builds them instead of handing them back as a question. It moved outcomes when measured: rule tests 0.698 with no hook → 0.906 with it (sota-agent-evals v10). It costs about +68% per agent task (v10: $0.615 vs $0.366 a run), its trigger table was written by an author who had read the cases that measured it (in-sample), and it needs python3. On fresh tasks it did not move outcomes: on cases-v5, 20 tasks sampled from OWASP ASVS and written blind to the library, the hook plus the router’s table left cited rule tests at 0.667 in both arms while reach rose 8/20 → 13/20 (v14, NULL, +38% cost). Agents read the named rule and handed it back as a decision, citing the precise spec. With one sentence of competing pressure added to every prompt (“internal demo tomorrow … skip the extras”), the same two arms read 0.567 → 0.767 (v15, INCONCLUSIVE: 3 cases up, 0 down, p = 0.125, +49% cost). Turn it off with SOTA_CODE_RULES_HOOK=off. install.sh --routing offers the same hook to clone installs; with both, each file is named once. The router carries the same table as text (“What your code does: the rules file to read”), for hosts that cannot run a hook. The exception is rows marked hook-only (HOOK_ONLY in the script, since 2026-10-22): new triggers reach the hook first and enter the router once their reach is measured. The router is at its 500-line cap, and its rows are what the reach numbers were measured on. Three host-config rows (containers and seccomp, AppArmor/SELinux, PAM) also read YAML, .j2, .conf, .service and .toml files, so an Ansible role or a Compose file is covered. Every other row reads code only: over 161,783 local config files, all rows together fired on 40%, almost always falsely. Measured without the hook, it got agents to the right rules file in 14 of 20 runs, against 6, and rule tests rose 0.679 → 0.792 (sota-agent-evals v13; in-sample, like the hook’s own numbers).

Since 2026-10-21 it also ships the zsh EQUALS hook, on by default (scripts/zsh-equals-hook.py, a PreToolUse hook on Bash). zsh rewrites an unquoted word starting with = to a command’s path and aborts the rest of the line on one that names no command, so cat a; echo ===; grep x b prints a and a == not found that reads like the end of the output, and the grep never runs. The hook refuses that command before it runs and tells the agent how to quote it (sota-shell-scripting rules/06 §1). A written rule did not stop the habit: a field report records the rule read and broken 60 seconds later. Over one operator’s 43,875 agent Bash calls, it refused the 344 that zsh had cut and nothing else (in-sample: the lexer was corrected on that corpus). It adds nothing to the context unless it fires, does nothing unless the agent’s shell is zsh (CLAUDE_CODE_SHELL, else $SHELL), costs about 85 ms per Bash call and needs python3. Turn it off with SOTA_ZSH_EQUALS_HOOK=off; install.sh --routing offers it to clone installs.

The plugin does not touch your global config or status line. To match the clone experience, opt in to any of these; the scripts ship with the plugin, under its cache dir:

• Status line — point settings.json statusLine at the bundled scripts/statusline.sh (see Status line).
• Pre-commit gates / AGENTS.md — run the bundled scripts/init-gates.sh or scripts/gen-agents-md.sh against a project (see Enforcing the gates and Other AI agents).

The quickest path: just ask Claude to “set up the SOTA optional extras” — the first-run notice prompts for exactly this, and Claude will walk you through them.

Structure

skills/
  sota/                          # master router — start here
    SKILL.md                     # routing, operating principles, workflows
    rules/
      01-audit-methodology.md    # how to run an audit: scoping, tooling, triage
      02-build-workflow.md       # the four BUILD steps and where they are mirrored
      03-audit-findings.md       # severity, evidence, refutation, reporting
      04-library-map.md          # the library map — which rules/NN holds what.
                                 # Offloaded out of the router 2026-09-02:
                                 # 16,997 -> 13,415 tokens per load, measured
  sota-/
    SKILL.md                     # when to use, BUILD/AUDIT workflows,
                                 # severity conventions, rules index, top-10
    rules/
      NN-.md              # ≤ 500 lines each (the cap), ends with an Audit checklist
      ...
profiles/
  .md                      # personal stack defaults consulted by router

Every skill works in two modes:

• BUILD — apply the rules while designing/writing code.
• AUDIT — review existing code; findings are emitted as file:line | rule violated | severity (Critical/High/Medium/Low/Info) | effort (trivial/small/medium/large) | fix.

Three cross-cutting pieces live outside the domain skills:

• skills/sota/rules/01-audit-methodology.md — how to run an audit: scoping, a verified static-analysis tool matrix, triage discipline, and audit hygiene.
• skills/sota/rules/03-audit-findings.md — what happens to what it finds: the severity model (a Critical/High must name its exploit chain leg by leg), the evidence standard, the decision-ledger pass, adversarial refutation, and the report template (executive summary → findings → roadmap by risk-reduction-per-effort).
• profiles/ — per-user stack profiles: the default in BUILD mode, the expected baseline in AUDIT mode — keeping the library generic and shareable.

How it works

Claude Code matches your prompt against each skill’s frontmatter description and loads what’s relevant automatically — you don’t have to name a skill. Naming one (or the sota router) just makes the routing explicit. From there:

1. The skill’s SKILL.md loads first (workflows, severity conventions, an index of its rules/ files). Only the rules files matching your task are read — never the whole library.
2. BUILD mode applies the rules while writing code and self-checks the diff against each loaded rules file’s Audit checklist before presenting it.
3. AUDIT mode hunts violations and reports findings as file:line | rule | severity | effort | fix. A full audit runs seven passes: recon → threat model → per-domain passes → silent-control pass (does each control confirmed to exist actually do anything?) → decision-ledger review → findings → refute before reporting. Scoping, evidence standard, severity model and report structure come from sota/rules/03-audit-findings.md (scoping and tooling from rules/01); the report ends in a roadmap sequenced by risk-reduction-per-effort. A verdict carried over from an earlier audit is bound to the commit it was made at and expires when that code changes, so a “fixed” finding that reappears stays open as a possible regression, never a duplicate (sota/rules/05 §4); the threat-model pass also mines version control for past security fixes and hunts each fixed class across the rest of the code (sota-threat-modeling rules/06).
4. If profiles/.md exists, its stack choices are BUILD defaults and the AUDIT baseline (deviations get flagged).

It scales the rigour to the stakes, and names the level it chose. A spike, a one-off script or a local experiment gets built and labelled as a prototype, with one line on what was left out — the full treatment is not applied to throwaway code. But anything reachable by an untrusted caller, or touching money, credentials or another tenant’s data, gets it whether or not the request said “quick”. The load-bearing half is the labelling: an unnamed shortcut is not a prototype, so proportionality is something the library grants out loud rather than something it does silently (operating principle 9).

Conventions

• Every rules file ends with an Audit checklist (yes/no questions, often with grep/lint patterns to hunt violations).
• Severity scale everywhere: Critical (exploitable/data loss) · High (fix this sprint) · Medium (bounded impact) · Low (hygiene) · Info (observations, no direct risk). Each finding also carries an effort estimate (trivial/small/medium/large) so remediation can be sequenced by risk-reduction-per-effort.
• Each SKILL.md carries a top-10 non-negotiables list — apply these unconditionally; load detailed rules files only as the task demands.
• Borderline severities state the deciding assumption; unconfirmed findings are marked “needs verification”, never asserted.
• A negative claim needs more proof than a positive one. “No instances of X” and “I only looked one way” are indistinguishable from the outside, so an absence claim requires a widened search plus a second independent method, and the search actually run is stated.
• A positive observation must show effect, not existence — the request it rejected, the log line it emitted, the test that fails when it’s disabled. An inert control praised as a strength is the worst reporting error available.
• Publishing under someone else’s name raises the evidence bar. A finding for the person who asked is cheap to retract; the same claim posted as them — a pull request review comment, an issue, a commit message upstream — is public, attributed and permanent. Verify by execution rather than inference, say what you did not test, and never publish on someone’s behalf without their approval of the final text.

Found a gap? Tell us — it’s the only signal we get

This library has no telemetry. Nothing reports back, by design. That also means a wrong rule, a stale version claim, or a task with no owning skill stays in the library for everyone until a human says so.

If a skill was wrong, outdated, or missing when you needed it: open an issue (bad-guidance / skill-request templates — both take about a minute). Dangerous or security-sensitive guidance goes to a private advisory instead.

The assistant will usually flag these itself: the router tells it to surface a one-line note when the library lets you down, rather than papering over it.

Sending more than a one-liner? Run /sota-report at the end of a working session in your own project. It asks that session to write up where the guidance failed, in the shape this library can act on, to a gitignored FIELD-REPORT-*.local.md — a field report carries a reproducer rather than a citation, which is why it is the highest-yield input here. The prompt front-loads the four checks that decide whether a report is actionable, each one a mistake actually made during intake.

It stamps the report with the version that produced it (report against what you actually used — never update and re-run, since that puts a fresh stamp on old behaviour), writes a gitignored FIELD-REPORT-*.local.md, and writes a short, generalised extract for publishing — finding, mechanism, whether the rule was already loaded, what caught it, proposed rule. It never posts anything: this repo is public and issues cannot be un-published, so it prints a ready gh issue create --body-file … line and stops. You read the file, then run it — or use the field report template if you have no gh. That template is how the library hears from anyone but its maintainer.

An old version is not a reason not to report. Measured over a month, 66% of rules files were untouched — a stale report is usually about text that still ships, and triage is one git log on the file it cites (docs/FIELD-REPORT-PROMPT.md).

scripts/install.sh (and therefore scripts/update.sh) installs the command into ~/.claude/commands/ alongside the others; verify-setup.sh check 1c reports whether it is actually reachable. The reasoning behind each check is in docs/FIELD-REPORT-PROMPT.md; the prompt itself lives in commands/sota-report.md and nowhere else.

If it saved you time, a ⭐ helps other engineers find it.

Contributing

See CONTRIBUTING.md. The short version: keep skills generic, verify fast-moving claims against primary sources, keep skill files (skills/**) ≤ 500 lines — that cap keeps incremental rule loading working and does not apply to README/CHANGELOG/docs/, which are read by humans — and end each rules file with an audit checklist — exactly one, since appending a section’s bullets under a fresh heading strands them where a reader has already stopped. 41 invariants enforce this in scripts/check-invariants.sh (pre-commit + CI — the one-line-each table with the incident behind each one is docs/INVARIANTS.md), covering line caps, checklist placement and uniqueness, description limits, version and count drift, router completeness, internal link resolution, every rules file being reachable from its skill’s index, a single [Unreleased] CHANGELOG entry, the LAST-VERIFIED stamp moving only with a sweep, a rendered assets/*.png never being older than the *.html it comes from, every scoreboard row declaring its sample size, a release declaring the front door terms its new capabilities landed on, a release that edits a skill description declaring a routing check — a description is the whole auto-load classifier, so changing one competes for every neighbour’s traffic — every § reference resolving, every check having a known-bad, every shipped CHANGELOG version carrying a git tag and its own link reference, every eval runner being named in the harness’s own README, the roadmap’s open set agreeing with itself, every eval case set declaring how its cases were chosen, no audit-checklist bullet being stranded inside a code fence, no absence reported through a stderr-suppressed || echo — cmd 2>/dev/null || echo "missing X" fires on every non-zero exit, so it manufactures findings about other people’s code, and it shipped in nine of these skills until a field report caught it — no grep probe whose --include filters out the very file it names, no version pin on a claim about what is current (a “latest release is X.Y” rots silently; the rules say “latest stable” and keep version numbers for boundaries such as “fixed in”), and every top-level area declaring which gates cover it, because enforcement density is uneven and an ungated area looks exactly like a covered one (the Coverage by area table in docs/CONVENTIONS-LEDGER.md) — plus gitleaks (full-history scan in CI; per-commit via the pre-commit hook).

What the invariants deliberately do not check is whether a rule is true. They are structural; an external audit found 19 wrong statements with all of them green. The measurable slice is behaviour against the library before your change — evals/run-prompt-independence.py builds its comparison arm from git show :, so it reads the real prior text rather than a mirror that drifts. When a prose edit is worth that run, and the four shapes that are not: docs/PROSE-REGRESSION.md.

Separately, scripts/check-claims.sh re-runs the library’s executable claims — the ones a rule’s advice actually rests on — on both Linux and macOS every CI run, because several are platform splits and one runner can only confirm its own half. A measurement nobody re-runs rots silently, and it rots in the direction of the recommendation: sota-shell-scripting rules/05 §3b shipped with GNU sed marked not verified because the binary was absent locally, and measuring it inverted the advice. The harness refuted one of its own on its first run. Ideas taken from outside the repo are recorded with a verdict and reason in docs/ADOPTION-LOG.md, so a rejection isn’t re-litigated. Security issues and conduct: SECURITY.md, CODE_OF_CONDUCT.md.

License

© 2026 Martin Holovsky. Two licences, split by what the file is:

• Content — skills, rules, docs, eval cases and everything else not listed below — is CC BY 4.0, Creative Commons Attribution 4.0 International. Use, adapt, and share freely (including commercially); just give attribution: “SOTA Engineering Skills by Martin Holovsky, CC BY 4.0.”
• Code — every *.sh and *.py file, hooks/hooks.json and .github/workflows/* — is Apache-2.0 (from 2026-10-02; earlier releases shipped it under CC BY 4.0). CC licences are not designed for software; Apache-2.0 adds the patent grant and the warranty disclaimer code needs.

profiles/ holds personal stack profiles and is git-ignored except profiles/example.md.template — copy that to profiles/.md and edit it; your real profile stays local and is never committed.