Tsukkomi MCP (ツッコミ)

A deterministic governance and evidence server for autonomous coding agents, version 0.8.0. The distribution is tsukkomi-mcp; the Python package is self_directing_mcp.

Overview

Prompt instructions alone do not keep an agent from context drift, false completion reports or a forbidden command. Tsukkomi MCP indexes the agent’s own session history on the local machine, checks proposed actions and finished work against explicit, user-registered contracts, tracks requirement checklists that need linked evidence, and optionally maintains a session knowledge graph.

Every check returns one of four verdicts: clean, violation, suspicious or unknown. unknown means the evidence needed for a verdict is missing or has changed, and it is never reported as clean. The one incomplete input that can still yield clean is a pure append to a growing session, which is judged as of the captured offset and reports that offset (see Coverage and verdicts). Whether a verdict blocks anything depends on the host (see Supported hosts). Rules are matched with regular expressions and structured evidence, not with an LLM judge.

                  ┌──────────────────────────────────────────────┐
                  │          Autonomous Agent Client             │
                  │ (Claude Code / AGY / Codex / OMP / Grok CLI) │
                  └──────────────────────┬───────────────────────┘
                                         │
                      Lifecycle hooks (host dependent) or MCP calls
                                         │
                                         ▼
┌────────────────────────────────────────────────────────────────────────────────┐
│                           Self-Directing MCP Engine                            │
├───────────────────────┬────────────────────────┬───────────────────────────────┤
│  Invariant Contracts  │   Verifiable Evidence  │      Session GraphRAG         │
│  - must / must_not    │   Checklist Gate       │  - Entity dependencies        │
│  - Session isolation  │   - No done without    │  - NeoGraph engine            │
│  - Regex + temporal   │     linked evidence    │  - Context injection (AGY)    │
├───────────────────────┴────────────────────────┴───────────────────────────────┤
│                 Local session index (SQLite, FTS5, sqlite-vector)              │
│     - Cosine vector search  - FTS5 BM25 and regex  - Known-secret masking      │
└────────────────────────────────────────────────────────────────────────────────┘

Session evidence stays on the local machine. Remote calls are limited to embeddings, GraphRAG extraction and the opt-in workflow classifier, all sent to OpenRouter. The canonical index keeps raw text so audits see the truth; everything that leaves it (tool results, errors, hook output, logs, remote payloads) is masked for known secret patterns, and the workflow classifier refuses context in which a secret pattern is found. See the masking contract, SECURITY.md and Verification scope.

What is new in 0.8.0

Version 0.8.0 (see CHANGELOG.md) is the result of the storage roundtable (#33): how the evidence and the contracts are stored, and what can move between devices without moving authority.

• A compact sparse index. sparse-v2.sqlite is an FTS5 contentless table: about 70 % smaller (a synthetic 100,000-event Codex-like corpus: 918.8 MB to 269.1 MB) with identical ranked results, and session-scoped searches 7 to 30 times faster. The old sparse.sqlite is never touched; convert once with python -m self_directing_mcp.maintenance sparse-rebuild. See Sparse index formats.
• Portable contracts. Contract bundles (a project file or a personal bundle) are only candidates until the user adopts them (adopt_contracts, anchored to a message of the local index and pinned by SHA-256); a changed or deleted file never weakens an adopted contract. export_contracts writes a bundle from the contracts of one device.
• pgvector as an opt-in dense backend. pip install 'tsukkomi-mcp[pgvector]' and SELF_DIRECT_DENSE_BACKEND=pgvector keep the vectors in a PostgreSQL server: 2 to 4 ms for a session-scoped search at every size and about 2.5 times faster exact scans at 100,000 vectors, against slower writes and 1.8 times the disk. The connection string comes only from a private file, a remote server requires sslmode=verify-full, and there is never a silent fallback to another backend. SQLite stays the default. See Dense backends.

---

What is new in 0.7.0

Version 0.7.0 (see CHANGELOG.md) mainly makes big sessions converge under the request deadlines. It was found by reproducing the report that timeouts also happen without an external model; the design record is #26.

• A sync that cannot finish keeps its progress. A request whose budget (8 s for hooks, audit_session and check_action; 45 s for the other tools) is too short for a cold or appended source stops at a record boundary and publishes the prefix it parsed; the next call continues. A 300,000-event synthetic session that never converged now converges in 5 calls. See “Large sessions and deadlines”.
• A re-index keeps its progress without touching the index that exists. A parser-version bump, a changed source or changed AGY receipts are staged in a spool next to the index and published by one ordinary commit; an interrupted re-index leaves the old events, anchors and vectors as they were.
• An audit of a big session converges. A slice is sized by what judging it costs, a comparison never takes the answer with it, and a resident server finishes what a deadline cut.
• A sync that runs out of time no longer takes the judgement with it. The stored index is judged with sync_deadline or sync_in_progress in the coverage issues: nothing is clean, a confirmed must_not match is still a violation.
• audit_status.ops and audit_status.evaluation say how each tool’s requests ended and how far a session’s monitor states lag, even while the engine is busy.
• Host-side GraphRAG extraction (graph_next_batch, graph_submit_batch, SELF_DIRECT_GRAPH_EXTRACTOR=host) and an O(batch) read of a graph batch.
• Incremental contract audits, source integrity and the adaptive shadow comparison (see their sections), an opt-in resident daemon (SELF_DIRECT_DAEMON=on) and an opt-in write-ahead journal (SELF_DIRECT_SQLITE_JOURNAL=auto).
• Two quadratic costs removed. SQLite planned chunk_id IN (...) lookups as a scan of the whole session; a derived pass over 150,000 events went from 64 s to 7.7 s and appending 100,000 events to a session of 100,000 from 12.7 s to 3.4 s.

---

What is new in 0.6.0

Version 0.6.0 (see CHANGELOG.md) mainly fixes memory growth, append handling and concurrency in long-running servers. It also adds a Claude Code integration, a shared hook policy and an egress masking contract.

• Dependency: neograph-engine==0.14.1 (was 0.12.1). zstandard>=0.23 is new on Python before 3.14, for compressed Codex sessions.
• Compressed Codex sessions. Codex CLI 0.156 compresses older rollouts in place to rollout-*.jsonl.zst. They are discovered, synced, audited and searched like plain rollouts, streamed through a bounded decoder and never held in memory. See Compressed Codex sessions.
• Appending sessions sync normally (#4). Source capture streams a SHA-256 over the file up to its length at the first measurement and keeps no bytes. Records appended afterwards are reported as unread_bytes instead of failing the sync.
• Audits judge a growing session as of the captured offset. Results carry judged_through_byte, unread_bytes and judged_prefix; absence-based must obligations stay unknown while bytes are unread. See Coverage and verdicts.
• check_action does not trust older history for a proposed action (#8). A prerequisite that requires success stays unknown while required history is unread, because a newer failed run may be in the unread part.
• Bounded memory (#5). Single-flight sync per session, streaming reparse, per-session id queries, float32 vectors, a cap on concurrent heavy calls (SELF_DIRECT_MAX_CONCURRENT_CALLS, default 4) and malloc_trim on glibc. See Memory and concurrency.
• Concurrent checklist updates (#6). update_checklist serializes plain edits and returns a checklist_conflict status instead of raising or overwriting.
• Duplicate session sources (#10). Copies and continuations of an indexed session are recognised instead of reparsed on every sync (alias_of, ambiguous_session_source).
• Parsers. OMP parser v8 (host notices, images, steering attribution, session_init, model_usage, branch_summary, redactedThinking; #9) and Codex parser v3 (multi-agent messages, tool search, new session_meta fields). Each bump reparses lazily, one session at a time.
• CI. A new Linux memory-audit-regressions job runs small offline memory, freshness and concurrency tests next to the existing smoke job.
• Claude Code (docs/claude-installation.md). A claude provider (parser version 2) for Claude Code session and subagent transcripts, and opt-in command hooks installed by scripts/install_claude_hooks.py for SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, PostToolUseFailure, Stop and SubagentStop. By default (SELF_DIRECT_CLAUDE_ENFORCEMENT=confirmed_only) they deny only a confirmed violation and ask on suspicious; they never answer allow, SubagentStop only reports, and every internal error fails open with a visible warning.
• One hook policy for AGY and Claude Code (docs/agy-installation.md). The AGY hook default is now confirmed_only (it was enforced): unknown passes with a visible warning unless the contract is marked risk: "high", a Stop continues the agent at most once per conversation and only for a confirmed auto_continue miss, and hook errors fail open with a hook_health banner. enforced stays an explicit opt-in, and advisory never gates an approved workflow either.
• AGY parser version 7. Truncated fields fall back to the same step of transcript_full.jsonl; what stays cut is unknown, never clean. RUNNING, ERROR, INVALID, CHECKPOINT and media records are handled, and results from before a conversation’s first hook receipt are classified as legacy_unlinked. Version 7 adds the Antigravity IDE’s typed result steps (RUN_COMMAND with its host exit_code, VIEW_FILE, CODE_ACTION, MCP_TOOL, LIST_DIRECTORY) and context-loading steps (CONVERSATION_HISTORY, KNOWLEDGE_ARTIFACTS), and accepts a host-written run of NUL bytes in front of an otherwise valid record.
• Contract fields risk, auto_continue and when, and four disabled high-risk presets. See Contract fields.
• Masking contract. Every egress path is masked, the sparse text index is masked for new rows, and a logging filter covers server logs. See the masking contract.
• Also in this release since 0.5.0: the shared-index snapshot and request-deadline work (#3), the AGY evidence workflow and approved-workflow layer, user path configuration in TOML, and a default index directory outside the checkout. Read Upgrading to 0.6.0 before restarting existing servers.

---

Key capabilities

1. Invariant contracts (upsert_contracts)

• Prohibitions (must_not) match a regex against proposed or recorded actions (decision: deny on hosts that enforce). A contract about a file must also match the calls of the host’s file-read tools (Claude Code Read, AGY view_file, OMP read), whose text is [tool_call:] {json} and not a shell command, so name the file itself rather than only shell readers such as cat; the fragility report in the response lists such spellings.
• Obligations (must) require matching evidence before a gated action (before_regex) or at a completion checkpoint, optionally with a successful linked tool result (requires_success: true).
• A contract’s origin must be a user message. source_event_id names the history event a contract was derived from, and only the user can be the origin of one: a tool result, runtime text or injected context cannot. In an audit, a source_event_id that resolves to anything but a user-role event (meta.role == "user") makes that contract’s findings unknown with the reason anchor_not_user (reference evaluator and monitors alike); at upsert time such an anchor is rejected with an error when the event is already indexed, and accepted when it cannot be resolved yet (the session is not indexed), leaving the audit to decide. What counts as a user message is each parser’s own classification: Claude Code origin.kind == "human" (plus a legacy prompt from before origin stamps), AGY USER_INPUT from USER_EXPLICIT, OMP a user-role message not attributed to an agent. Codex cannot tell a typed prompt from user-role context injected into the model’s input, since it records both as user turns, so both are accepted. applies_from_event_id may still be any event. The upsert response reports each anchor check (anchor_checks); it is not stored on the contract.
• Weakening a contract needs a user anchor. Disabling a contract, or replacing it with one that may enforce less (a different regex, an added or changed exception_regex, a different before_regex, a narrower scope, fewer roles, an added when, a narrower provider or session, dropping requires_success, lowering risk, dropping auto_continue), must carry a source_event_id that resolves to a user message. Without one the proposed revision is stored under pending in the contracts file and listed by list_contracts, the previous revision stays active (hooks and audits keep enforcing it), and the response says NOT APPLIED. Repeating the change with a user anchor applies it and clears the pending entry. Strictly tightening edits (a new verifier, an added role, a wider scope, removing an exception, enabling) need no anchor, and moving applies_from_event_id is not judged. The pending key is omitted from the file while empty.
• A call the host refused is a rejected proposal, not an executed violation. When a hook or the user’s permission prompt denies a tool call, the transcript still holds the call. Claude Code records its decision on the record of the linked tool result: a top-level permissionDecision object (kept as meta.permission_decision) beside toolDenialKind, with an error result. A decision of {"decision": "reject", "source": "hook", ...} (whatever its source: hook, user, rule, …) marks its call as refused, provided that call is the only one with that id; the older forms "deny" and {"behavior": "deny"} are kept. This shape was verified against Claude Code 2.1.299 with a synthetic probe (a PreToolUse hook answering deny for Bash). Any other value (accept, allow, an unknown or missing decision) is an attempt. A must_not match in a refused call is not an executed match: it is reported separately in the finding’s proposals_rejected (contract_id, event_id, call_id, reason, denied_by (the decision’s source, or host), denial_event_id, at_byte; omitted while empty; at most 20), the verdict comes from the remaining evidence (clean with the reason no executed forbidden match ... when every match was refused), and an exception_regex is matched only against what ran. Only the host’s own record counts: a result that merely says error, or a decision of allow or ask, is still an attempt. AGY already treated a model proposal that the Pre hook denied (host_rejected_proposal) as non-execution; it is now also listed in proposals_rejected (tool_name, at_byte, denied_at_byte). Codex, OMP and Cursor / Grok Bot have no host-recorded denial that their parsers can read reliably (Codex hooks never block; the OMP extension’s block leaves no marker the parser knows), so a blocked call there stays an attempt. A must obligation and a rule that reads tool output (scope tool_result, or any with the tool role) are unchanged: a result that echoes the command still matches. The Claude Code and AGY Stop notes list rejected proposals on their own line (“Rejected by the host, not executed …”), apart from “Not verified” and from any executed violation.
• requires_success needs a success that belongs to the required command. A host records one exit status per command line. pytest -q || true, || :, pytest; exit 0 (any later command that replaces the status), a command sent to the background (&, or Claude Code’s run_in_background), pytest | tee log without set -o pipefail earlier in the same line, and pytest --collect-only / --co all end in a recorded success without the tests having passed, so that success is not evidence: the finding is unknown with the reason success_unverifiable (never clean, and never a violation). A result recorded as failed is still a confirmed failure (violation), a call with no linked result keeps its no unambiguous completion status reason, and the command itself is never blocked. The check is a small quote-aware parse of the line (audit/shellstatus.py, also reading bash -c '...', ( ... ) and { ...; }): an operator inside quotes, $( ) or a comment is not an operator, set +e alone changes nothing, || exit 1 and ; exit $? keep a failure, and the last command the contract’s regex matched is the one whose status counts. The latest matching call still decides, so a newer masked run supersedes an older clean one. Both evaluators call the same function.
• AGY returns force_ask for suspicious exceptions so a cached Always Allow grant does not replace confirmation; Claude Code, which has no force-ask, returns ask.
• Contracts are global or scoped to one session. A description-only contract stays unknown, and search rankings are never violation evidence.
• A tool-call contract matches the payload, not the model’s commentary. Some hosts let the model write a free-text field about a call: Claude Code’s description on Bash (and on the other built-in tools that take one), AGY’s toolAction and toolSummary. That prose is excluded from the call text a contract is matched against, in indexed history and in the proposed action a hook asks about alike (one builder, tool_payload.py), so ls -l data/app.sqlite3 is no longer a sqlite3 violation because its description says “without connecting to it with sqlite3”. Only confirmed prose fields are excluded and only when their value is a string; every other field stays payload, an MCP tool’s own description argument included, and a violation inside command is caught whatever the description says. The prose is kept in the event’s meta.tool_prose. Matching it with an explicit scope: "prose" (at most suspicious) is not implemented yet. Fields suspected but not confirmed (AGY Description/Instruction, Codex justification, OMP _i) are deliberately not excluded.

Contract fields: risk, auto_continue and when

Three optional fields refine how hosts act on a contract. They never change what a regex matched. At their defaults they are omitted from the stored contract, so existing contracts.json files load and are rewritten unchanged and older processes keep reading files that do not use them.

A high-risk must_not must have a regex naming the actions it covers: a description-only one would apply to every tool call and could never be verified, so every call would be asked (a denial in headless runs). upsert_contracts rejects it.

Contract regexes (regex, before_regex, exception_regex, when.tool_call_regex) and search_history regex queries run on the regex engine with a time limit: 0.25 s per search and 2 s per contract per evaluation. A pattern that exceeds it makes only that contract unknown (regex_timeout), never clean or a violation, so a pathological pattern cannot stall a hook past the host’s timeout. The syntax is still Python re syntax.

Set these only when the user asked for them. The high-risk presets in src/self_directing_mcp/resources/high_risk_presets.json are four example must_not contracts with risk: "high": preset-no-recursive-delete, preset-no-write-outside-workspace (a path-pattern approximation to replace with your own boundary), preset-no-force-push and preset-no-branch-delete. They ship with enabled: false and exist in the contract store only when SELF_DIRECT_SEED_EXAMPLE_CONTRACTS=true, where they are added if their id is missing and are never altered or re-enabled. The owner enables the ones they want.

Contract fragility report (informational)

A regex over command text is easy to bypass with an equivalent spelling (rm -r -f for rm -rf), channel (python3 -c "shutil.rmtree(...)") or lexical trick (a backslash-newline, busybox rm, a GNU option abbreviation). A regex written around shell commands also never sees a host’s own tool call: a model told not to print the secrets in .env can open the file with the file-read tool, whose call text is [tool_call:Read] {"file_path": "/work/.env"}, and (cat|head|tail|less)\s+\S*\.env does not match it. The upsert_contracts response therefore ends with a fragility section that says which of those spellings the regex you just saved does not match.

• How it is chosen. The server holds a versioned library of equivalent spellings (resources/contract_variants.json) in thirteen families: recursive_delete, force_push, discard_worktree, world_writable, remote_script_exec, read_secret_file (.env), read_env_production, read_ssh_private_key, read_aws_credentials, stage_all, db_cli_access, skip_tests and tls_verify_off. A family is reported only when the regex matches one of its canonical (standard) spellings; the section then lists that family’s other spellings the regex misses, with their platform, technique and preconditions. Families the regex is not about are not mentioned. For a must_not contract the regex is analyzed; for a must it is the gated action (before_regex).
• File reads and tool calls. The four file-reading families (one per file: .env, .env.production, ~/.ssh/id_rsa, ~/.aws/credentials) have plain shell readers as their standard spellings (cat, head, tail, less, more) and list, besides other shell readers (sed -n, awk, grep, cp, xxd, base64, python3 -c, source), the file-read tool calls of three hosts as spellings of the platform tool_call: Claude Code [tool_call:Read] {"file_path": ...}, AGY [tool_call:view_file] {"AbsolutePath": ...} and OMP [tool_call:read] {"path": ...} (a Grok Bot Read call with a path argument is the same text under the case-insensitive match; Codex reads files through its shell, which the shell spellings cover). Each is the text the session parsers and the hooks build, and each is listed on its own in a capped list. A regex that already matches those calls, such as one naming the file (\.env), is told nothing about them, and a regex that matches only tool-call text selects no family. Because selection is a substring search, a regex that names the .env prefix (cat\s+\.env) also matches the .env.production standard spellings and is reported for both families; end the name (\.env(?![\w.])) to stay with .env. A regex about .env.production is not told about the .env family.
• Informational only. It never blocks a registration, never edits or broadens a regex and never changes a verdict: findings still come from the saved regex alone. Use it as a prompt to widen the regex deliberately, if you want to. exception_regex and when are not analyzed.
• Unverified candidates. Every library entry has verified: false: equivalence was not verified by execution, and the response says so ("verified": false and a note). A listed spelling may behave differently on your platform or tool version; read its preconditions.
• Deterministic and offline. No model is involved. The report names the library_version that produced it. Each spelling is matched as plain text (a command line, or for a tool call its [tool_call:] {json} text) with the contract regex engine (case-insensitive, multiline, the same time limits). A regex that exceeds them produces an explicit "status": "regex_timeout" entry for that contract, never a failure of the save.
• Bounded. At most 8 spellings per family (missed_total counts all misses, omitted the unlisted ones; a capped list takes one spelling per technique in turn, and every tool call counts as a technique of its own), 4 families per contract (families_omitted), 20 contracts per call (contracts_not_analyzed) and about 3 s per call (skipped_time_budget). A contract without such a pattern, or whose regex matches every listed spelling, is omitted.
• Hook paths never load it. Only upsert_contracts imports the library, after the contracts are saved. If the library cannot be read, the response carries "fragility": {"status": "unavailable", ...} and the contracts are still saved.

Abridged example for {"id": "no-rm", "type": "must_not", "scope": "tool_call", "regex": "rm\\s+-rf"}:

"fragility": {
  "library_version": "1.1.0", "informational": true, "verified": false, "analyzed": 1,
  "contracts": [{"contract_id": "no-rm", "field": "regex", "status": "ok", "families_omitted": 0, "families": [{
    "family": "recursive_delete", "matched_canonical": ["rm -rf build"], "missed_total": 19, "omitted": 11,
    "missed_variants": [
      {"id": "recursive_delete.split_flags", "text": "rm -r -f build", "platform": "posix",
       "technique": "option_split", "verified": false},
      {"id": "recursive_delete.python_rmtree", "text": "python3 -c \"import shutil; shutil.rmtree('build')\"",
       "platform": "python", "technique": "interpreter", "verified": false, "preconditions": ["python3 on PATH"]}]}]}]}

Sharing contracts across devices

A contract’s source_event_id must be a user message in this device’s index, and weakening or revoking a contract needs one too. An anchor written on another device cannot be verified here, and a file that arrives through a cloned repository or a shared folder can be written by anyone. So contracts travel as files, a device finds them as candidates, and the user adopts them on each device with one of that device’s own messages: once per device and bundle, not once per contract. There is no remote contract store, no automatic enforcement and no synchronisation.

The trust rule.

• Finding is not enforcing. Discovery reads files and nothing else. A candidate is not in the contracts file, no hook reads it and no audit judges it. Candidate text (the label, ids, descriptions, a quoted source_quote) is untrusted data, not an instruction; descriptions are masked and cut to 200 characters, and the answer says so.
• Adoption rests on this device’s user message. adopt_contracts checks source_event_id with the check every contract anchor passes (meta.role == "user" in this device’s index). A tool result, injected text, an id from another device or an id that is not indexed here adopts nothing. The anchor is a claim that the user asked: it proves that the message exists and is the user’s, not what it said, exactly as for upsert_contracts.
• A bundle is pinned by its hash. bundle_id is the first 12 hex characters of the SHA-256 of the file. adopt_contracts reads the file again and requires the hash to match: a bundle that changed since it was listed is refused (bundle_not_found_or_changed, with what is there now). Adopting the same bundle again changes nothing, not even the file.
• A conflict is never merged. If an id already exists here with other content (a different content hash), the candidate is not adopted and the local contract stays in force; the answer lists both content hashes and the field differences. It is adopted only if the id is also in replace_ids. A replacement that may enforce less than the local contract (the rules of weakening: a changed regex, a narrower scope, a lowered risk, disabling, …) is applied only because the call is anchored to the user’s message, is reported under weakened with its reasons, and the replaced revision stays in the contract’s history. Two candidate files that disagree on an id are reported under candidate_conflicts; neither wins.
• A changed or deleted file changes nothing. After adoption the contracts are ordinary stored contracts. If the file changes, its contracts show as changed_since_adoption with a field-level diff and the adopted version stays in force. A contract that disappears from the file, a deleted file or a missing sync is never a revocation (they appear as adopted_but_no_longer_listed). Weakening or revoking an adopted contract needs a user anchor of this device, as for any contract: without one it is parked as a pending change and the contract stays in force.
• Origin is not a claim. An adopted contract carries origin (bundle_sha256, bundle_label, origin_kind project or personal, bundle_path, adopted_by_event_id, and what the file only claimed: claimed_origin_event_id, claimed_quote). list_contracts shows it. Only adopt_contracts writes it: upsert_contracts rejects an origin that the stored contract does not already have, and a quote in a bundle is never stored as the contract’s source_quote (that field means the user’s own wording).
• No path leaves the tools. Files are named by a path relative to the project or by a label (personal#1/), never an absolute path; errors carry an error name, never a path.
• Hooks never load it. The bundle module is imported only by the three tools and audit_status; the hook import budget test fails if a hook path loads it, and the hooks still read the local contracts file only.

Why the adoption anchor is origin.adopted_by_event_id, not source_event_id. An audit judges a contract whose source_event_id is not an event of the audited session as unknown (“contract source or activation event is not in this session”), global contract or not. A global contract anchored to one session’s event would therefore be enforced in that session only and unverified everywhere else on the device. The adoption anchor is checked when the contract is adopted and recorded in origin; the stored contract is global with no source_event_id, like any global contract registered without one.

The file. .tsukkomi/contracts.json, or any file listed in SELF_DIRECT_CONTRACT_BUNDLE_PATHS:

{
  "schema": "tsukkomi.contracts-bundle/1",
  "label": "team contracts",
  "contracts": [
    {"id": "no-rm", "type": "must_not", "scope": "tool_call", "regex": "rm\\s+-rf", "severity": "high",
     "description": "Never delete recursively.", "roles": ["assistant"]},
    {"id": "tests-ran", "type": "must", "scope": "tool_call", "regex": "pytest", "risk": "high"}
  ]
}

A contract has the portable fields of a contract: id (stable, at most 200 characters), type, scope, severity, description, regex, before_regex, exception_regex, search_query, requires_success, risk, auto_continue, when, roles, provider, enabled. It has none of the device-bound ones: source_event_id, applies_from_event_id, session_id, revision and origin are rejected by name. source_quote and claimed_origin_event_id are accepted as plain text (shown as claimed, never an anchor). The file is read strictly and as data: UTF-8 JSON of at most 1 MiB and 200 contracts, no includes, nothing executed, no network; unknown keys, duplicate keys, wrong types, control or bidirectional characters, over-long text (regexes at most 1,000 characters), a regex that does not compile and a high-risk must_not without a regex (the checks upsert_contracts applies) reject the whole file, and the reasons are listed under refused.

Where candidates are found. The project bundle: from the project directory up to the project boundary, the first directory that holds .git (a file or a directory; a nested repository is its own boundary), else the project directory itself; every .tsukkomi/contracts.json on the way is read, nearest first. The walk never looks at the user’s home or the filesystem root, so a dotfiles repository in the home directory is not a boundary. project_dir defaults to the working directory the session’s own index names (Codex session_meta.cwd and Claude Code records); OMP and AGY sessions name none, so pass project_dir (the answer says so). Personal bundles: the files listed in SELF_DIRECT_CONTRACT_BUNDLE_PATHS. Every file is opened once and checked on the descriptor: a symbolic link out of the project is never followed, and on POSIX a file that is not a regular file, is owned by another user or is world-writable is refused (symlink_leaves_project, not_a_regular_file, not_owned_by_user, world_writable, too_large, not_json, invalid_bundle, …). On Windows the owner and ACL of a file are not inspected.

audit_status has a contract_bundles section, {candidates, pending_adoption, changed_since_adoption, conflicts, refused, project_directory_known} (bundles waiting for adoption, contracts changed since adoption, ids in conflict), counted for the project directory the session names (no project bundle is counted without one; personal bundles always are), with no path and no contract text.

Settings. SELF_DIRECT_PROJECT_CONTRACTS (candidates, the default: discovery only, nothing is enforced without adopt_contracts; off: the project bundle is not looked for, and export_contracts writes no project file) and SELF_DIRECT_CONTRACT_BUNDLE_PATHS (absolute paths separated by :, ; on Windows; TOML contract_bundle_paths in [paths]; also the only places besides the project directory that export_contracts may write). Keep a working-directory .env private: the server reads it, and it can name these paths.

Export. export_contracts writes a path inside the project (relative to its root, such as .tsukkomi/contracts.json) or a configured personal file (its path or personal#N), and refuses any other path, a path that leaves the project through a link, replacing a file that is not itself a valid bundle, and writing through a symbolic link. The file is written to a temporary file and renamed (mode 0644 for a project file, 0600 for a personal one, whatever the umask) and holds the portable fields only: no anchors, revisions, quotes or origins. Contracts limited to one session and (unless include_disabled) disabled ones are skipped and listed. The answer gives the sha256 and how many anchors were dropped.

Not done. No remote contract store and no synchronisation: the file moves with your repository or folder. No automatic enforcement and no hook hint that a bundle is waiting (a hook would have to load the bundle module; audit_status carries the count). ContractRepository (in audit/contracts.py) is the protocol of the operations the engine performs on its contracts; the local file store is its only implementation, and the hooks do not go through it. An older version cannot read a contracts file that holds an adopted contract (the origin key is new): upgrade every process that shares the index first.

2. Checklist with evidence gating (update_checklist)

• Track user requirements with provenance and completion criteria.
• A completion claim without evidence is downgraded to pending_verification. An item becomes done (verified=true) only when its evidence_chunk_ids resolve to successful tool results linked to their calls and session coverage is complete. While appended bytes are unread, coverage is incomplete, so the item stays pending_verification until a later call sees the caught-up history.
• Concurrent updates do not lose each other’s edits. An update that needs evidence evaluation is retried up to three times; if the checklist or evidence keeps changing, the call returns {"ok": false, "status": "checklist_conflict", "action_executed": false} and saves nothing.

3. Session GraphRAG and context injection (get_graph_context)

• Entity dependency graph (nodes, relations, cursors) in SQLite, executed through the NeoGraph engine.
• Typed relations: IMPLEMENTS, DEPENDS_ON, MODIFIES, PRODUCES, CHECKS_VERSION, EVIDENCED_BY, SUPERSEDES.
• Graph updates are validated against session history and applied idempotently. Updates require an explicit call, run_neograph_update or graph_next_batch plus graph_submit_batch; lifecycle hooks never send session text to a model.
• The extraction can also be done by the calling agent instead of an OpenRouter model: Host-side GraphRAG extraction.
• In Google Antigravity, active graph relationships are delivered into agent context before each tool invocation.

4. Hybrid history search

• Dense cosine similarity (sqlite-vector, native, by default; NumPy or PostgreSQL with pgvector when selected: see Dense backends), SQLite FTS5 BM25 and regex, fused with reciprocal-rank fusion.
• Secret masking: known credential patterns are masked before embedding, in sparse search text and in everything returned to the client. This is not a guarantee that arbitrary personal data or every secret format is removed; see the masking contract.

---

Supported hosts and parser versions

A parser version is recorded per session. When a provider’s version is raised, each already-indexed session is reparsed from byte 0 on its next sync or audit, one session at a time. Unchanged event ids, anchors and cached vectors survive, and no embedding API is called when embed=false. A running MCP process only picks up a new parser after it is restarted.

AGY parser (version 8)

Fields the host cut (truncated_fields) are restored from the same step of the sibling transcript_full.jsonl when it matches; otherwise the visible text is kept, marked as truncated, and any verdict that depends on the cut field is unknown with a truncated_ reason. A match that is visible still counts. RUNNING results are pending (pending_running), ERROR, INVALID and CANCELED results are never success, CHECKPOINT and ERROR_MESSAGE records are runtime context, and media payloads become placeholders. Results from before the conversation’s first hook receipt are legacy_unlinked and leave dependent obligations unknown (unsupported_legacy); coverage reports receipts_since, legacy_unlinked_count and result_dependent_complete. Details are in docs/agy-installation.md.

Version 8 keeps the model-written toolAction and toolSummary labels of a tool call (they sit beside CommandLine in the recorded fixtures) out of the call text, for hook receipts and for model proposals, and records them in meta.tool_prose; every other argument stays payload.

Version 7 reads two more shapes. The IDE writes a tool result as a step of its own type, RUN_COMMAND, VIEW_FILE, CODE_ACTION, MCP_TOOL or LIST_DIRECTORY, and the parser treats it exactly like the CLI’s GENERIC result: it is linked to its call only through a host receipt, truncated content falls back to transcript_full.jsonl or leaves dependent verdicts unknown, and RUNNING is pending. A linked RUN_COMMAND with status DONE and host exit_code 0 is success evidence, a nonzero exit_code is a failure, and an unlinked result is never success. CONVERSATION_HISTORY and KNOWLEDGE_ARTIFACTS are runtime metadata. A contiguous run of NUL bytes at the start of a line is stripped when the rest is a valid record (host_padding_bytes, counted in agy_evidence.host_padded_records); any other NUL use, and any other unknown step type, stays a parse error or an unsupported event. See docs/agy-installation.md.

Hook receipts cost what was appended, not the size of the transcript. A receipt hashes the file once and parses only the records after a prefix it has verified; a sync checks every receipt anchor in the pass that hashes the file; and a receipt reindexes the conversation only when it can change an event that is already indexed. On a synthetic 10 MB transcript with a linked receipt per result, one PreToolUse hook (receipt and contract check) fell from 20.5 s to 0.51 s and a PostToolUse hook (receipt and sync) from 20.4 s to 0.58 s, against 0.38 s and 0.36 s at 0.1 MB (whole hook processes, wall time). The mechanisms and what they fall back to are in AGY hook cost; scripts/bench_agy_hooks.py reproduces the table.

Claude Code parser (version 3)

Only a record the host stamped origin.kind == "human" (or, in a main transcript, a legacy prompt from before origin stamps, or a human queued_command prompt) is a user turn. Task notifications, peer and coordinator messages, injected meta messages, compaction summaries, attachments, hook output, tool results and a subagent’s task prompt are runtime evidence. Unknown record, attachment, block and origin shapes are unsupported events that keep coverage incomplete. Opaque payloads (thinking, images, hook output) are placeholders. A subagent transcript is its own session, _agent-. Version 3 keeps the model-written description of a built-in tool call (Bash, Task, Monitor, …) out of the call text and records it in meta.tool_prose; the call text contains only the other input fields, exactly like the proposed action a PreToolUse hook asks about. An MCP tool’s own description argument is payload. Version 2 accepts the system record scheduled_task_fire (a scheduled task fired) as runtime metadata: its prompt is automation text, not something the user said, so it is validated, never indexed, and cannot satisfy or violate a user-role contract; every other unknown system subtype stays unsupported. See docs/claude-installation.md.

Grok CLI parser (version 1)

updates.jsonl is the only file read (chat_history.jsonl and system_prompt.txt hold the rendered system prompt and are refused by the resolver). A user_message_chunk is a user turn only when Grok stamped it with its promptIndex, it does not open with the `` wrapper Grok puts around hook-injected text and it is not replayed from another session; everything else is runtime text and cannot anchor a contract. Streamed agent_message_chunk records are joined into one message (a run still open at the end of a read is emitted as it stands); agent_thought_chunk becomes a [thinking] placeholder, never indexed text. A tool_call is a call and a tool_call_update with a terminal status its result, linked by toolCallId (failed, a non-zero exit code, a timeout or a signal are failure; a call with no result is in flight or cancelled, never success). The host’s own hook_execution and hook_annotation records, which describe this server’s denials, are skipped; any other unknown record kind is an unsupported event that keeps coverage incomplete. A spawn_subagent child is another session directory and is not followed.

Tool names are rendered in Claude Code’s vocabulary at the one boundary where text is matched (the parser, the hook’s proposed action and the hook-side pre-checks), so the contracts you wrote for Claude Code match the same call from Grok. The map (src/self_directing_mcp/grok_tools.py, TOOL_MAP_VERSION = 1; the native name and arguments stay in the event metadata):

A contract that names a Grok spelling (run_terminal_command, target_file) does not match; write contracts against the Claude vocabulary. Details, the reply formats and what a live run should verify are in docs/grok-installation.md.

OMP parser (version 8)

Runtime records are indexed as metadata or runtime-role events. They are never user instructions and never proof that a tool ran.

• Host notices, mode_change, ttsr_injection, developer messages, session_init, model_usage and branch_summary are runtime metadata. Only the parent’s task text of session_init is searchable; its prompt, tools and schema are not indexed.
• Image blocks become [image:] placeholders and thinking/redactedThinking blocks become placeholders; base64 and opaque reasoning payloads are never indexed, including inside blocks of an unsupported shape (only their key names are kept).
• Attribution: among steering messages only attribution: "user" is a user turn. Any user-role message with attribution: "agent", with or without steering metadata, is runtime text and cannot satisfy or violate a user-role contract.
• Partial steering metadata (steering or liveSteered without the full valid shape) and unknown attribution values are rejected as parse errors, which makes coverage incomplete.
• Unknown event shapes and unmatched execution-start markers still make coverage incomplete. service_tier_change is metadata only when serviceTier is null or a map from the known providers (openai, anthropic, google) to known tiers.

Codex parser (version 3)

• response_item agent_message (multi-agent messaging) and inter_agent_communication_metadata are runtime metadata, neither user authority nor tool evidence. encrypted_content blocks become a placeholder.
• tool_search_call / tool_search_output are runtime metadata. The search query is indexed; the returned tools list is not.
• session_meta headers, including the newer fields, are shape- and type-checked, and only a fixed subset of identifying fields is indexed. base_instructions, dynamic_tools, context_window, git, history_base, account ids and workspace roots are never indexed. The older behaviour of indexing a legacy header’s whole payload is gone.
• A complete new-style session_meta for another thread in the middle of a sub-agent file (an inherited parent header) is accepted as inherited metadata. A first-line id mismatch is still an error.
• For these newly supported record types, an envelope with metadata.client_authored: true is treated as unsupported because its meaning is unclear.
• Earlier support remains: world_state, token_usage_record, compacted and native web_search_call records. Unknown event types stay visible as unsupported.

Compressed Codex sessions

Codex CLI 0.156 compresses older rollouts in place: sessions/YYYY/MM/DD/rollout--.jsonl.zst, one zstd frame that declares its content size. A forked or sub-agent rollout may carry a second _ suffix. Session discovery, session_id lookup and explicit path arguments accept rollout-*.jsonl.zst under the configured Codex root with the same root, symlink and traversal checks as .jsonl. The session id comes from the session_meta header, as for plain files; a file name with two different UUIDs never names a session by itself. While a plain rollout and its compressed twin both exist, a lookup by id returns the plain (live) one.

• Offsets address the decompressed stream. Event ids, anchors, byte_offset, judged_through_byte and the stored prefix hash are exactly what the plain file with the same bytes yields, so Codex compressing a rollout in place, or a resume decompressing it and appending, is the same session: nothing is reparsed, no event id, anchor or cached vector changes, and no duplicate session or ambiguous_session_source appears. If the compressed file holds more records than the plain file had when it was indexed, only the new part is parsed. Plain and compressed copies that exist together follow the duplicate source rules.
• A compressed file is an immutable snapshot. There is no growing tail: coverage is complete when the stream has been read to its end (unread_bytes stays absent unless the file ends with an unterminated record, which is never parsed). A changed compressed file is a rewrite like any other: identical decompressed content (for example recompressed at another level) is no change, a changed earlier record is reindexed (source_changed), and more records after an identical prefix are a continuation. Growth of the compressed file itself is never treated as pure append.
• Streaming only. The decoder reads in bounded blocks; a cursor in the middle of a compressed source is reached by decoding and discarding from the first byte, so memory does not depend on the session size. On a 54.5 MB synthetic session (28 MB compressed) a full sync peaks at the same traced memory as the identical plain file, 16 MB, and an audit-style refresh at 3.5 MB; a 60 MB stream that compresses to 6 KB is decoded with a 2.4 MB peak. Source observation validates a compressed snapshot by the SHA-256 of the compressed bytes (no second decode); each audit or status check of an indexed compressed session decodes the file once.
• Strict frames and visible errors. A compressed source must be exactly one zstd frame that declares its content size, without a dictionary and with a window of at most 128 MiB (zstd’s own default). Anything else, and every unreadable file, is an explicit error with a code, never a skipped file or a short read: zstd_corrupt, zstd_truncated, zstd_size_mismatch (more or fewer bytes than the header declares), zstd_unsupported_frame, zstd_too_large and zstd_unavailable. sync_session returns ok: false with that error, nothing is indexed or changed, and an audit is unknown with the code in coverage.issues. A source that was indexed and later goes bad reports the code in coverage instead of source_unavailable_or_truncated.
• Size limit. SELF_DIRECT_MAX_DECOMPRESSED_SESSION_BYTES (default 4 GiB, about ten times the largest observed session) bounds the size the frame header declares; a larger file is refused as zstd_too_large before any byte is decoded, and a stream that disagrees with its header stops at the declared size. A plain .jsonl has no such limit.
• Decoder. Python 3.14+ uses the standard library compression.zstd; older interpreters use the zstandard package (zstandard>=0.23, installed with the project). Without either, a .zst source reports zstd_unavailable and plain .jsonl sessions keep working. The Codex parser stays at version 3: the parsed records and event ids are unchanged, so existing indexes are not reparsed.

---

MCP tools reference

Host-side GraphRAG extraction

run_neograph_update sends one batch per call to an OpenRouter model. Two tools let the calling agent (or a restricted subagent) do the extraction instead: they make no model call and no network call and do not use the OpenRouter key (a server that is not in local-only mode still needs its embedder configuration to start, as for every tool). They work with SELF_DIRECT_LOCAL_ONLY=true and with either SELF_DIRECT_GRAPH_EXTRACTOR value, like propose_graph_update and commit_graph_update, which make no model or network call either. SELF_DIRECT_GRAPH_EXTRACTOR=host also turns run_neograph_update off, so this server sends no session text to a model for graph extraction.

1. graph_next_batch(session_id) returns the next batch: complete events, masked exactly as run_neograph_update masks them, up to 100 complete events and 24,000 characters by default. max_events and max_chars ask for less, or for more up to 100 events and 95,000 characters, the limits run_neograph_update always uses. An event over the limit is graph_event_too_large (with the needed_max_chars to ask for, unless it exceeds 95,000 characters) and is never truncated. The result carries a batch_id, remaining_events, the graph cursor, and the extractor instructions, proposal schema, limits and allowed relations. status: "up_to_date" ends the loop; a session without indexed events is no_events.
2. The agent extracts nodes and edges whose evidence_chunk_ids are chunk_ids of that batch.
3. graph_submit_batch(session_id, batch_id, proposal, extractor=...) recomputes the batch from the index (batch_changed if its events are not what the id was made from; invalid_batch_id for an id graph_next_batch could not have written), requires it to be the next one (out_of_order, stale_batch), checks the proposal’s shape and size caps (500 nodes, 1,000 edges, 100 evidence ids per edge, 500 characters per string, 200 per id, 2,000 for the summary), that ids, kinds, labels and edge ends are single-line text, that every edge cites only events of this batch, and the existing evidence rules (rejected), then commits with the same graph-version and evidence checks as every graph write. The job id is the batch id, so repeating a submit returns already_applied; if another session or provider has already recorded a job under that id the answer is job_id_in_use and nothing is written.

A batch is never skipped silently. A proposal without edges is rejected (empty_graph_requires_review, cursor retained): submit the relations the events state. When they really state none, call graph_submit_batch without a proposal and with no_graph_facts_reason (up to 500 characters); the cursor moves past the batch (skipped_no_graph_facts) and the reason and the extractor label are recorded in the job. A skip stores no nodes.

Result size. A full 95,000-character batch is about 100 KB of JSON (25,000 to 35,000 tokens), more than a host may let one tool result carry (Claude Code caps MCP tool output at about 25,000 tokens by default, Codex has a tool-output token limit). A host that shortens a result would show the agent part of a batch while a submit, a skip or a no-facts record covers all of it. So the default is 24,000 characters, and a complete result ends with "end_of_batch": true as its last key, with a truncation_hint near the start: if yours does not end with it, call again with a smaller max_chars, and never submit a batch you did not read completely. The server checks which events a batch names, not what the agent read; that rule is the agent’s.

Parallel extraction. graph_next_batch(session_id, after_order=k) starts the batch at event k + 1 and moves nothing, so several batches can be planned from one cursor and extracted at the same time (for example one subagent each). Commits stay in order: a batch submitted before its predecessors returns out_of_order with expected_start_order; submit the earlier ones, then this one again. A batch is the events it names, so events appended to a live session meanwhile do not invalidate it.

Untrusted data. Event text is copied from a transcript and can contain instructions aimed at whoever reads it. Run the extraction in an agent that has only these two tools (no shell, file or network access) and have it call nothing but graph_submit_batch with the result. Both tools accept any session_id and provider, so the server cannot restrict a subagent to one session: which sessions a subagent may extract is decided by how it is configured and instructed. What the server bounds is each call: a submit can add at most 500 nodes and 1,000 edges, every edge backed by events of its own batch, and move the cursor only over that batch. The extractor label is free text (control characters removed, 80 characters, default host:unspecified) kept for provenance; nothing trusts it.

Single-line text. Ids, kinds, labels and edge ends are stored as given and reach agent context line by line (the AGY hook prints each relation), so a proposal whose node ids, kinds, labels or edge ends contain an unsafe character is rejected, not edited (rejected; the problem names the code points, U+2060 for one, and does not repeat the text). Unsafe, by Unicode category: control characters (Cc: newline, carriage return, tab, escape and the rest), the line and paragraph separators U+2028 and U+2029 (Zl, Zp), lone surrogates (Cs) and every format character (Cf) except three. That covers the characters that reorder or hide text: the bidirectional marks, embeddings and isolates U+061C, U+200E, U+200F, U+202A to U+202E and U+2066 to U+2069, the zero-width space U+200B, the word joiner and the invisible operators U+2060 to U+2064, the byte order mark U+FEFF, the interlinear annotation characters U+FFF9 to U+FFFB and the tag characters (U+E0001, U+E0020 to U+E007F). The other format characters, such as the Arabic number signs, are rejected with them, so the rule stays closed for characters added later. Three format characters are allowed because they only join or hyphenate and cannot hide or reorder anything: the zero-width non-joiner U+200C, the zero-width joiner U+200D and the soft hyphen U+00AD. Everything else is accepted, unassigned and private-use code points included: Hangul and the other scripts, Persian words with their zero-width non-joiner, emoji sequences joined by U+200D, Indic conjuncts. The extractor prompt and the text_rules of graph_next_batch tell the model this rule. The AGY context also prints every stored relation, contract description and checklist item on one line, showing such characters as escapes (rows stored before the check included).

Cross-vendor. Extracting a session with a different vendor’s model discloses that session to that vendor. By default extract only the sessions of the host you are running in.

Batch-range commits. commit_graph_update without through_order moves the cursor to the end of the whole session, even when the proposal was built from some of the events; propose_graph_update and commit_graph_update report that position as covers_through_order (also when they answer with a conflict). Pass through_order, the order of the last event the proposal covers (from the first unprocessed event to the last event, otherwise invalid_through_order and nothing is stored), to advance only that far. It is also the explicit way past an event too large for a batch: read it with get_chunk and commit the relations you find with through_order set to its order. A commit without a job_id is named by a digest of provider, session, the event order the commit moves the cursor to (through_order, or the last event of the session as it stands when none is given) and the proposal: repeating the identical call at the same place is already_applied; any other commit applies, the same proposal after the session has grown included.

Limits. The cursor is an event order, not an event identity. A session rewritten shorter and then grown again can skip events: with 10 events and the cursor at 9, a rewrite to 5 events reads as up_to_date, and after 10 more events are appended the orders 5 to 9 are never extracted. run_neograph_update behaves the same way. Job ids are global, not per session: a caller who has seen a batch id can claim it first, through another session’s commit_graph_update with that job_id, and so wedge that batch’s host path (graph_submit_batch then answers job_id_in_use, explicit and never a false already_applied, and the other session’s row is not touched) until the batch is committed with the documented through_order commit, which needs no job id.

---

Quickstart

Prerequisites

• Python 3.12+ and uv (recommended) or venv
• The official sqlite-vector 1.1.0 native library (the setup script below downloads and verifies it)
• An OpenRouter key for the default embedding and retrieval path; not needed in explicit local-only mode
• Optional: pip install 'tsukkomi-mcp[pgvector]' and a PostgreSQL server with the pgvector extension, only to keep the dense index there (Dense backends)

1. Clone and install

git clone https://github.com/fox1245/tsukkomi-mcp.git ~/.mcp-servers/tsukkomi-mcp
cd ~/.mcp-servers/tsukkomi-mcp

uv venv .venv --python 3.12
source .venv/bin/activate
uv pip install -e ".[dev]"

2. Provision sqlite-vector

python scripts/setup_sqlite_vector.py --output-dir /absolute/path/to/native

The script pins release 1.1.0, verifies the official archive’s SHA-256 and checks native cosine search. Without --output-dir it writes to .native/ in the current directory. Set SELF_DIRECT_SQLITE_VECTOR_PATH (or sqlite_vector_path in TOML) to the resulting vector.so, vector.dll or vector.dylib. With no setting, the runtime only looks for native/vector.* inside the source checkout. If the library cannot be loaded, engine initialization fails with an error; it does not fall back to NumPy. NumPy (SELF_DIRECT_DENSE_BACKEND=numpy) and SELF_DIRECT_USE_FAKE_EMBEDDER=true are explicit offline and test options.

3. Configure

Settings come from, in order of precedence: explicit Settings arguments, environment variables, the TOML path file, a .env in the current working directory, then defaults. A server started by an MCP host usually has an unrelated working directory, so do not depend on a .env there; use explicit SELF_DIRECT_* variables in the host’s server entry or the TOML file below. Lifecycle hooks (AGY and Claude Code) never read a working-directory .env: they use the environment and the TOML only, because that file is content the audited agent can write.

User path configuration (TOML)

Use ~/.config/tsukkomi-mcp/config.toml on any supported platform, or set SELF_DIRECT_CONFIG_FILE in the process environment to select another file. Copy and edit config.example.toml; the edited file belongs outside the repository and agent-writable workspaces.

[paths]
index_dir = "~/private-tsukkomi/index"
omp_sessions_dir = "~/.omp/agent/sessions"
openrouter_api_key_file = "../credentials/openrouter.env"
sqlite_vector_path = "native/vector.so"

The ten supported keys are index_dir, codex_sessions_dir, omp_sessions_dir, claude_projects_dir, grok_sessions_dir, grokbot_transcripts_dir, agy_app_data_dirs, contracts_path, sqlite_vector_path and openrouter_api_key_file. AGY roots use a string array; Grokbot accepts its existing delimiter-separated string or an array of individual paths. Empty arrays disable those optional roots.

TOML paths expand ~; relative paths resolve against the canonical TOML file’s directory, not the working directory or a symlink’s directory. ${ENV_VAR} interpolation is not performed. The selector SELF_DIRECT_CONFIG_FILE is environment-only, so a .env cannot silently select another TOML file. An absent default file keeps the defaults. An explicitly selected missing file, malformed TOML, an unknown section or key, or an invalid path value fails instead of silently selecting another index. Settings capture the selected sources, so restart affected servers and extensions after changing them.

Environment settings

All settings use the SELF_DIRECT_ prefix except the key itself. The names below are the runtime fields in src/self_directing_mcp/config.py; .env.example lists the commonly used ones.

Credentials

Credential values are not TOML options. A key file selected in TOML or by SELF_DIRECT_OPENROUTER_API_KEY_FILE is authoritative: a missing, unreadable or keyless file is an error, never a fallback to another key. Without a selected key file, OPENROUTER_API_KEY follows the precedence above (explicit argument, environment, working-directory .env). Keep the key file outside the repository and agent-writable workspaces. Workflow sandboxes reject mounts that expose the selected TOML, a host dotenv or the selected credential file.

---

Platform configuration

Google Antigravity (AGY)

Configure tsukkomi-mcp in your mcp_config.json:

{
  "mcpServers": {
    "tsukkomi-mcp": {
      "command": "~/.mcp-servers/tsukkomi-mcp/.venv/bin/python",
      "args": ["-m", "self_directing_mcp"]
    }
  }
}

To enable lifecycle interception in AGY, register scripts/agy_hook.py as your workspace hook command. By default only confirmed findings gate a tool call. See docs/agy-installation.md for the hook file, enforcement modes and host boundaries.

OpenAI Codex CLI

Install advisory lifecycle hooks and register the MCP server in one command:

python scripts/install_codex.py --trust-hooks
python scripts/install_codex.py --verify

The installer registers explicit SELF_DIRECT_INDEX_DIR, session-root, backend and key-file settings in the Codex MCP entry, so it is not affected by the default index location. See docs/codex-installation.md.

Claude Code

Register the MCP server and, if the user wants lifecycle controls, review and then install the opt-in hooks:

claude mcp add --scope user tsukkomi-mcp -- /absolute/path/to/venv/bin/python -m self_directing_mcp
python scripts/install_claude_hooks.py --print --python /absolute/path/to/venv/bin/python
python scripts/install_claude_hooks.py --apply --settings /absolute/path/to/settings.json --python /absolute/path/to/venv/bin/python

--print is the default and changes nothing; --apply merges into the settings file you name, keeps unrelated hooks, backs the file up when it changes and replaces it atomically. Add --index-dir /absolute/path/to/index to point the hooks at the index whose contracts should apply (the one the MCP server writes to); hooks never read a project .env. See docs/claude-installation.md for the policy table, the Stop budget, subagents, fail-open behavior and verification steps, and docs/claude-hooks.example.json for the complete hooks block.

Grok CLI

Register the MCP server with Grok (see the Grok documentation for your version; use an absolute interpreter), then, if the user wants lifecycle controls, review and install the opt-in hooks. They go into one dedicated file, /hooks/tsukkomi.json, and nothing else is read or written (never ~/.claude/settings.json, config.toml or trusted_folders.toml; the grok binary is never run):

python scripts/install_grok_hooks.py --print --python /absolute/path/to/venv/bin/python
python scripts/install_grok_hooks.py --apply --python /absolute/path/to/venv/bin/python --local-only --index-dir /absolute/path/to/index
python scripts/install_grok_hooks.py --verify --python /absolute/path/to/venv/bin/python --local-only --index-dir /absolute/path/to/index
python scripts/install_grok_hooks.py --uninstall

--print is the default and changes nothing; --apply writes (atomically, with a backup only when an existing file changes, and nothing at all for an unchanged result) and respects GROK_HOME or --grok-home; --verify also runs the registered command twice on synthetic data in a temporary directory. User-level hooks are always trusted by Grok and need no folder trust. Pass calls to the MCP tools with provider: "grok".

• Modes. confirmed_only (default) denies a confirmed violation, asks on suspicious, and allows unknown with a warning the model receives (an ask for a risk: "high" contract); advisory never gates; enforced also denies unknown. The same hook_policy table as the AGY and Claude Code hooks. Every PreToolUse reply is explicit ({"decision": "allow"} at least, never {}); Stop keeps the agent working at most once per session, never while stopHookActive, and never at the session-end Stop (reason other than end_turn).
• Failures are visible and fail open. Settings, contracts or engine unavailable, an invalid payload: the hook still allows, names the failure on the first line of stderr, records hook_health and exits 1 (Grok shows the failed hook and blocks nothing; the model hears of the unverified call with the next PostToolUse, because Grok drops the context of a hook that exits non-zero). Exit code 2, which Grok treats as a deny, is never used.
• Supported: the six events above, the tool-name map (see “Grok CLI parser” in Supported hosts and parser versions), the Claude-style evidence and Stop obligations on the parsed history, a transcript that shrinks, and a tool input that Grok truncated in the payload (toolInputTruncated), becoming unknown instead of compliance.
• Not supported (cut from this version): input rewrite (updatedInput) and output redaction (updatedToolOutput) — future work, left out because the model acts on altered input and output without knowing it, a partial mask can leave a secret reconstructible, binary and very large outputs have no safe text form, and the original result stays in updates.jsonl anyway; context injection at SessionStart and UserPromptSubmit (Grok discards it); the resident daemon; project-level .grok/hooks; following a spawn_subagent child session.
• Together with the Claude Code hooks. Grok loads ~/.claude/settings.json by default, so installed Claude hooks also run on Grok’s events. scripts/claude_hook.py recognises the Grok payload (a .grok/sessions transcript path, or Grok’s own camelCase envelope) before it imports the package and answers with silence: nothing printed or written, exit 0. Earlier versions printed a “check FAILED (PathTraversalError)” warning on every Grok tool call.

See docs/grok-installation.md for the event table, the exact reply formats, the parser rules and the list of assumptions a live Grok run should verify, and docs/grok-hooks.example.json for the complete hook file.

OMP (OpenRouter-backed retrieval, deterministic contract hooks)

Register this server in OMP’s user mcp.json and opt in to extensions/tsukkomi-omp.ts as an OMP user extension. Configure both processes with the same private TOML file (or environment overrides) for the index, OMP session root and authorized credential/vector references. The extension starts Python in --omp-bridge mode, so the shared Python settings source owns validation; TOML-only paths need not be duplicated in the extension environment.

SELF_DIRECT_LOCAL_ONLY=false
SELF_DIRECT_CONFIG_FILE=  # omit to use the default user file
TSUKKOMI_OMP_MONITOR=1
TSUKKOMI_OMP_PYTHON=
TSUKKOMI_OMP_SOURCE=/src  # omit when installed in that Python

Interpreter/source bootstrap and SELF_DIRECT_LOCAL_ONLY remain environment options, not [paths] keys. Remote bridge startup requires a usable explicitly selected key file and an existing native-library path; it does not make an API call or eagerly initialize the index. Environment paths already present in managed registrations override TOML until the owner removes or changes those entries.

OpenRouter-backed hybrid/dense history search, sync_session(embed=True) and NeoGraph updates are available by default; embedding uploads sanitized session chunks, so use an authorized key and session root. The hook’s check_action and audit_session stay deterministic and refresh JSONL without embedding; they do not call OpenRouter on every tool or turn. Explicit offline-only mode is SELF_DIRECT_LOCAL_ONLY=true, which disables those remote functions. NeoGraph extraction also masks known secret patterns in session events before sending a prompt to OpenRouter; keep the original session files and local index private.

Keep machine-specific values in local OMP configuration or a private extension copy, not this repository. Start a new persisted OMP session; --no-session cannot provide history. Register only user-stated, machine-checkable contracts with upsert_contracts (provider: "omp" and the actual OMP session ID for session scope). The extension checks pending model-issued tools through local MCP before execution, blocks confirmed violation/suspicious, and re-audits after results and turns. unknown and transport failure are reported but do not block; zero applicable contracts is not clean. Direct shell or host actions outside OMP’s tool pipeline, skipped hooks, in-memory sessions and a result not yet flushed to disk are not covered.

The extension also registers the read-only tsukkomi_session_context tool. Call it before creating a session-scoped contract or checklist item: its session_id comes from the active OMP session manager. Never infer a session ID from PI_SESSION_FILE or a shell variable, which can belong to a parent OMP process. If no persisted session is available, do not silently create a global contract. AGENTS.md instructions alone are not audit contracts: register the user’s explicit, machine-checkable prohibitions; broader subjective mistakes are not automatically judged by an LLM.

Cursor / Grok Bot

Set SELF_DIRECT_GROKBOT_TRANSCRIPTS_DIR (or grokbot_transcripts_dir in TOML) and pass provider="grokbot" to the MCP tools. There is no lifecycle hook for this host in this repository.

Dense backends

The dense half of search_history sits behind one interface (index/vector.py::VectorIndex). The canonical event store, the sparse (FTS5) index, the contracts and the embedding cache are always local SQLite files; only the vectors can live somewhere else. A backend is chosen with SELF_DIRECT_DENSE_BACKEND and is never replaced by another one: when the selected backend cannot work, the call fails with an error that names the repair, and audit_status says so.

All three score 1 - cosine distance (1 is identical, a zero vector scores 0), apply allowed_ids before the top-k, and break exact ties by id in byte order (NumPy: arbitrary inside a tie). tests/test_dense_backend_parity.py holds pgvector and sqlite-vector to NumPy’s exact scan on the same seeded vectors, filters and write sequences: identical top-k except inside groups of exactly equal scores, scores within 1e-6 (float32 arithmetic; the largest difference seen was 2e-7).

pgvector

pip install 'tsukkomi-mcp[pgvector]'          # psycopg 3; imported at the first query, never by hooks or the default install
# once, by a database administrator, in the target database (this process never installs it):
#   CREATE EXTENSION vector;
(umask 077; printf '%s\n' 'host=db.example.org port=5432 dbname=tsukkomi user=tsukkomi password= sslmode=verify-full' > /absolute/path/pgvector.dsn)
export SELF_DIRECT_DENSE_BACKEND=pgvector
export SELF_DIRECT_PGVECTOR_DSN_FILE=/absolute/path/pgvector.dsn

• The connection string is read only from the file named by SELF_DIRECT_PGVECTOR_DSN_FILE (one line, host=... dbname=... or a postgresql:// URI; on POSIX not readable or writable by group or others) and from the standard libpq environment (PGHOST, PGPASSFILE, PGSERVICE…). It is never accepted as a setting value, never logged, never echoed: errors show host:port/database, the server’s primary message with any password removed (also URI-encoded), and nothing else. audit_status shows host, port, database and schema, never the user or any option.
• TLS. A server that is not this machine (anything but localhost, a loopback address or a socket directory) is reached only with sslmode=verify-full and a root certificate that verifies its name; disable, allow, prefer (libpq’s default), require and verify-ca are refused before a connection is opened, so no credential is sent in the clear. A connection that only names a service hides its host, so it is forced to verify-full. A loopback server (a local container, a socket) needs no TLS.
• What leaves the machine. Chunk ids and embedding vectors (float32, SELF_DIRECT_EMBEDDING_DIM numbers per chunk). Never session text, contracts or the canonical store. Vectors are not anonymous: an embedding can be partly inverted to recover the text it stands for, so treat them like the text. Selecting pgvector is therefore an explicit opt-in to that data flow, and a real remote server needs the owner’s approval before it is first used; try it with synthetic data and a local container (tests/pgvector_support.py starts one). SELF_DIRECT_LOCAL_ONLY=true still forbids any dense index.
• Never on the hook path. Hooks and audits do not use the dense index at all; the command hooks publish the canonical events and leave the vectors to the next sync_session of the server, so a database outage cannot delay or hide an audit. Only sync_session (vector upkeep) and search_history (dense and hybrid modes) talk to the server.
• Failure is an error, never a fallback. An unreachable server, bad credentials, a missing vector extension, a dimension or embedding-policy mismatch each raise an error that names the repair (the DSN file, reachability, CREATE EXTENSION vector;, another SELF_DIRECT_PGVECTOR_SCHEMA). sync_session and dense or hybrid search_history return error: dense_backend_unavailable (the canonical events of a sync were already published and the journal keeps the vector work it owes; the next sync resumes). sparse and regex searches and all audits keep working. audit_status reports dense_backend: "pgvector", dense_count (null when the server cannot answer), dense_error, the degraded flag dense_backend_unavailable, and dense_info: host, port, database, schema, dimension, extension_version, server_version, tls, loopback.
• Time budget. Every statement runs with statement_timeout and lock_timeout equal to the time the request has left, so a slow or blocked statement ends with the request’s own request_deadline; a connection attempt takes at most 10 seconds or the time left. A peer that vanishes is noticed through TCP keepalives (idle, about a minute) and tcp_user_timeout (30 s with data in flight). One connection per index object, opened lazily, validated on connect, replaced once if it drops (every operation is idempotent), closed with the engine. A write call is one transaction that commits only if the request is still live, so a stopped request leaves nothing half-applied.
• The schema (SELF_DIRECT_PGVECTOR_SCHEMA, default tsukkomi), created under an advisory lock on first use by a role that may CREATE in the database, or by an administrator beforehand: chunk_vectors(chunk_id text COLLATE "C" PRIMARY KEY, embedding vector(N) NOT NULL) and tsukkomi_meta(key, value), which records schema_version, dim (the vector dimension) and policy (the prefix of the embedder’s cache key, which names the model, dimension and encoding). A process whose dimension or policy differs gets an explicit error instead of mixed vectors. Use one schema per index directory: the table is keyed by chunk id only. Up to 1700 dimensions the vector column is stored inline (SET STORAGE PLAIN): about 46 % more disk than pgvector’s default out-of-line storage for 2.4 times faster scans (50,000 x 1024: 392 MB and 58 ms against 268 MB and 141 ms); above that the default is kept.
• Migrating is not an in-place conversion: change the setting, and the next sync_session refills the new index from the local embedding cache (dense_restored; no embedding call for text that is already cached). The old vectors-v2-* files stay untouched, and so does the PostgreSQL schema when you go back (reverting the setting is the whole downgrade).
• Versions. Tested with PostgreSQL 17.11, pgvector 0.8.7 and psycopg 3.2.3 and 3.3.6 on Linux. Exact search needs only the vector type and ``; HNSW needs pgvector 0.5.0 or newer. Other PostgreSQL and pgvector versions, a remote server, TLS to a real server and Windows are not verified (Windows needs only the optional extra; the default install does not).

HNSW (optional approximate search)

SELF_DIRECT_PGVECTOR_HNSW=true creates chunk_vectors_embedding_hnsw (cosine, pgvector’s defaults m=16, ef_construction=64; at most 2000 dimensions) and serves searches without an id filter with hnsw.ef_search = max(100, 4 x top_k). It is off by default because the exact scan is the reference and because of what it costs: the index is as large as the table, every write maintains it, and it is approximate. Searches with an id filter are always exact, and search_history always passes one (the session’s ids, or every id of the provider), so with the current engine the index is built and kept but not used: turn it on only when calling the backend directly. A large table’s index takes minutes to build; build it outside a request with PgVectorIndex(...).build_hnsw_index(maintenance_work_mem='2GB', timeout_ms=0) (a build inside a request that runs out of time says so and names this command). The recall trade-off below is measured on clustered synthetic vectors; vectors with no neighbourhood structure (uniform random in 1024 dimensions) are the worst case, with recall@20 of 0.03 at 100,000 vectors and ef_search=40.

pgvector numbers

Synthetic unit float32 vectors of dimension 1024 (uniform random for the comparison, 500 Gaussian clusters for HNSW recall), top_k = 20, 30 searches per cell after 2 warm-ups, p50 / p95 in milliseconds, one client on one machine (16 logical CPUs, WSL2, shared with other work: repeated runs differed by up to 35 %, so read the ratios, not the digits). pgvector: PostgreSQL 17.11 and pgvector 0.8.7 in a loopback container with a tmpfs data directory, shared_buffers=512MB, PostgreSQL’s default of 2 parallel workers per scan, inline vector storage, batches of 2000 rows per call. sqlite-vector 1.1.0 on a local file; NumPy in memory (loaded by one call, because every call re-stacks the whole matrix). Ids are 65 characters long, like the engine’s chunk ids. The filters are 500 stored ids and 50,000 ids (ids that are not stored pad the set when N is below 50,000). scripts/bench_pgvector.py reproduces everything; the 300,000-vector runs needed a larger tmpfs (8 GB) and, for HNSW, --shm 6g --maintenance-work-mem 4GB.

Where pgvector is slower: writes (about 9,000 rows/s, a sixth of sqlite-vector’s: one transaction per batch, and a 4 kB vector fills a whole 8 kB page), present (a round trip: 2.4 to 4.4 ms against 0.5 to 1.7), disk (810 MB against 448 MB at 100,000, because inline storage wastes half of each page; pgvector’s default storage is 32 % smaller but scans 2.4 times slower), the first query of a process (a validation round trip), a 50,000-id filter on a large table when the filter names less than a quarter of it (the ids are sent and looked up: 229 ms at 300,000 against NumPy’s 142), and everything on a small index: below about 10,000 vectors the local backends answer first. Where it wins: the exact scan once the index is large (2.5 times faster than sqlite-vector at 100,000 vectors and at 300,000, 1.7 to 1.8 times faster than NumPy: pgvector’s SIMD distance and parallel scan against a single-threaded one; with max_parallel_workers_per_gather=0 expect less), a session-scoped search (500 ids: 2 to 4 ms at every size, an index lookup per id, where sqlite-vector scans everything and filters afterwards, 20 to 682 ms, and NumPy copies the matrix, 4 to 44 ms), a search over most of the index (a filter naming a quarter of the table or more is applied after one unfiltered scan, so no ids are sent), and concurrent readers and writers.

How a filter is sent matters more than it looks. A 50,000-id filter of 65-character ids is 3.4 MB (split by the server in 44 ms as one joined text parameter, which is what is sent; with 8-character ids a text[] parameter took 42 ms against 12 ms joined, because psycopg dumps an array element by element). A hashed = ANY scan with such ids was 4 times slower than the scan alone (290 against 72 ms at 100,000 rows) and slower than looking the ids up by primary key (146 ms for 50,000 ids), so the planner’s choice is left alone. A filter naming a quarter of the table or more therefore sends nothing: the top rows of the whole table are fetched, the ids that are not allowed are dropped here, and the fetch is widened (4 times as many rows, at most three scans) until enough allowed rows are in hand; the answer is exact, and after three scans that were not enough the ids are sent after all.

HNSW on clustered vectors (build with 2 parallel workers; ef_search is the search width; p50 latency at ef_search=100, the default here, against the exact scan):

A LAN adds round trips, not computation. A search or present costs about one round trip; a write batch pipelines its rows. Measured through a loopback proxy that delays each direction (10,000 vectors, p50): 0.25 ms each way (a 0.5 ms round trip) adds about 0.5 ms to present and nothing measurable to a search; 1 ms each way (2 ms round trip) adds 2.3 ms to present and about 1.5 ms to a search, and lowers bulk upserts from 6,700 to 6,000 rows/s (bandwidth was not limited: 4 kB per vector is 30,000 rows/s per gigabit). A filter sent as ids is 3.4 MB per 50,000 ids, 27 ms on a gigabit link.

These are one machine’s numbers on synthetic data. They do not include concurrent load, a cold cache or a remote server, and loopback understates every network cost.

---

Coverage and verdicts

sync_session, audit_session and check_action report a coverage object (audit_status reports the same data under session when given a session id). It holds the indexed source path, byte_offset (how far the index has parsed), file_size, chunk_count, parser_version, an issues list and complete, which is true only when issues is empty. While a sync is still in progress it adds backlog_bytes and the issue sync_in_progress (Large sessions and deadlines). The same responses carry an integrity object that says how the source was verified for that request and when it was last verified in full (Source integrity).

Audit and check_action results also carry requires_attention, which is true whenever the verdict is not clean.

Format drift: unsupported records

A record the parser does not know (an unsupported record type or step, a parse error) is kept visible and makes coverage.complete false (unsupported_events, parse_errors). It may hold the forbidden call or the commit an order rule waits for, so a history audit does not vouch for the records it could not read:

• A must_not with no match and a before_regex rule with no triggering action are unknown (reasons incomplete evidence cannot prove the absence of a forbidden match / ... that no triggering action occurred), and so is the overall verdict.
• A must obligation that rests on missing evidence is unknown (as before), and a proposed action’s before_regex / requires_success prerequisite cannot be established by incomplete history.
• A violation confirmed from the parsed evidence stays a violation, in the findings and overall; a proposed action that matches a prohibition is a violation whatever the history coverage; a clean that rests on positive evidence (a satisfied prerequisite) is not touched.
• Appended, unread bytes are not drift: they keep the judged_prefix behaviour below.

Growing sessions

Agents append to their own session file while they work, so an audit usually sees a file that is still growing. The audit judges the bytes captured at the start and says so.

The result’s notes then contain Judged as of byte N; M appended byte(s) were not reviewed. Within that prefix:

• A must_not violation seen in the captured range is reported as violation.
• Otherwise the verdict is clean as of that offset. Appended records are judged by the next audit.
• A must obligation that would be a violation only because evidence is absent, or failed, becomes unknown while bytes are unread. An obligation already evidenced inside the prefix stays clean as of the offset.
• Any other gap (parse errors, unsupported events, an outdated parser, a missing source) is not a pure append. The verdict follows the normal incomplete-coverage rules and judged_prefix is absent.

A rewrite of earlier bytes, truncation, file replacement or deletion is never judged as growth. It stays unknown (source_changed, source_unavailable_or_truncated, source_changed_during_evaluation). If metadata, contracts or AGY receipts change while an audit is evaluating, the result is unknown with the earlier result kept as snapshot_verdict.

check_action and current prerequisites

A proposed action runs now, so unread records come before it, not after it. When a contract with a gated prerequisite (must with before_regex) applies to the proposed action and the history is incomplete:

• With requires_success: true, a clean finding becomes unknown and the overall verdict is raised to at least unknown, because a newer failed or pending run may be in the unread part (#8).
• A presence-only prerequisite (requires_success: false) stays clean while the only gap is appended bytes, because an observed earlier run is not undone by later records. For any other coverage gap it becomes unknown.
• A must_not violation already found is not hidden by this rule.

Duplicate session sources

Several files can carry the same session id, for example when sub-agent session files are copied under another parent folder. Codex, OMP and Grok Bot compare the stored prefix hash with the requested file using streaming hashes, without loading either file. AGY is excluded because its hook receipts bind to an exact path, so a changed AGY path is still reindexed.

A rewrite or truncation of the same path is still reparsed or rejected as before. A plain rollout and its zstd-compressed twin (.jsonl and .jsonl.zst) are compared by their decompressed bytes, so every row above applies to that pair too; see Compressed Codex sessions.

Incremental audits

An audit used to re-read and re-evaluate every stored event of the session, so its cost grew with the session even when nothing had been appended. Audits now keep a small monitor state per contract and fold in only the events that arrived after it. The answer is the same: verdicts, findings, evidence, notes, coverage (including judged_through_byte, unread_bytes and every unknown reason) and the NeoGraph stage trace are identical to the full evaluation, which remains the reference. The audit still runs through the native contract_audit graph; only what its load_evidence and evaluate_contracts stages do changes.

• What is stored. meta.sqlite gains derived tables (audit_monitor_state, session_commit_log, session_generations, audit_incremental_health, audit_shadow_state, audit_review_ranges) and, once a requires_success contract or a Claude Code result contract is audited, an index on tool-call ids. Nothing in them is authority: the events and the cursor are. A missing, unreadable or disagreeing row only means the contract is evaluated in full again.
• Per contract kind the state holds: anchors and the activation point (applies_from_event_id); whether a when condition was seen; for must_not the match count, the first 20 matches and (with an exception_regex, which is matched against the join of all matches) the match texts and, for Claude Code, its matching tool calls (up to 64 evidence items and 512 call ids, so calls the host refused can be subtracted; over a cap the rule goes to the full evaluator); for must the last three regex hits and, for requires_success, a per-call tracker (how many calls share the latest hit’s id, how many results followed, whether the first succeeded and whether that success is the required command’s at all, see success_unverifiable); for before_regex the worst verdict, the distinct reasons, the first 20 evidence items and the last trigger position, because each triggering action is judged at its own position from the prefix alone. Provider evidence classes keep Claude Code’s cut results and refused calls and AGY’s truncated fields, still-RUNNING results, legacy results, host-logged proposals and denied proposals.
• A normal append does not invalidate state. A state is reused only if all of this holds, otherwise the contract is rebuilt by a full pass: same contract (revision and content), parser version, evaluator version and needs (which events had their text loaded); same index generation and cursor path; the stored (byte_offset, prefix_hash, event_count) is a row of the session’s commit log, and the log’s newest row is the current cursor; and the number of events after the stored offset is what the cursor’s count implies. A reindex, a rewrite (source_changed), a parser bump, a moved or copied source (alias/path change), an append that overwrites a stored event, or a cursor written by an older server process each start a new generation or fail one of those checks. State and the evaluated cursor are written in one transaction, after the index commit they depend on, and never over a newer state.
• Limits are explicit. Tables are bounded (hit texts for an exception_regex, pending results, blocked proposals, cut results, notice targets). A run that would exceed one reports it, evaluates that contract with the full evaluator and remembers the limit, so the next audit goes straight there. Where state cannot be summarized faithfully the contract stays on full evaluation: a must_not whose exception_regex has more match text than the cap; an AGY denial whose proposal lies beyond the 4,096-event look-back window for a contract that has its own matching proposals; any contract disabled by a shadow mismatch (below). Adding a contract that needs text of events no other contract reads changes the needs and rebuilds every state once. An AGY conversation rebuilds when a hook receipt changes an event that is already indexed (see AGY hook cost); an ordinary receipt is an append.
• Shadow verification is a sample with forced exceptions, not a proof. Some requests also run the full evaluator on the same stored events and compare it with the incremental result: the findings and, when every contract is compared, the verdict, notes, coverage and trace. On a difference the full result is returned, incremental.incremental_mismatch names the contracts, a row with status incremental_mismatch is written to audit_incremental_health, that contract is evaluated in full in that session until the evaluator version changes, and the schedule below starts again at 1/4. A comparison is skipped (incremental.shadow_skipped: "deadline", or "deferred" where a resident engine’s idle comparison will make it) when the request’s remaining time budget could not cover the full evaluation, and one that starts and cannot finish in the room it has leaves the incremental answer standing, unverified (see Large sessions and deadlines). Which requests are compared (incremental.shadow_reasons says why):

  • Forced, whatever the rate. rebuild, premise, warmup: a state was rebuilt in this request, and for the first 4 compared requests after a rebuild or a change of premise (below), the session is compared in full. violation: the incremental answer holds a violation or suspicious finding of a contract it decides (a wrong block is the costly error; for an action check only the contracts that apply to the action count, since only those can block it). flip: a contract that the previous request of the same kind and action (an audit, or the same proposed action) judged violation or suspicious is clean now (events only append, so a real flip is rare, and a wrongly missed violation is the costly error). high_risk: a risk: high contract applies to the proposed action, or this request changed its stored summary. The last three compare only the contracts that call for it (incremental.shadow_scope: a reference evaluation of those contracts alone), so a high-risk contract in the session makes no unrelated request compare anything and does not make the other contracts dearer; a scoped comparison does not count towards the rate below, and a scoped mismatch is settled by a full comparison in the same request. These three are not repeated for an input that was already compared and matched (the same stored events, contracts, request and incremental answer, with every contract asked for among those compared: shadow_skipped: "verified_before"): the reference is deterministic, so a repeat could find nothing. A comparison that did not run (no time) is not remembered as done. A violation, a flip or a changed high-risk summary is only known once the first pass has answered, so the comparison it calls for is a second pass over the same request. That pass gets the time that is left minus a reserve (a quarter of it, at most one second), and no comparison is started unless its estimated cost fits in what is left after that reserve; if the pass cannot finish, because that time is up or the index is busy, the answer of the first pass is returned unverified (shadow_skipped: "deadline" or "busy"), the violation stays remembered for the next request of that kind, and no answer is lost. A request that was cancelled, or is past its own deadline, is still an error.

  • Sampled, for everything else, at an adaptive rate. The rate starts at 1/4 and halves (1/8, 1/16, 1/32) after every 8 consecutive comparisons that actually ran, covered every contract and matched; the floor is 1/32. The 4 comparisons of the warm-up count towards that streak, so the 1/4 stage of a new session is the 4 sampled matches that follow them. The rate returns to 1/4 only when a premise of the verification changes (incremental.shadow_reset names it): the contract set or one contract’s revision, the parser or evaluator version, the session generation (a reindex or a rewrite) or a mismatch. A normal append does not reset it. A comparison that was skipped for lack of time is no match and does not interrupt the streak. The draw is a hash of the schedule state and the cursor, so it is deterministic and reproducible. The schedule is kept per session and provider in audit_shadow_state, written with the monitor states (a resident engine holds it with them), so a restarted process continues it; a row that cannot be read (damaged, written for other constants, or with any invalid field, the remembered comparison included) only means “start again at 1/4 with the warm-up”, which compares more, never less. Parallel requests cannot undo a reset: an older epoch never replaces a newer one, in the table or in the memory a resident engine holds (and a reset another process stored wins over the older schedule a resident engine still holds); two writers of the same epoch keep each other’s remembered violations, each replacing only the entries of the request kinds it handled. The action-check and the audit variants keep monitor states of their own, so the first request of each variant is a rebuild and owes the warm-up again.

  • These constants are not the roundtable’s, on purpose. Issue #18 R2 asked for 1/4 down to 1/64, halving after 32 matches and a warm-up of 8. The review chair changed that to the constants above after measuring the policy functions themselves (scripts/measure_shadow_detection.py --share, --session and --simulate, 300 seeds and 20,000 draws of a synthetic session; --constants 4,8,16,32,64/32/8 replays the original ones). With the original constants a session compared more of its requests than the fixed 1 in 16 until it had run about 2,000 requests, so no p95 got lower in a session of a few hundred requests. Share of the requests compared after 10 / 100 / 300 / 1,000 / 2,000 / 5,000 requests: the fixed 1 in 16 (plus its rebuild at the first request) 15.3% / 7.2% / 6.6% / 6.4% / 6.3% / 6.3%, the original constants 85.1% / 30.5% / 18.8% / 10.0% / 6.5% / 3.5%, the constants as built 54.8% / 16.8% / 8.9% / 4.9% / 4.0% / 3.5%. Comparisons in a session of 300 / 1,000 / 3,000 requests, and the mean number of requests a missed violation stays unnoticed in it (starting at ten times spread over the session): 1 in 16 19 / 63 / 188 and 15 / 16 / 16; original constants 57 / 100 / 146 and 6 / 14 / 38; as built 27 / 49 / 112 and 15 / 26 / 30. A floor of 1/64 (1.6% of the requests) buys no percentile: it is still above the 1% a p99 sees, and it doubles the time a silently missed violation goes unnoticed at the floor (45 / 188 / 627 requests, against 22 / 95 / 269 at 1/32); 1/32 (3.1%) is below the 5% a p95 sees, which is the point of an adaptive floor. The constants are audit.shadow.RATE_DENOMINATORS, HALVE_AFTER and WARMUP_REQUESTS.

  • Settings. SELF_DIRECT_AUDIT_SHADOW_EVERY is adaptive by default. An integer pins the previous fixed behaviour and switches adaptation off, the forced comparisons stay: 1 compares every audit (the test suite does), 0 only the forced comparisons, N about one audit in N (16 was the previous default). audit_status reports the policy, the schedule of a session (rate, streak, forced_comparisons_left, last_match_*, reset_reason), the contracts whose incremental path is switched off and the ranges that need review under incremental_verification.

  • Idle comparison (resident engines only). The daemon’s engines and the stdio MCP server (never a hook process) compare the latest state of a session with a full evaluation once the engine has been idle for SELF_DIRECT_AUDIT_SHADOW_IDLE_SEC seconds (default 2; 0 switches it off), if the session answered requests without a full comparison since its last match (a comparison skipped for lack of time is such a request, "deferred"), or if an audit of it was cut by its deadline before its monitor states existed: the job audits the session with the budget of a background job and keeps the states (see Large sessions and deadlines). One job is pending per session and one runs at a time; it never blocks a request (a request that arrives cancels the running job at its next cooperative check, and the session stays pending). On a mismatch the contract’s incremental path is disabled, the rate returns to 1/4 and the requests answered since the last match are recorded as a range that needs review (audit_review_ranges, reported by audit_status; a mismatch found by a request records one too). The three changes are one transaction, all or none, and not undone by a cancellation that arrives meanwhile; if the index cannot be written the answer is still the full evaluator’s and the report says incremental.mismatch_not_recorded, the session stays pending and the next comparison finds the difference again. The range starts at the persisted last match, so it is recorded after a restart too; the count of requests in doubt (of both kinds) is known only to a resident engine that was running at the last match, and the detail of the range says when it is not. A job that a request cancelled is not a failure and is retried as soon as the engine is idle; a job that ran out of its own budget, found the index busy, failed or could not compare backs off for 30 seconds, doubling to 15 minutes, and sessions are tried in turn, so one that keeps failing cannot starve the others; a session whose contracts all run on the full evaluator has nothing left to verify and is dropped. The worker never reopens an engine that was closed. The job reads the events under the index’s file lock like a request (while it reads them, not while it evaluates), and only requests to its own engine cancel it: a request of another process that arrives meanwhile (a hook process that does not use the daemon) waits up to SELF_DIRECT_LOCK_WAIT_TIMEOUT_SEC (1 second) and is then answered index_busy, which a hook treats as unknown. It does not retroactively guarantee earlier responses: a match says the latest state agrees with the full evaluator, nothing about the answers given before it.

  • What this does not promise. Not every response equals the full evaluation. An answer that is wrongly clean passes between two comparisons unnoticed; a violation, a violation that turns clean and a high-risk contract are checked in the request that carries them, a silent miss is not. Fault injection (scripts/measure_shadow_detection.py: a monitor that ignores forbidden events, in a synthetic Codex session with five contracts; requests counted from the one that carries the violation, 1 = noticed in it) gives, as median / p95 / max requests until the defect was noticed (the wrong answers returned before that are one fewer). Engine runs use 200 seeds; --simulate plays the schedule alone for 20,000 draws, which is what the tails need:

    a violation the monitor misses (answers clean)	previous fixed 1/16	adaptive, engine	adaptive, --simulate
    in the warm-up (request 3)	not applicable	1 / 1 / 1	1 / 1 / 1
    just after it (request 9)	11.5 / 53 / 92	3 / 12 / 36	3 / 11 / 64 (1/16: 11 / 47 / 138)
    request 40	10.5 / 47 / 123	5 / 23 / 47	6 / 23 / 73
    at the floor (1/32)	not applicable	19.5 / 77 / 155	22 / 95 / 269

    other defects, noticed in the request that carries them	previous fixed 1/16	adaptive, cold start (20 seeds)	adaptive, at the floor (20 seeds)
    a block the full evaluator would not give (violation for clean)	12 / 46 / 92	1 / 1 / 1	1 / 1 / 1
    a violation that turns clean	10 / 41 / 88	1 / 1 / 1	1 / 1 / 1
    a high-risk obligation that wrongly says clean	12 / 52 / 113	1 / 1 / 1	1 / 1 / 1

    (The previous-1/16 column is the earlier run of the same harness against the commit this work started from, 200 seeds; the fixed schedule is the same code in the merged tree. tests/test_shadow_detection.py repeats the forced rows at the floor and with SELF_DIRECT_AUDIT_SHADOW_EVERY=0.) The adaptive schedule trades a longer worst case for a lower steady cost: after a long clean streak a silently missed violation is noticed after 22 / 95 / 269 requests (median / p95 / max) where 1 in 16 needs 11 / 47 / 138, and every request in between returns the wrong clean. It still compares more than 1 in 16 did in a young session (54.8% of the first 10 requests, 16.8% of the first 100, 8.9% of the first 300) and falls below the 6.25% of 1/16 after about 600 requests (4.9% over 1,000, 4.0% over 2,000, 3.5% over 5,000; 3.1% at the floor), where the original constants needed about 2,000.

• Small sessions. A session with fewer than SELF_DIRECT_AUDIT_INCREMENTAL_MIN_EVENTS stored events (default 500) is evaluated in full and no state is kept (incremental.mode: "full", below_min_events). Below that size one full evaluation costs less than storing states, rebuilding them and shadow-checking the rebuild; on the scripted evaluation harness (short sessions) keeping state roughly doubled the Codex Stop latency. 0 keeps state for every session. A resident engine that holds its state writes no commit for it, which lowers the break-even to about 250 events; the default is unchanged (see Warm judgement benchmark).
• Reporting. Results carry an incremental object: mode (delta reads only appended events, full reads all), events_read, rebuilt (contract to reason), fallback (contracts on full evaluation and why), shadow, shadow_reasons, shadow_scope, shadow_rate, shadow_reset, shadow_compared, incremental_mismatch, retry and state (saved; unchanged; deferred when a resident engine holds the write, see Held writes; or why it was not written). SELF_DIRECT_AUDIT_INCREMENTAL=false switches the whole feature off.
• With source verification. A state describes the stored events, not how the source was verified: it is written before the fast-verification gate and is the same whether the request verified fast or full. If a later full verification finds a rewrite, the reindex starts a new generation and deletes the states. A cursor that migrates from a plain digest to block digests, or that an older process rewrote, no longer matches the commit log, so states keyed to the old digest are rebuilt and never reused.
• With contract time limits. Contract patterns run on the time-limited engine. A monitor whose pattern times out takes no more events and is dropped for that run; its contract is judged (unknown, never clean) by the full evaluator, its stored state is deleted and nothing is marked unhealthy. When a shadow run’s full evaluation itself times out for a contract, the full (unknown) answer is returned, the contract’s state is dropped, and it is reported as incremental.shadow_skipped: "reference_regex_timeout", not as a mismatch.
• What it does not do. It does not make an unknown clean and does not change coverage: unread bytes, parse errors, unsupported events and rewrites are judged by the unchanged ingestion and coverage code, and a state describes the index, not the source file. The source verification (hashing) that precedes every audit is a separate cost; see scripts/bench_audit.py for a before/after measurement on a synthetic session of any size.

Source capture

Sync captures at most the length measured at the first fstat, in 1 MiB blocks. It keeps only small hash state: the SHA-256 of every complete 1 MiB block (the block digests), and for a source whose cursor is a plain digest (.jsonl.zst, AGY, or an index written before this version) SHA-256 states at the cursor, at the last complete record and at the end. No file bytes are retained, so memory does not scale with the session size.

• One full scan of the prefix per request. The capture pass is the scan. Every later stage of the same request (the two ingestion stages, coverage, the audit’s own stage check) re-reads a bounded amount instead of hashing the prefix again: the file identity, size >= captured length and the SHA-256 of the last 64 KiB as it was streamed during the pass. Before this version one check_action hashed the whole prefix five times (measured: 5.0 passes on a 300 MB session; now 1.0). validate_prefix() is that bounded re-check; validate_prefix(locked=True) is the same without any prefix re-read, for use under the shared publication lock; validate_prefix(full=True) re-reads the whole prefix and is used only when asked for or when the cheap signals demand it (below). A mismatch, once detected, stays failed.
• Cheap signals escalate, they never excuse. Without growth, an unchanged size with a moved modification time means the file was written (or touched). The unlocked check then settles it with a full re-read: identical bytes pass and the new time becomes the baseline; different bytes fail. Equal stamps prove nothing; they are never a reason to skip a check that would otherwise be made.
• A change during preparation is retried, then reported as before. If a later stage finds that the file was rewritten, truncated or replaced between capture and publication, the observation is discarded and the file is captured again, in full, at most three times in all. If it still does not hold still, the call fails with source changed during preparation and nothing is published, as before.
• Growth after the first measurement is not an error. An overwrite, truncation or replacement during the pass is. If the file grew while it was being hashed, the bounded check compares the last 64 KiB with the bytes that were streamed; this no longer costs a second scan.
• validate() compares metadata only (device, inode, size, mtime, ctime). Equal stamps do not prove equal content.
• stat and fstat can give st_ctime different meaning on some runtimes (for example 9p/drvfs), so full stamps are compared only within the same API and identity, size and mtime across them. On Windows st_ctime is the creation time and is never used as evidence of change. When either side reports st_ino 0, which some Windows filesystems do, the inode comparison is skipped and size, mtime and the hash decide.
• A .jsonl.zst source is an immutable snapshot: its capture is one streaming decode, and its later checks re-read and hash the compressed file (which is small), so any change to it is a rewrite.
• Limits, stated precisely. Capture and validation are not an atomic snapshot. A rewrite that leaves the identity, the size, the last 64 KiB and the modification time unchanged and lies outside the last 64 KiB, made after the capture pass of the same request, is accepted by that request’s later stages. So is a rewrite outside the last 64 KiB made while the file also grew (the modification time then moves legitimately). The next full scan (the next sync’s capture, or a full verification) compares the stored cursor hash and reindexes. This is the same residual window the lock-time check always had; it now starts at the end of the capture pass for every later stage.

Source integrity: full and fast verification

Re-hashing a 300 MB session on every hook made the cost of a tool-call check proportional to the size of the session file. A cursor of a plain file therefore keeps, next to the byte offset, the block digests of the indexed prefix: the SHA-256 of every complete 1 MiB block, the digest of the partial last block, the newline count and a root over all of it (blocks1: is the cursor’s prefix_hash). They are built while the file is read anyway and extended, never recomputed, as the cursor advances: a sync hashes only the new complete blocks and the partial tail. The table lives in the index metadata (source_blocks, created by a schema migration on first open; existing data is untouched) and is bound to exactly one cursor publication. A table that is missing, stale (a cursor written by an older process), damaged (its root does not add up) or for another path is never trusted: the next request is a full verification that rebuilds it.

Every sync_session, check_action and audit_session response, and the context a Codex hook emits, carries an integrity object:

"integrity": {
  "mode": "fast",
  "full_verified_through_byte": 314572800,
  "full_verified_at": "2026-10-15T12:00:00.123456+00:00",
  "source_generation": "g1:830-269c96:314572800:5b0e8e1b98536aaf",
  "index_generation": "i1:93d3c13bb79981cf",
  "fast_checked": {"tail": true, "recent": [309, 310, 311, 312], "sampled": [3, 8, 41, 77, 90, 134, 201, 250]}
}

Which operations verify how.

What a fast check does. Fast verification compares, against the stored digests: the file identity and size (it may only grow); the partial tail block; the latest SELF_DIRECT_INTEGRITY_FAST_RECENT_BLOCKS blocks (default 4); and SELF_DIRECT_INTEGRITY_FAST_SAMPLE_BLOCKS (default 8) random other blocks, drawn from fresh system randomness for every call. It then parses only the bytes appended after the cursor, each of them hashed block by block while consumed. No other block is read. Cost is a few MiB plus the appended bytes, whatever the size of the file.

Measured with scripts/bench_integrity.py on a synthetic 314.6 MB Codex session of 75,000 events (WSL2, ext4, Python 3.12; median of three runs; “passes” is the bytes through SHA-256 divided by the file size):

The cost of the gate row is the evaluation of the indexed history (it grows with the number of events, not with the bytes of the file); making that incremental is separate work.

Cheap signals only ever escalate to full: a different file identity (a new inode), a smaller file, an unchanged size with a moved modification time, a block or tail digest that does not match. A mismatch is source_changed and a reindex, exactly as a mismatch of the whole-prefix hash always was; it is a content finding, never a runtime failure to fail open on. reindex_reason: "source_changed" is reported as before, and a stale index is never allowed to certify the new bytes: extending the table copies the digests of blocks that were not compared, so a rewrite the fast check missed is still found by the next full verification.

What a fast verdict may rely on. Contract evaluation is the same code and the same indexed history as in a full check; nothing is cached across calls. A fast verdict adds three rules: a clean finding that rests on a satisfied prerequisite is accepted only when every event it cites lies inside the region that was last verified in full (it then carries verified_through); otherwise the request is settled in full before it answers (full_reason: "evidence_beyond_verified_region"); and a fast verdict never turns unknown or a coverage gap (unread_bytes, parse errors, unsupported events) into a satisfied obligation. A fast answer is labelled fast and is never presented as full.

Residual risks, stated precisely.

• A rewrite of a block that no fast check compared (it is not in the tail, not among the latest blocks and was not drawn) is not noticed by fast checks until a full verification reads it: at the latest SELF_DIRECT_INTEGRITY_FULL_VERIFY_INTERVAL_SEC after the previous one, at the next audit_session, Stop checkpoint or background scan, whichever comes first. Until then a fast verdict is computed from the index as it was verified, and says fast.
• A source that is rewritten and restored between two verifications (every block compared is, again, what it was) is not detected, and no claim to the contrary is made.
• The file identity, size and modification time that the table remembers are advisory hints. They are not compared across machines as a guarantee (a Windows and a WSL process can see different st_dev/st_ino for one file); a difference only costs a full verification. File-system notifications are not used as evidence.
• The residual window of a request’s later stages is the one described under Source capture.

Compatibility. The cursor’s prefix_hash is a plain SHA-256 until a source gets a table: the first full request after this version migrates it in place, without a reindex (full_reason: "migration"). A process of an older version that meets a block cursor does not understand it, treats it as a changed source and reindexes once; restart every process that shares an index after upgrading (see Upgrading to 0.6.0). A plain rollout and its .jsonl.zst twin still share one cursor: the cursor’s hash form follows the source form.

SQLite journal mode, index files and backups

The index directory (SELF_DIRECT_INDEX_DIR) holds these SQLite files and one JSON file:

SELF_DIRECT_SQLITE_JOURNAL is delete (the default: the rollback journal, as before this setting existed) or auto (write-ahead logging, “WAL”, for meta.sqlite and agy-receipts.sqlite on a file system that is positively identified as local, the rollback journal everywhere else). Both policies use synchronous=FULL, set explicitly on every connection. synchronous=NORMAL is never used for events, cursors or receipts: in WAL mode it can lose the last acknowledged commits at power loss, and scripts/bench_commit.py measures it for reference only.

Why auto is not the default. WAL makes the commit itself cheaper (one fsync of an appended log instead of the journal file, the database and the journal’s deletion), but a command hook is a new process that opens the database, commits once and closes it, and SQLite folds the log into the database when the last connection closes: the saving moves to the end of the process. Measured on ext4 (the tables below; 300 interleaved samples per cell), a resident engine (the experimental daemon, the stdio server) is clearly faster, and a fresh process is not: the append transaction (commit_events) has a p50 of 21.0 ms with the rollback journal and 6.8 ms with WAL in a resident engine (-68 %; the whole append operation -56 %), but 25.3 ms against 20.4 ms in a fresh process (-20 %) and the whole operation, which is what a command hook experiences, goes from 29.2 ms to 32.2 ms (+10 %). The roundtable’s rule asked for at least 25 % lower p50 on the append commit on ext4 on the real paths; the standalone command hook, the default install, does not meet it, so the default stays delete and the chooser, the reporting and the WAL-aware copy helpers ship switched off. auto is for resident processes: set it together with SELF_DIRECT_DAEMON=on, or for a long-running stdio server. Skipping the fold at process exit was tried and is worse (the log grows to the autocheckpoint size and every next process has to recover it; see the benchmark below).

What auto does.

• Which file systems. journal.choose_journal(path) returns JournalChoice(mode, reason). On Linux it reads the mount that holds the (symlink-resolved) directory from /proc/self/mountinfo after an os.statvfs check. Allowed: ext2/3/4, xfs, btrfs, tmpfs, f2fs, zfs, and an overlay only when the mount holding its upperdir can be shown (same device, one of those types). Denied: 9p (WSL drive mounts, also reported as drvfs), nfs*, cifs, smb*, fuse* and virtiofs. Anything else (ntfs3, vfat, an unrecognised type, an unreadable mount table, a read-only file system, another operating system) is unknown. On Windows the volume must be a fixed drive (GetDriveTypeW is DRIVE_FIXED); UNC paths, \\?\UNC\, device paths and mapped network drives are refused. The Windows branch is verified by tests that fake the Win32 calls and by the Windows CI job; it could not be measured. Unknown means DELETE. That is the old behaviour, not a statement that DELETE is safe on that file system.
• Reporting. audit_status returns journal (policy, synchronous: "FULL", and per database the actual mode, the wanted mode, the reason, a pending reason when they differ, and fallback) and journal_fallback, a list of the databases that use the rollback journal because the file system is denied:* or unknown:* under auto (empty under delete, where the reason is setting:delete). The same fact is logged once per directory and reason as journal_fallback ... at INFO.
• When the mode changes. Only when a connection is opened (apply_journal), never inside a transaction, and never by waiting longer than 50 ms for another connection. A refused switch (another connection, possibly an older process, holds the file: SQLITE_BUSY) keeps the existing mode, is recorded in pending and is tried again at the next open (every receipts connection; the next process for the index; for a resident engine, once it has been idle for 30 s or more, from its idle flush). It never fails a request. Switching back with delete (or on a file system that is no longer allowed) works the same way, in the other direction. The mode is stored in the database file, so every later connection finds it.
• Older processes. A release that knows nothing about this setting opens a WAL database with plain sqlite3, reads and writes it, and sees every committed row (tests/test_journal.py does exactly that with a second plain connection). Two processes may even use different policies on one index; the file’s mode is whatever was set last.
• Checkpoints. SQLite’s automatic checkpoint (1000 pages) is unchanged. A clean engine.close() folds the log of both databases into the files (PRAGMA wal_checkpoint(TRUNCATE) with a 250 ms bound for readers), and so does a resident daemon when its idle exit closes its engines. A process that dies without closing leaves a log that the next connection recovers; the kill test below shows nothing acknowledged is lost.

Copying, moving, backing up and deleting index files. With auto a committed row can live only in meta.sqlite-wal. Copying meta.sqlite by itself then loses the most recent commits, and a stale -wal next to a replaced database file would be replayed into it. Therefore:

• Copy a live index with SQLite’s online backup API, never with cp: journal.copy_database(src, dst) / journal.copy_index_dir(src, dst) (every *.sqlite / *.db through sqlite3.Connection.backup, sidecars skipped, other files copied as is), or the sqlite3 shell’s .backup. scripts/bench_audit.py, scripts/bench_agy_hooks.py, scripts/benchmark_large.py and the AGY equivalence test use them. A directory can also be copied after every process using it has closed cleanly (its logs are then empty), or after PRAGMA wal_checkpoint(TRUNCATE).
• Delete or replace a database together with its -wal, -shm and -journal files (journal.remove_database(path)). The installers (scripts/install_codex.py, scripts/install_claude_hooks.py) back up configuration files only and never touch the index.

Receipt and event pairing (AGY). The receipts live in their own database file, so an event commit and a receipt commit are two commits. They are tied by content, not by a request id: with every cursor the index stores the SHA-256 of the receipt rows the events were derived from (session_receipt_versions.digest) and the receipts those events depend on (session_receipt_deps), and every status, sync and judgement compares that digest with the digest of the receipts database as it stands. A receipt committed without the events commit that follows it, or an events commit whose receipt commit is missing, differ in that digest and report receipts_not_indexed (the coverage is incomplete, so a verdict that depends on it is unknown) until a sync brings the two together. Receipts are written before the events that cite them, each with synchronous=FULL, so the order survives a crash in either journal mode. A receipt for a step whose result is not indexed yet is still a plain append. Nothing about what a receipt may prove changed; tests/test_journal_engine.py pins both directions.

---

Sparse index formats

The sparse search index (SQLite FTS5, BM25) is derived data: the canonical store (meta.sqlite) can always rebuild it. Until now it was the largest file of an index (a Codex index of 365,768 events: 4.6 GB of 7.0 GB), because its FTS5 table kept a copy of every masked text next to its inverted index and that text is never read back (a search returns ids and scores). There are now two formats, chosen by SELF_DIRECT_SPARSE_FORMAT (auto, legacy, compact; default auto):

Measured (scripts/bench_sparse.py: a seeded synthetic coding-agent corpus of chat turns, shell commands, multi-KB logs, diffs, JSON, stack traces, paths, hashes, ids and some Korean text; the same events written through the real write path, 500 per transaction; SQLite 3.53.1; 20,000 events on SQLite 3.45.1 gave the same sizes and the same parity result):

(mask_secrets alone takes 58 s of the Codex-like and 14 s of the OMP-like build in either format.)

Size and build time are linear in the number of events (20,000 to 100,000 checked), so apply the percentage to your own index: the 4.6 GB sparse file of the 365,768-event Codex index above would be expected near 1.4 GB, the 556 MB file of the 300,474-event OMP index near 170 MB (an estimate; your texts decide). Where the bytes were: in the legacy Codex-like file 592.8 MB was the stored text (and 27 MB of it page waste), 303.5 MB the inverted index; the compact file is the 246.7 MB inverted index plus 17 MB of id map. A bigger SQLite page size (8, 16 or 32 KiB) does not help (+0 to +4 %): the stored text, not overflow pages, was the cost.

The results are identical, not similar. For the same chunks written in the same order, both formats return the same ranked ids with the same BM25 scores (to the last bit in the benchmark: 200 queries per run, over single and multi-token, hyphenated and snake_case, Korean, no-match, frequent and rare terms, session_id, allowed_ids and top_k 5/20/100, before and after deleting and re-inserting 1,000 events and after optimize: 0 divergent queries on both 100,000-event corpora with SQLite 3.53.1 and on both 20,000-event corpora with SQLite 3.45.1), ties included (equal scores rank in write order, as the legacy query returns them). Two details make that true and are tested (tests/test_sparse_compact.py, with mutation checks in tests/test_sparse_compact_mutations.py):

• FTS5 does not take a deleted row out of its BM25 statistics (document count, total tokens) in a contentless table, so after a delete the scores drift from an index that never held the row (158 to 162 of 200 queries diverged in the benchmark). The compact format corrects the statistics record in the transaction of the delete, from the stored document sizes, and proves the mechanism on the running SQLite before it opens a file (compact_support).
• Tokenization is unchanged. Two cheaper variants were measured and rejected because they change results: detail=column drops term positions, which turns every hyphenated or snake_case query (a quoted token that tokenizes into several terms is a phrase) into an error and makes every term frequency 1 (170 to 177 of 200 queries differ), and tokenize='unicode61' without the porter stemmer is no smaller (the dictionary dominates) and changes stemming (68 of 200 differ).

Speed. A search with a session filter is 7 to 30 times faster than before, because it no longer reads the large stored rows; the others are as fast or a little faster. p50 / p95 in milliseconds at 100,000 Codex-like events: a global frequent term 13.2 / 25.2 legacy, 12.7 / 24.6 compact; session_id 181.8 / 232.7 and 5.3 / 8.0; with allowed_ids (what search_history always passes) 197.9 / 397.6 and 14.6 / 34.8. Deleting 1,000 events and inserting them again takes 0.5 s + 1.5 s legacy and 0.8 s + 1.3 s compact; PRAGMA integrity_check takes 2.8 s instead of 6.2 s. VACUUM and the FTS5 optimize command do not pay: together they shrank the compact file by 4 % in about 8 seconds, with the file locked.

What the file holds. No text, masked or raw: the masked text is tokenized and only the lower-case, stemmed terms and their positions are kept. That is strictly less than the legacy file held (see the masking contract).

Choosing a format. auto (default) uses the compact format when this process’s SQLite supports it and otherwise the legacy file, and says so: audit_status shows sparse_format: "legacy", sparse_format_reason (the SQLite version and the repair) and the degraded flag sparse_legacy_fallback. compact never falls back: an unsupported SQLite is an error that names the repair (a Python build with SQLite 3.43 or newer, or SELF_DIRECT_SPARSE_FORMAT=legacy), and no file is created. legacy is exactly the previous behaviour, in the same sparse.sqlite. Every process that shares an index should use the same format; a process on an older version keeps using sparse.sqlite.

Non-ASCII queries. Sparse search used to keep only [A-Za-z0-9_-] from a query, so a Korean query found nothing even where mode="regex" found the text. The query builder (_fts_query) now handles other scripts; the index, its tokenizer and both file formats are unchanged (nothing to rebuild; legacy and compact answer alike). The index stores a Korean word with its particle as one token (색인을, not 색인), so a word of Hangul, CJK ideographs or kana becomes a prefix term ("색인"*) and a one-character word stays exact. A word of Hangul syllables that ends in 에서 으로 에게 까지 부터 을 를 은 는 의 and keeps at least two syllables also searches its stem: 색인을 is "색인을"* OR "색인"*. 이 가 도 로 와 과 만 에 are not cut, because they end real nouns (도로, 어린이, 비만). The query is normalized to NFC first, at most 8 prefix terms are used per query (the stems counted, the words first, in query order; the rest are exact), and the same word typed twice is one term. Other non-ASCII words (café, Cyrillic, Greek) are exact terms that unicode61 folds like the stored text, and a query of ASCII only is unchanged. Limits: a prefix term also matches longer words (색인 finds 색인표) and BM25 does not rank the exact word above them; Chinese and Japanese, and Korean written without spaces inside a code-mixed run (v2색인), are one stored token per unspaced run, so a query matches from the start of the run only and an inner word needs mode="regex"; a particle that is not in the list is not cut; a one-syllable word is exact, so 집 does not find 집에서; and a prefix on a very frequent stem costs several times an exact match and grows with the number of events that contain its forms (synthetic corpus of 100,000 events, scripts/bench_sparse.py --stages korean: 34 to 43 ms exact against 114 to 125 ms prefix for a stem in nearly every event, about 0.1 against 0.3 ms for a rare one; see the changelog).

Converting an index. Nothing is converted in place. The compact index is a new file that fills the way a missing derived index always has:

• A sync_session writes the sessions it syncs, and a sparse or hybrid search_history writes what its scope lacks before it searches (in batches of 500 documents, each its own transaction, so a search that runs out of time keeps what it wrote and the next call continues; it never answers from a partial index and never from the old file). The first search over a large index would have to build all of it, so convert it once, outside a request:

  python -m self_directing_mcp.maintenance sparse-rebuild [--budget-sec 600] [--index-dir DIR]
  

  The command writes the missing documents from the canonical store (the old file is neither read nor written), then deletes the documents the compact file holds and the store does not (orphans, below), shows progress on stderr, stops at the time budget, is safe to interrupt (every batch is committed on its own) and continues where it stopped when run again; on a complete index it changes nothing. Exit status 0 complete, 3 partial (run it again), 2 nothing to do or failed, 130 interrupted. It writes about 1,100 Codex-like events a second (10,000 events of 5.6 KB: 9 s in two runs, the first stopped by --budget-sec 5), so a 365,768-event index takes about 6 minutes, most of it mask_secrets.

• audit_status shows both files and both counts: sparse_format, sparse_files (name, format, active, exists, bytes) and sparse_counts (legacy and compact, the file that is not in use is read through a read-only connection and never modified), next to the existing sparse_count (the active file) and total_chunks. The conversion is complete when sparse_counts.compact equals total_chunks. sparse_extra_ids is max(0, sparse_counts.compact - total_chunks), the number of orphans (below), computed from those counts without a scan; it is a lower bound while the compact file still lacks documents (they offset the orphans) and null when there is no compact file.

• Orphans. A compact index can hold documents whose event the canonical store no longer has (an event leaves the store and the delete of its sparse document does not happen: a re-index that was interrupted, a process of an older version; the legacy file of a long-used index held more than a fifth of its documents this way). They never reach a result, since a search passes the canonical ids of its scope as allowed_ids, but they cost disk, they make every filtered search scan more, and they take part in the bm25 statistics (the document count and the term frequencies), so a scope’s scores differ slightly from those of an index that never held them. Nothing used to remove them. Now:

  • sparse-rebuild (rebuild_sparse()) deletes them after it has written the missing documents (a missing document makes a search incomplete, an orphan only costs space, so a budget that runs out leaves the more useful index), in batches of 500 that are each their own transaction, under the same --budget-sec, interrupt and resume rules: run it again to finish, and a finished index is not written at all. Its JSON line reports orphans (found), orphans_removed, orphans_spared and orphans_remaining (not yet examined: nonzero when it stopped early); progress for them goes to stderr too. The bm25 statistics are corrected in the transaction of every delete, so statistics.ok holds afterwards and scores equal those of an index that never held the orphans (tested on a seeded corpus, multi-token queries included).
  • A document the store holds is never deleted. Both id sets come from one snapshot taken under the publication guard, and every batch asks the store again, under the guard, right before it deletes: an event that a sync published (or re-published) after the snapshot is kept and counted in orphans_spared. A sync publishes its events under that same guard, so there is no window between the second look and the delete. The ids are compared with the whole store, never with one provider’s: a document of another provider is no orphan.
  • A sparse or hybrid search_history over all sessions (it already scans the index’s ids to find what is missing) deletes the orphans too, after it has written what its scope lacks and before it searches. It is housekeeping, not part of the answer: it stops after 2 seconds or a quarter of the time the request has left, whichever is less, and a busy index ends it quietly, so the search still answers (every finished batch is kept; the next global search or the rebuild continues), while a cancellation or the request’s own deadline ends the request as it does for the missing documents. When the counts show nothing extra (sparse_extra_ids would be 0) it costs a cached count, so the usual search is not slower. A search of one session leaves orphans alone: the compact file has no index on a document’s session, so listing one session’s ids would scan the whole id map on every such search, and most of those searches are for a small session.
  • Deleting leaves free pages in the file, which later writes reuse; the file does not shrink by itself (VACUUM, with every process stopped, returns the space). The legacy sparse.sqlite is never read or written by any of it.
  • Measured (synthetic corpora, SQLite 3.53.1, 500 documents per transaction in id order; scripts/bench_sparse.py --stages orphans and a 360,125-event engine): 0.74 s per 10,000 orphans when a random tenth of a 100,000-document OMP-like file goes (the longest transaction 0.15 s) and 0.94 s per 10,000 for a tenth of a 30,000-document Codex-like file; 1.2 to 1.6 s per 10,000 when a fifth of a 50,000 to 200,000-document file goes (44,000 of 200,000 in 6.3 s, the longest transaction 0.6 s), so roughly 12 s for 85,000 orphans. The search repair without orphans took 660 ms before this change and 646 ms after (medians of 25 runs over 360,125 events; the added work is one cached count, 9 microseconds, or 11 ms once after the store changed), and the repair of a 100-event session 0.27 ms and 0.26 ms (the same code). Listing one session’s ids through session_no costs a scan of the id map, 22 ms at 360,125 documents, which is why a session search does not delete.

• What to delete, and when. The program never deletes, truncates or rewrites sparse.sqlite; it keeps taking its space until you remove it. When the compact index is complete and no process uses the legacy format any more (restart every process that shares the index first), stop them, delete /sparse.sqlite (and a leftover sparse.sqlite-journal) and start them again. audit_status.sparse_files lists the sizes. To go back, set SELF_DIRECT_SPARSE_FORMAT=legacy: the legacy file is used as it was (it lacks the sessions synced while the compact format was active, which the next sync or search adds).

Limits: a very large first search or sync over an unconverted index is bounded by the request deadline (run the rebuild command instead); deletes leave free pages inside the file (from 1 % of its pages after the build to 2 % after 1,000 deleted documents and 11 % after deleting the largest session, 13,471 of 100,000 Codex-like events), which later writes reuse and VACUUM returns, and the file never shrinks by itself; the compact format deletes the highest rowid first because FTS5 grew the file by 17 % when the same 1,000 deletes arrived in ascending rowid order; the compact format cannot be opened by a SQLite older than 3.43.

Resident daemon (experimental, off by default)

In production a coding agent’s integrations add up to many processes: Claude Code and AGY start a fresh Python process for every hook event, each OMP session starts its own stdio MCP server, and AGY starts one per session (16 processes and about 1.3 GB in the case that prompted this, #15). The resident daemon replaces them with one process per user and OS environment; hooks and MCP servers become thin clients of it. It is a stage-1 skeleton behind a flag: Linux (including WSL) only, off by default, and judged by the same code as the standalone processes.

Claude Code / AGY hook event ─ scripts/*_hook.py (stdlib client) ┐
OMP / AGY MCP server ───────── self-directing-mcp (stdlib relay) ┼─ Unix socket ─ python -m self_directing_mcp.daemon
                                                                 ┘                 ├─ hook lane: claude_hooks.run_hook / agy_hooks.run_hook
                                                                                   ├─ heavy lane: the FastMCP tool functions of server.py
                                                                                   └─ one SelfDirectEngine (+ integrity and shadow workers) per resolved-settings key

Turning it on, and off

Set SELF_DIRECT_DAEMON=on in the environment of the hook command (the installers’ env) and of the MCP server entry. The entry points read it from the process environment before importing anything else (a .env or the TOML cannot switch it; config.Settings and the light hook reader validate the same two values, off and on, identically, and an invalid value is a settings error like any other). off, the default, is the explicit standalone mode: nothing about hooks or servers changes. On a platform other than Linux on behaves as off and says so once on stderr (a marker file in ~/.cache/tsukkomi/notices/ keeps later processes quiet).

self-directing-mcp and tsukkomi-mcp now point at self_directing_mcp.entry:main, which decides before importing the MCP SDK. Installations whose console script still points at server:main get the relay too (server.main delegates), at the cost of the imports. python -m self_directing_mcp.daemon --status prints the running daemon’s counters (no request content), --stop asks it to drain and exit.

Threat model

The authentication boundary is the operating-system user, and nothing finer.

• The runtime directory is $XDG_RUNTIME_DIR/tsukkomi/ when that variable names a usable directory (yours, not group- or world-writable), else ~/.cache/tsukkomi/run/. It is created mode 0700 and verified on every use: a real directory (never a symbolic link), owned by the current user, mode exactly 0700; otherwise the daemon and the clients refuse it with a clear error and nothing is created or reused. The verified directory is held open as a file descriptor, files are opened relative to it and a long path is reached through /proc/self/fd//, so the check and the use cannot be separated by a swapped path component.
• The socket (mode 0600, no TCP of any kind) lives in that directory, and every accepted connection’s SO_PEERCRED user id must equal the daemon’s; another uid is closed without a word.
• Processes of the same user are not isolated from each other: any of them can connect, run hooks and MCP sessions as that user, and drain the daemon. That is no weaker than their access to the index and the contracts file today, and no stronger.
• No credential travels. A client forwards a whitelist of its environment (SELF_DIRECT_*, TSUKKOMI_*, HOME, CODEX_SESSIONS_DIR, CLAUDE_CONFIG_DIR), never OPENROUTER_API_KEY or any name that looks like a key, token, secret or password (the key-file path setting is fine); the daemon re-checks what it receives. The daemon reads a key file itself, exactly as a standalone process does, so use SELF_DIRECT_OPENROUTER_API_KEY_FILE: an OPENROUTER_API_KEY in a client’s environment is not forwarded and is ignored by the daemon, which starts with a scrubbed environment and never uses an ambient key. For an MCP session the daemon also reads .env in the client’s working directory, as the standalone server would.
• daemon.log (mode 0600, rotated at 256 KiB) holds event names, sizes, durations and error classes. Request and response bodies, payloads, paths and exception messages are never logged, and a log field that is not a short identifier is dropped.
• A frame is a 4-byte length plus UTF-8 JSON, at most 16 MiB; a zero or oversized length, a body that is not a JSON object, and a connection that stays silent after connecting are protocol errors that close that connection only.

Lifecycle

• Start. The first client that finds no daemon takes a non-blocking lock, starts one detached (own session, stdio on /dev/null) and waits for its socket for a bounded time (AVAILABILITY_FAST_SEC 0.3 s for PreToolUse, PostToolUse, UserPromptSubmit and PreInvocation; AVAILABILITY_SLOW_SEC 3 s for Stop, SubagentStop and SessionStart; an MCP relay waits up to 15 s). The interpreter is the checkout’s .venv Python, else the client’s own; SELF_DIRECT_DAEMON_PYTHON overrides it. The daemon’s environment carries no settings: every request brings its own.
• One instance. A lock file (flock) is held for the daemon’s lifetime; a second daemon exits at once, and a successor started while its predecessor drains waits up to 10 s for the lock. Never two writers.
• Leaving. After SELF_DIRECT_DAEMON_IDLE_EXIT_SEC (900) with no connection and no request; after a request when its resident set exceeds SELF_DIRECT_DAEMON_MAX_RSS_MB (400): it stops accepting, finishes what is running and exits, and the next client starts a new one; on SIGTERM/SIGINT; and when a newer client asks (below). Each exit flushes its counters, closes its engines and removes the socket and the pid file.
• Judging. Hook requests run on their own thread pool and MCP tool calls on another, so a long sync_session never queues a PreToolUse. Requests are bounded per session (SELF_DIRECT_DAEMON_SESSION_BOUND, 16) and in total (128); beyond the bound the answer is unknown(queue_full). Each request’s environment is a per-request view of os.environ (managed names come from the request alone), settings are resolved with settings_core.load_hook_settings (hooks) or Settings (MCP; --omp-bridge validation through the same server.omp_bridge_settings), and engines are created lazily, one per resolved-settings key and reused, each with one resident integrity worker and one idle shadow-comparison worker (see Incremental audits). An engine serializes its own operations exactly as in a standalone process.
• Held writes. A hook that appends events and judges them used to make up to five synchronous SQLite commits, an fsync each (10 to 25 ms on the benchmark machine; the CPU work of the whole judgement is 4 to 21 ms, and was 4 to 75 ms before the Stop hooks stopped reading every stored event for an empty checklist and writing the full-text index): the events and the cursor, the incremental monitor states, the “verified in full at T” stamp of the block table, and for AGY the receipt and the receipts-indexed marker. The daemon’s engines (SelfDirectEngine(defer_derived=True)) keep the authority commits synchronous (the events and cursor, the AGY receipt) and hold the four rows that carry no authority: the monitor states, the verification stamp, the AGY indexed marker and the shadow-comparison schedule (which carries its own premise and epoch) are validated against the commit log, the generation, the cursor and the block table whenever they are read, and a missing, stale or lost one only ever means “rebuild from the events” or “verify in full again”, never clean. A held row is read back in the same process through the merge rule the database applies (the newer (generation, byte_offset) wins; a stamp applies only to the exact table, synced_at and root, it was made for) and is written in the transaction of the next authority commit of that database, after the daemon has been idle for 1 s, at shutdown, or when 2048 rows or 16 MiB are held. A rewrite or reindex of a session, a dropped or demoted monitor and a replaced block table remove the rows they invalidate at once; a row whose lineage moved on by flush time is dropped instead of written. Another process that shares the index (a standalone hook) sees only what was written, so it rebuilds, which is correct and costs only time. A crash loses a cache. incremental.state reports deferred for a held write. SELF_DIRECT_DAEMON_DEFER_DERIVED=off makes the daemon write everything synchronously, as a standalone process does. Shadow comparison, source verification and coverage are not touched: the whole test suite also passes with every engine deferring (TSUKKOMI_TEST_DEFER_DERIVED=1 python -m pytest).
• Receipts read once per change. An AGY request read all the receipts of the conversation a dozen times (the authority tokens that detect a change between two phases, the capture of a sync, the checkpoint list). A daemon engine keeps one read-only connection to the receipts database and re-reads a conversation’s receipts only after SQLite’s own change witness (PRAGMA data_version, which differs exactly when another connection committed) moved; the file’s identity is compared with the path on every use, so a deleted, replaced or recreated database is never served from the old connection. A write that bypasses SQLite moves no witness (the index store’s row-count cache makes the same assumption).

Failure behaviour: never an in-process fallback

With the flag on, a hook’s whole answer comes from the daemon or from a visible fail-open. If the daemon cannot be reached or started the client never imports the engine and never judges itself; it answers through claude_hook_state.fail_open / agy_hook_state.fail_open: no deny, a visible warning with the reason, the failure recorded in hook_health, and ask when a contract marked risk: high matches the proposed action (the same policy as every other unavailability).

The availability budget is not the judgement deadline: once connected, a request has the normal hook budget. An MCP relay behaves the same way: a failure becomes a JSON-RPC error for the host’s first request (initialize) plus a non-zero exit (2 for a configuration the daemon rejected, such as an --omp-bridge without its key file or native library; 1 otherwise), never a silent standalone server. If the daemon goes away while a session is running (memory cap, version switch, crash), the relay answers the requests that were in flight with a JSON-RPC error, reconnects (starting a successor), replays the host’s initialize handshake and carries on; five recoveries within a minute end the process.

Versions

hello reports proto, package_version, index_schema, parser_versions, evaluator_version, features and pid. A client whose package version or proto is newer than the daemon’s asks it to drain (it stops accepting, finishes in-flight requests, closes its engines and exits) and starts the successor; an older client never replaces the daemon: it gets unknown(version_mismatch) and its hook fails open. During any switchover other hooks fail open (daemon_unavailable) rather than being judged twice. The version is the package version: two checkouts with the same number share one daemon, so after switching branches without a version bump run python -m self_directing_mcp.daemon --stop.

Observability

MCP tool results the daemon served carry served_by: "daemon" (a standalone server’s results are unchanged, so existing consumers see no new key). Host hook JSON gets no extra key; where a hook was served is recorded in the hook state database that already holds hook_health (/claude-hook-state.sqlite, agy-hook-state.sqlite), in a served_by table of counters per event, written at most every 30 s and at exit; a database without daemon rows was never served by it. Failures a client could not hand to the daemon are hook_health rows as usual.

Settings

The SELF_DIRECT_DAEMON_* limits are read from the daemon’s own environment (clients pass them on when they start it); a malformed value stops the daemon with an error instead of using a default.

Limits of this stage

Linux and WSL only; one daemon version at a time; the .env-in-working-directory behaviour of an MCP server depends on the client’s directory; every client of a user must see the same XDG_RUNTIME_DIR (or all fall back to the cache directory), or they would start separate daemons (index locking keeps that safe, not efficient); a request that exceeds its budget leaves its worker thread running until its cooperative deadline; the judgement is the standalone judgement (same verdicts, same verification, same shadow comparison), made cheaper by what a resident process can keep (see Held writes), and what is still slow is listed in the measurements below (one synchronous commit per appended request, the shadow comparison, and for AGY the receipts commit); a hook arriving while the daemon is restarting fails open instead of waiting for it. Daemon-side served_by for hooks is a counter, not a per-call record.

Measurements

Measured with scripts/bench_daemon.py --runs 30 on synthetic data (AMD Ryzen 7 5800X under WSL2, ext4, Python 3.12, other work running on the machine: load average 2.0 to 3.1 during the runs; 1 MB sessions, 10 contracts, local-only, 30 interleaved runs per cell after a discarded warm-up). The “daemon” columns are the real hook script as a fresh process talking to a warm daemon; the standalone columns are the same script with the flag unset on a separate, identically built index. A cell that shows before → after compares the daemon commit this warm-path work started from with the result of it, on the same machine and benchmark.

What the daemon removes is the process cost: interpreter start, imports, settings, engine and index setup (about 190 ms of a 250 ms Claude Code PreToolUse). The thin client alone costs about 20 ms end to end (12.6 ms of it is the bare interpreter start; the client adds the standard-library imports and one socket round trip), its peak resident set is about 14 MB (a bare interpreter: 11 MB; a standalone judging hook: 68 to 77 MB), and the daemon’s own handling beyond the judgement is about 0.25 ms. What is left is the judgement, and the warm-path work cut it from 32 to 133 ms to 9 to 59 ms by removing commits and repeated work, never checks. Profiled with scripts/bench_warm.py (below), what remains of a judgement is: one synchronous fsync-ed commit for every request that appends events (10 to 15 ms on this machine; AGY makes a second one for the host receipt, and a third when a result binds to its receipt), 4 to 21 ms of CPU, and the shadow comparison. Under the fixed 1-in-16 sample these tables were measured with, Claude Code PreToolUse (nothing appended) is a request whose offset the deterministic sample selects; with nothing appended every request is at that offset, so this cell measures the comparison (a full evaluation of the same events) in every request: 25 ms, against 4 ms without it (SELF_DIRECT_AUDIT_SHADOW_EVERY=0). Because 1 in 16 is more than the 5% a p95 leaves out, the p95 of every cell with appended events is the cost of a request that was shadowed, which is a full evaluation more: with 10 contracts 17 ms on Claude Code PreToolUse, 42 ms on Stop and 64 ms on AGY Stop (its evidence classes make the full evaluation dearer); with 100 contracts 270 ms on Claude Code Stop and 360 ms on AGY Stop. Which of the 30 requests of a cell are shadowed is chance, so the p95 of those cells moves by 10 to 15 ms between two runs (a second run of this benchmark gave Stop with 4 KB appended 29 / 73 ms instead of 27 / 60). That was the fixed sample. The default is now adaptive (see Incremental audits), and Shadow comparison cost repeats these cells before and after.

Idle MCP sessions (12, one shared index, after the handshake and one tool call): twelve standalone servers use 971 MB resident (588 MB proportional); twelve relays plus one daemon use 251 MB (119 MB proportional): each relay 13.7 to 14.4 MB, the daemon 83.5 MB. After the hook runs above the daemon, holding two engines, was at 99.9 MB. (Measured before the warm-path work; it holds a few more kilobytes per session and is not expected to change.)

Against the roundtable’s provisional targets: relay at most 15 MB, met (14.4 MB); daemon at most 120 MB, met (83.5 to 99.9 MB); 12 sessions at most 300 MB in total, met (251 MB); the new-process hook p95 of at most 40 ms is met for events without a judgement (UserPromptSubmit: 23 ms) and not met for the judging events (Claude Code PreToolUse 58 / 66 ms, Stop 33 / 81 ms; AGY 42 to 90 ms); the daemon-side processing p95 of at most 15 ms is met for the daemon’s own work (0.3 ms beyond the judgement, 1 ms for UserPromptSubmit) and for a warm judgement that appends nothing and is not shadowed (Claude Code PreToolUse 4 ms, Stop 9.7 ms, AGY Stop 16 ms), and not met wherever a request appends events or is shadowed (one commit is 10 to 15 ms before any CPU; the shadow comparison is a full evaluation). A Stop with 100 contracts (bench_warm.py --contracts 100, 60 requests per cell) is 15 / 18 ms at p50 / p95 with nothing appended and 40 / 51 ms with 4 KB appended on Claude Code (112 / 136 ms before), 57 / 66 ms on AGY with the shadow sample off, against the target of 80 ms at p95, met except in a window that holds a shadowed request (310 ms on Claude Code and 420 ms on AGY with 100 contracts when every request is shadowed). Two things would still move these numbers: write-ahead logging for meta.sqlite and the receipts database, now implemented as the opt-in SELF_DIRECT_SQLITE_JOURNAL=auto and measured in the commit benchmark (a resident engine’s append transaction is 68 % cheaper at p50, a fresh process’s whole append is 10 % slower, which is why it is not the default), and a faster or rarer shadow comparison (a weaker check, deliberately not done).

---

Memory, concurrency and deadlines

Long-running servers sharing one index had grown to many gigabytes in a real deployment (#5). The causes were whole-file captures per request, concurrent reparses of the same session and heap that was freed but not returned. 0.6.0 changes:

• No retained source bytes (see Source capture).
• Single-flight per (index, provider, session id). Concurrent hooks for one session wait for one capture/parse/publish, then find the cursor current. Copies of one session id in several folders share the same flight. Across processes, a state-token compare-and-swap still decides which writer publishes, and a stale capture is retaken instead of rewinding a cursor that another process already moved forward.
• Streaming reparse. Parsed rows go through a disk-backed spool (RowSpool, memory below 1 MiB) and are published in batches inside one transaction.
• Per-session id queries. Sparse and dense catch-up query only the synced session’s ids, in batches. Candidate vectors are held as float32 (the optional NumPy backend still stores float64).
• Slim audit loading. An audit pages in only the rows its contracts inspect, 512 at a time, and prerequisite search is linear rather than quadratic.
• Bounded heavy calls. At most SELF_DIRECT_MAX_CONCURRENT_CALLS (default 4) heavy calls run at once: sync, search, audit, check_action, checklist, graph, activity and hook calls. A request waits for a slot only within its own deadline; one that expires while queued returns request_deadline without starting work. A slot is held until the worker thread really finishes, even if its caller timed out. list_contracts, upsert_contracts, revoke_contract, get_chunk, audit_status and the workflow_* tools bypass the limit.
• malloc_trim on glibc. When the last running heavy call that took at least 0.5 s finishes (at most once per 5 s), the server calls malloc_trim(0). It is skipped on Windows, macOS and musl, and failures are ignored.
• audit_status no longer scans FTS. The sparse_count reads the chunk_rows mirror table instead of counting the FTS5 table while holding the shared lock.

Measurements

Measured by the maintainer; the real-session figures use private corpora that are not in this repository.

The issue’s 19.9 GB observation came from one process after about 3.5 days with a growing ~310 MB session. The roughly 7 GB figure for that case in #5 is an extrapolation from the 100 MB synthetic run, and the PR did not measure a 310 MB live session. Memory under such a session has to be confirmed by observation after deployment.

Deadlines and errors

The shared-index fix (#3) separates stable source preparation, guarded publication and detached audit evaluation. Audit snapshots contain only scoped authoritative event rows and counts plus detached rule content, and deterministic checks do not repair FTS or restore embedding caches. Explicit sync still repairs derived indexes, and sparse/hybrid search materializes deferred canonical evidence before querying.

Audit evaluation and source hashing or parsing do not hold the shared writer lease. Before a result is published, metadata, source observations, rules and AGY receipts are revalidated; changed authority makes the result incomplete (unknown) while keeping the captured findings. Startup may add the AGY receipt-version side table without rewriting the six-column cursor schema or changing SQLite journal modes.

Requests share an absolute deadline and cancellation control through nested calls and native callbacks. Non-authorizing responses distinguish index_busy, request_deadline, request_cancelled and operation_timeout; unrelated SQLite failures are not converted into successful empty searches. A stopped request cannot begin a later publication, but an admitted commit or one already-running blocking call may finish. This is cooperative stopping, not forced thread termination or proof of rollback.

Default combined writer-lock wait (1 second), audit budget (8 seconds), ordinary budget (45 seconds) and explicit-tool cap (600 seconds) are unchanged. Real writes, initialization, raw captures and derived persistent mutations remain guarded and can still contend. See implementation and verification boundaries.

Ti