mupozg823/timecode-agent

TIMECODE-AGENT helps coding agents turn long video into a local, reusable evidence ledger.

• Transcript first. Ingest produces a timestamped transcript (reusing the uploader’s captions when they exist) plus deterministic placement signals — scenes, audio energy, OCR, faces — before any frame is sent to a model.
• Lazy visual verification. The coding agent inspects only the moments where a claim needs visual confirmation, and records what it verified as time-addressed checkpoints instead of a one-shot answer.
• Evidence that persists. Checkpoints, capture provenance, and edit decisions live in append-only ledgers; indexes, a static corpus browser, and wiki pages are rebuildable projections over them.
• Grounded editorial handoff. EDL, FCPXML, and OTIO exports are derived from recorded checkpoints or a pinned sequence — not from an unstored model reply; clip extraction and SRT rendering are supporting utilities that work on any span or the full transcript.

Typical uses: answering questions about long footage without re-watching it, cutting highlights or shorts with receipts for every cut, and growing a searchable corpus and wiki across many videos.

Quickstart

git clone https://github.com/mupozg823/timecode-agent.git && cd timecode-agent
uv tool install --python 3.12 .

va ingest show.mp4 --model small --signals
va brief va-out/show
va capture va-out/show -t 1:23 -t 95
va status va-out/show

The first pass builds reusable local artifacts. Your coding agent reads those artifacts, inspects decision-critical gaps, and records supported checkpoints before any grounded edit is exported.

Recently shipped

• Legacy workspaces promote in place — va rebind backfills pre-0.3.0 workspaces into revision binding (dry-run by default) and refuses when the source is gone, so frozen corpora accept new evidence again.
• Procedures compile into skill drafts — recurring, visually grounded task-* steps become approval-pending skill drafts gated on tool-consensus intent coherence; va skillgen --route names what still blocks each candidate.
• Beat-quantized rhythm editing — va beats extracts a beat grid from a music track and va beat-eval gates sequence joins against it (p90 offset <= 40 ms), with a ledger-preserving snap proposal.
• Story Map — both ledgers projected onto one proportional timeline with grounding cross-highlight, a cut-provenance lane and speaker ribbons.
• Persistent BM25 search index — fingerprint-incremental FTS5 with hybrid Korean recall and RRF-routed ranking.
• Experimental Windows support — the workspace lock has an msvcrt fallback and CI runs a Windows smoke job; see Requirements.

All shipped to main. The current release is v0.4.0; v0.1.0 was the first.

---

Why it exists

A fixed-sampling workflow sends frames to a model before it knows which moments matter. Short events can fall between samples, similar frames consume the same budget repeatedly, and the next question often starts with no durable state from the previous one.

TIMECODE-AGENT separates two concerns:

1. Placement: deterministic signals and bundled perception tools locate moments that may deserve attention.
2. Selection: the coding agent decides what a claim means, what evidence can support it, and whether more inspection is worth the cost.

The package stores those decisions as time-addressed artifacts instead of leaving them in a one-shot model response.

This table describes an architectural contrast, not a performance claim. The included public benchmark is a regression gate; it does not establish answer superiority, optimal frame selection, or human editing preference.

How it works

flowchart TB
    V["video file or URL"] --> I["ingest"]
    I --> T["timestamped transcript"]
    I --> S["placement signalsscenes · audio · OCR · faces"]
    T --> A["understanding loop — Kubrick"]
    S --> A
    A --> C["checkpoints.jsonlclaims · spans · support"]
    C --> P["indexes · static view · wiki · answers"]
    C --> E["editing loop — Kuleshovread-only understanding snapshot"]
    E --> D["sequences.jsonlselection · order · trim"]
    D --> O["clips · EDL · FCPXML · OTIO · SRT"]
    D -. "never rewrites facts" .-> C

Ingest reuses transcripts the uploader already made: a manual caption track wins, then the uploader’s original-language automatic track (*-orig is requested first); a different-language track is only a last-resort fallback, and caption requests are currently limited to ko/en. Only when no usable caption track exists does whisper transcribe the audio — --force-whisper opts back into transcription when word-level timing matters.

The signal layer proposes where evidence may be useful; it does not establish semantic truth. The coding agent performs meaning-level selection and records claims with resolvable time spans and evidence references.

Understanding and editing are separate write domains. Editing reads a pinned understanding snapshot and records its own decisions. If a fact changes, the checkpoint is corrected upstream and the affected edit is re-derived.

The two loops: the Kubrick engine and the Kuleshov loop

The two procedures above carry director names in the Agent Skill, so a request lands in the right loop with the right write permissions. The names describe procedures, not installed modules.

Using Kubrick — this is what the Agent Skill runs when you ask “analyze this video”; every step also works by hand:

va ingest lecture.mp4 --signals   # transcript + placement signals
va brief va-out/lecture           # entry point: what is known so far
va filmstrip va-out/lecture --auto   # low-res overview of uncertain stretches
va capture va-out/lecture -t 95 --reason "speaker change"
# capture prints the evidence path — a nonempty --reason is slugged into it
va checkpoint add va-out/lecture --json-file - <<'JSON'
{"id":"cp-001","span":[83,125],"status":"verified",
 "hypothesis":"Q&A opens; a second speaker joins",
 "confidence":0.9,
 "visual_evidence":["frames/t000095000-speaker-change.jpg"]}
JSON
va status va-out/lecture          # coverage and readiness (advisory)

The loop converges when decision-critical claims have current support and another observation is unlikely to change the answer — not when a fixed frame budget is spent.

Using the Kuleshov loop — what runs when you ask for a defensible cut; the Cut workflow is its everyday entry:

va highlights va-out/lecture --json          # placement candidates
va sequence add va-out/lecture --json-file - <<'JSON'
{"id":"seq-001","intent":"lecture highlight",
 "cuts":[{"span":[83.0,125.0],"order":1,"role":"hook",
          "checkpoint_ids":["cp-001"]}],
 "status":"assembled"}
JSON
va boundary-eval va-out/lecture --sequence seq-001   # score cut edges and joins
# inspect and re-snap, then promote the same id to boundary_verified —
# exporting an "assembled" sequence would leave the receipt's
# checkpoint-revision map empty
va sequence add va-out/lecture --json-file - <<'JSON'
{"id":"seq-001","intent":"lecture highlight",
 "cuts":[{"span":[83.0,125.0],"order":1,"role":"hook",
          "checkpoint_ids":["cp-001"]}],
 "status":"boundary_verified"}
JSON
va clip va-out/lecture --start 83 --end 125 --accurate
va export va-out/lecture --format otio --sequence seq-001 -o cut.otio --receipt

The boundary between the loops is one-directional: the Kuleshov loop reads the fact ledger and never writes it. A cut that rests on a wrong fact is not patched in the sequence — the checkpoint is corrected in Kubrick and the edit is re-derived.

Grounded editorial handoff

After an agent has checked the example span and the captured frame supports its reading, record that support before exporting:

va checkpoint add va-out/show --json-file - <<'JSON'
{"id":"cp-001","span":[83,125],"status":"verified","hypothesis":"Verified answer segment","confidence":0.9,"visual_evidence":["frames/t000095000.jpg"]}
JSON
va export va-out/show --format edl --ids cp-001 -o cut.edl
grep -q '^001 ' cut.edl

The values are illustrative; use the span and evidence established for your own video. The final command fails this example check if the EDL contains no edit event. For revision-pinned, machine-gated multi-cut delivery, record a terminal sequence and export it with --sequence.

What it looks like

va view writes a self-contained static corpus browser and per-video players — single HTML files, no server and no frontend framework. The pages follow the OS color scheme; dark mode is shown here.

Live tour: filter the library, explore the relation graph, then click a scene record — the player jumps to that timestamp.

The demo GIF and screenshots show a demo corpus built by running the tool on the Blender Foundation open movies Sintel, Elephants Dream, Big Buck Bunny, Tears of Steel, and Charge (CC-BY, © Blender Foundation — blender.org) — including the same freely fetchable fixtures the reproducible benchmark uses.

What persists

Each workspace keeps a manifest, transcript, append-only checkpoint events, capture provenance, corrections, and optional edit decisions. Indexes, HTML views, wiki pages, and NLE files are rebuildable projections.

va-out/
├── INDEX.md
├── view.html
├── show/
│   ├── manifest.json
│   ├── transcript.json
│   ├── checkpoints.jsonl
│   ├── image-provenance.jsonl
│   ├── corrections.jsonl
│   ├── sequences.jsonl
│   ├── .workspace.lock
│   ├── .checkpoint.lock
│   ├── .image-provenance.lock
│   ├── .sequences.lock
│   ├── frames/
│   └── wiki/
└── another-video/

Ingest publishes source, transcript, and timing revision IDs in the manifest. New checkpoints, image-provenance events, sequences, and NLE receipts carry that binding. va ingest will not overwrite a published workspace or a manifest-less directory that already contains durable evidence; use va brief to resume, or a fresh -o path for a new source or transcript generation. If the first ingest fails before revision publication, the manifest remains building and the same source plus -o path can be retried safely. Seconds remain the CLI display/input surface, while new ledger records also store a half-open stream PTS/time-base span. Before a revision-bound NLE handoff, the complete decoded video timeline must prove one uniform rational cadence, including every frame duration and the final boundary; avg_frame_rate == r_frame_rate is only a hint. VFR, unknown, or irregular timing stays blocked until decoded-frame snapping/export is available. Pre-revision workspaces remain readable, but are read-only for new checkpoints, image provenance, and sequences; create a fresh ingest workspace before adding evidence. A new draft created directly through the current Python API binds its source, transcript, and timing on the first evidence write.

va search "" queries a persistent FTS5 index (word and trigram retrievers fused per source family) across workspace transcripts, checkpoints, and OCR, synced incrementally per workspace fingerprint. va view writes a self-contained static corpus browser. va wiki projects entities, relations, quotes, and scene notes into Markdown. No vector database is the source of truth.

va index, va view, and va wiki accept one corpus root, or workspace paths that share one explicitly supplied common root. Independent roots are rejected before any projection is written so links cannot silently collide. Commands hold a shared workspace operation lease while reading or writing; ingest and whole-workspace garbage collection take the exclusive side of that lease, so an active capture, transcription, signal pass, or projection cannot be removed underneath the command. Corpus commands operate on the exact workspace snapshot they leased rather than rediscovering mid-run, and ingest keeps its exclusive lease through final manifest or signal-summary output. Published legacy workspaces without .workspace.lock serialize each command through an exclusive lease on the existing workspace-directory inode.

Command surface

The same CLI is available as va and tca.

A few commands are worth a direct mention beyond the table above:

• filmstrip tiles a time window into a contact-sheet grid for fast visual scanning; --auto splits a long span into density-ruled windows instead of one oversized image. Run it locally against your own footage; this repository does not ship any third-party contact-sheet images.
• diarize adds who-spoke-when speaker turns; --backend auto prefers the gated pyannote model when a Hugging Face token is present (environment variable or hf auth login cache) and otherwise uses the ungated sherpa backend. Run it before recording checkpoints or other transcript-dependent evidence: it advances the transcript revision, and is refused once such evidence exists.
• va checkpoint observe --id --frame --subject "seated man" --state present|absent|uncertain --hypothesis "..." records one query-selected, provenance-tracked, uncropped frame as a typed person_presence checkpoint. The command derives the observation timestamp and matching visual_evidence path from that frame. Use uncertain whenever the frame does not resolve presence; it remains a hypothesized observation.
• va ask "how many times did the seated man leave the screen?" accepts narrow English and natural Korean variants of that one event-count intent. Its deterministic reader consumes typed person_presence observations and current, provenance-tracked full-resolution frames. It reports verified departures separately from uncertain intervals; ambiguous subjects, actions, or count requests fail with a specific diagnostic. Default --format human --lang auto renders Korean questions in Korean and English questions in English; --lang ko|en overrides detection. --format agent-json projects the same once-calculated envelope as compact JSON with English identifiers and reason codes. It preserves a normalized reply_locale (for example, ja-JP becomes ja) so a host LLM can localize without parsing human prose. The CLI itself neither invokes an LLM nor translates. va ask is read-only and never appends or rewrites checkpoint or image provenance. Missing evidence is not a zero count or proof.

va ask va-out/clip "앉아 있던 남자가 화면 밖으로 몇 번 나갔나요?"
va ask va-out/clip "how many times did the seated man leave the screen?" --lang en
va ask va-out/clip "how many times did the seated man leave the screen?" \
  --format agent-json --lang ja-JP

{"v":1,"intent":"person_exit_count","subject":"seated_man","reply_locale":"ja","status":"partial","count":1,"verified":[{"from_ms":12000,"to_ms":18000,"evidence":["frames/frame-000012.000.jpg","frames/frame-000018.000.jpg"]}],"uncertain":[{"from_ms":0,"to_ms":12000,"code":"UNOBSERVED_PREFIX"},{"from_ms":18000,"to_ms":30000,"code":"UNOBSERVED_SUFFIX"}],"next_ms":[6000,24000]}

• boundary-eval scores a recorded sequence’s cut points and joins (speech-cut, breath, loudness step) so an edit can be re-snapped before export instead of after review.
• beats extracts a fixed-tempo beat grid (beats.json) from a music track; beat-eval gates a sequence’s output-timeline joins against it (p90 offset <= 40ms), and --snap prints a span proposal that lands every join on a beat.

Options reference

Every flag below is taken from each command’s --help; defaults shown are the argparse or effective runtime defaults, not documentation guesses.

Commands that accept it also take --json for machine-readable output; capture, diarize, and filmstrip do not.

Measured numbers

Numbers below are from one machine and one input, stated so they are reproducible rather than persuasive — they describe this repository’s own ingest path, not a comparison against another tool.

Device: Apple M4 Pro (8 performance + 4 efficiency cores), macOS. Input: a 43-minute (2,580 s) 1080p Korean YouTube vlog.

On one sampled phrase in this video the caption track was also the more accurate source: the small model misheard “세계 정복” (world domination) as “세계정보” (world information) while the uploader’s caption carried the correct text — a sample, not a generalization.

Caption-derived cues land on roughly 10-second boundaries after rolling normalization (2,530 raw cues → 238 normalized cues covering 5.9–2,569.8 s in this input); pass --force-whisper when a cut needs tighter word-level timing than that grain.

This table describes one input on one machine and is not a benchmark; see Reproducible benchmark for the regression gate.

Case study: a silent transcription stall

va ingest transcribes with faster-whisper 1.2.1. On the same 43-minute Korean input above, its decode loop occasionally stopped emitting segments past 1,092 seconds — no exception, no warning, exit code 0.

Isolating the boundary ruled out the obvious causes: ffmpeg’s audio extraction was a complete, correctly-durationed 2,580.1 s WAV, and Silero VAD found 2,303.8 s of speech overall, including 1,274.6 s of it after the 1,092 s cutoff. The fault lived inside whisper’s own decode loop, not the input.

It was also nondeterministic on identical input: 1 of 3 back-to-back runs collapsed to 92 segments, while the other 2 completed with 757 and 895 segments respectively. A downstream consumer felt this directly — va brief on the collapsed transcript concluded the video was “19% speech, visual-led”; on a complete transcript, the same video is “95% speech, speech-led.”

The fix does not try to make faster-whisper’s decode loop deterministic. Instead, va ingest compares transcribed duration against VAD-detected speech duration and records the ratio as transcript_coverage in the manifest. Below a 0.5 coverage threshold — and only for files with at least 60 s of detected speech, so short or near-silent clips aren’t churned needlessly — it re-transcribes once with condition_on_previous_text=False and keeps whichever attempt covers more of the detected speech, stamping transcript_repair when that happens. In the two runs above, the collapsed transcript scored 0.211 coverage (0.289 below the 0.5 gate) and the healthy one 0.809 (0.309 above it) — not a hair’s-width threshold call on this input. (Implementation: _transcript_collapsed in src/video_agent/ingest.py.)

The manifest also stamps asr_backend — which backend produced the adopted transcript (faster-whisper, or an MLX fallback), suffixed +(tail) when a tail retranscribe contributed part of it — so a quality comparison can name its own source instead of guessing. It is null when no ASR ran at all, which is exactly the uploader-caption path: read transcript_source (subtitles) to identify that case.

Requirements

• Python 3.12 on macOS or Linux (uv provisions it: uv tool install --python 3.12 .)
• ffmpeg and ffprobe on PATH
• yt-dlp for URL ingest
• A coding-agent harness that can discover the included Agent Skill
• Interface language: mixed. Most --help text is English, while diagnostics, the va brief/va status summaries, and the va view UI are Korean; no global locale switch exists. The narrow va ask surface is the exception: deterministic human output supports Korean and English, while agent-json carries other reply locales to the host LLM. Ledgers and handoffs speak whatever language you write into them; see Notes.
• Windows is experimental: the core CLI, ledgers, and workspace locking run (shared locks degrade to exclusive via msvcrt), a CI smoke job exercises install → import → lock round-trip → CLI startup, and the Apple-backed cues (OCR, faces, semantic audio events) are unavailable as on Linux

Verify both are reachable before the first ingest:

ffmpeg -version && ffprobe -version
yt-dlp --version

The first va ingest for a given --model size downloads that faster-whisper model into the Hugging Face hub cache (HF_HOME/~/.cache/huggingface) — small measured 484 MB on disk here; check upstream before assuming a number on a metered connection, since published weights can change.

Media ingest, signal extraction, and persistence run locally by default. URL ingest uses the network. scripts/install.sh prepares the ungated default model artifacts up front; a direct package install may download them on first use. The coding-agent harness may use an external model service or incur its own cost.

Set VIDEO_AGENT_CACHE_DIR to choose the cache root for diarization model assets and the glossary cache (whisper models live in the Hugging Face hub cache above). On Linux, XDG_CACHE_HOME is honored when the explicit override is absent.

Installer

The portable installer checks dependencies, installs the standalone CLI with Python 3.12 and every supported runtime backend, prepares faster-whisper, MLX (Apple Silicon), and ungated sherpa model assets, and copies the Agent Skill so it survives moving or deleting the checkout:

scripts/install.sh --dry-run
scripts/install.sh

The installer includes the diarize extra, so pyannote’s Python runtime is present after scripts/install.sh; its remote model stays account-gated by Hugging Face, and without an accepted token the prepared ungated sherpa backend handles diarization immediately. A bare uv tool install --python 3.12 . skips pyannote’s ~700 MB torch stack entirely (sherpa still works); opt back in any time with uv tool install --python 3.12 '.[diarize]'. --skip-models is the explicit opt-out from initial model preparation.

Every feature starts enabled. Runtime policy can be inspected and changed without reinstalling:

va runtime status
va runtime set profile low-power
va runtime set asr-backend mlx
va runtime set clip-encoder hevc-videotoolbox
va runtime set feature.ocr off
va runtime reset

balanced deliberately retains the measured stable paths: faster-whisper for ASR and software encoding for accurate delivery clips. low-power selects MLX and VideoToolbox only on Apple Silicon; MLX uses speech-only timestamps and falls back to faster-whisper when its repetition/quality gate fails. Installing all backends therefore adds download time and disk use, but does not force the slower path or import every heavy model on ordinary commands.

Optional Obsidian and Media Extended setup is available through scripts/install.sh --all. The default installer does not alter an Obsidian vault. Contributors who intentionally want the CLI to follow a development checkout can use:

uv tool install --python 3.12 --editable .

Usage

# A YouTube / URL source — always pass -o so you know the resume path
va ingest "https://youtu.be/..." --signals -o va-out/my-video
va brief va-out/my-video

# A local file, Korean transcript, signals in one call
va ingest lecture.mp4 --model small --lang ko --signals

# Captions exist but a cut needs word-level timing — transcribe into a
# FRESH workspace (re-ingesting a populated one is refused before mutation)
va ingest "https://youtu.be/..." --force-whisper -o va-out/my-video-whisper

# Login-walled source (e.g. Instagram): add --cookies-from-browser chrome
# (or safari / firefox) — only pass this when you need your own session
va ingest "https://instagram.com/reel/..." --cookies-from-browser chrome -o va-out/reel

Ask questions through your coding agent (the Agent Skill below), then come back to the same workspace later with va search "" and va brief — stored artifacts answer before any re-inspection. Open the corpus browser any time with va view (writes va-out/view.html).

Named workflows

The Agent Skill names the four everyday command chains so a request can be routed without memorizing flag combinations. Import and Search feed the Kubrick engine; the Cut workflow is the everyday entry into the Kuleshov loop; the Archive projects what both loops recorded:

Saying “import and analyze this file” or “cut workflow, 9:16” to a harness with the skill installed resolves to these chains.

Agent Skill

skill/ is the canonical Agent Skill source. Install the whole directory through your harness’s documented skill mechanism. The installer copies it to the supported Claude Code and Codex-compatible discovery locations; symlinks are reserved for explicit development mode through --link-skills.

The default copy targets are $HOME/.claude/skills/timecode-agent and $HOME/.agents/skills/timecode-agent. Invoke it explicitly as /timecode-agent in Claude Code or $timecode-agent in Codex; compatible harnesses may also discover it from the task description.

The installed directory must be named timecode-agent so its basename matches the Agent Skill frontmatter name. The repository keeps the canonical source under skill/; scripts/install.sh applies the required installed name. Use timecode-agent as the directory name for manual links or uploaded archives.

The skill defines the transcript-first inspection loop, evidence selection, checkpoint updates, convergence checks, and editorial handoff. The package does not embed an LLM.

Reading the corpus

va index and va wiki emit plain Markdown, so the corpus does not require a particular notes application. Open va-out/INDEX.md in any Markdown viewer, or use va view for the generated static browser.

Obsidian is an optional reader for links, backlinks, properties, graphs, and timestamp navigation:

• Open va-out/ directly as a vault.
• Use Obsidian’s bundled graph, properties, backlinks, search, and file explorer features.
• Optionally install Media Extended for timestamped playback from note links.

See Obsidian setup for the guarded installer path.

Reproducible benchmark

The public regression gate uses declared CC-licensed Blender open-movie fixtures that are fetched at run time:

uv run --python 3.12 --no-dev python benchmarks/run_bench.py --set public --fetch

Notes

• Language is mixed, and handoffs inherit yours. Roughly two thirds of the --help strings are English, while diagnostics, the va brief/va status summaries, and the va view UI are Korean; no global locale switch is implemented. va ask alone has a narrow dual surface: direct human output follows Korean-versus-English question detection (or --lang ko|en), while compact agent-json preserves another requested reply locale for the host LLM. It does not perform translation itself. Timestamps and structural fields are language-neutral, but prose is not: srt carries transcript text, md renders checkpoint hypotheses, and edl/xml/fcpxml/otio embed the checkpoint situation or hypothesis in comments, markers, and metadata. Whatever language you write into the ledger is the language your NLE handoff speaks — the agent writes those fields in the language you asked in.
• Only ingest content you have the right to download and analyze. The --cookies-from-browser option exists for your own authorized sessions — never ship credentials in a repository.
• One workspace per source/transcript revision. Resume with va brief — re-running va ingest into a ready or pre-revision workspace is rejected before manifest or transcript mutation. Only an explicitly incomplete building ingest can retry the same source and -o path; use a fresh -o path for a different source or transcription. Reclaim space with va gc (dry-run by default) rather than deleting files by hand.

Documentation

• Architecture — implemented boundaries and current system model
• Research basis — adjacent research and claim limits
• Agent Skill — operational loop used by coding agents

License

MIT — see LICENSE. Benchmark fixtures are CC-licensed Blender Foundation open movies fetched at benchmark time; no third-party media ships in this repository beyond the attributed interface screenshots. Every image file in this repository is one of: brand artwork under assets/brand/, a synthetic fixture rendered locally by scripts/render_homepage_preview_fixture.py, or an interface screenshot under assets/screenshots/ whose visible video frames come from the CC-BY Blender Foundation open movies credited in What it looks like.