caught-in-4k

Your agent said “done ✅”. It never watched it run.

Agents write UI they’ve never seen move. They can’t watch video, so flicker, frozen animations and layout jumps ship with a confident “looks good!”. caught-in-4k turns a recording into something an agent can see, so it catches its own UI bugs before you do.

21 seconds of a bug repro, as one image. Red boxes mark what changed. At #7 the user deletes “Buy milk”, but “Call mom” disappears instead.

True story

An agent built a looping bonsai animation and called it done. It wasn’t watching:

$ npx caught-in-4k bonsai2.mp4
1 keyframes, 1 sheet(s)        ← the whole 4.8s loop never changed

The “animation” was a still image (0.5% of pixels moving). The agent rebuilt it, ran caught-in-4k again, and got 3 keyframes with 15% motion in the tree. That’s the loop: record → get caught → fix → get a clean run.

What it does

• Picks the frames that matter. One frame per action, once the screen settles: the result of a click, not every letter typed on the way. Blank fade-ins are skipped, and long recordings stay evenly covered.
• Catches flicker. A 60 fps pass finds flashes that last one frame and snap back, and marks them FLICKER. A model sampling video at 1 fps misses them.
• Boxes what changed since the previous frame, so the agent knows where to look. Mouse jitter and full-screen cuts are left unboxed.
• Sharp crops on big screens. On a Retina recording, the changed areas are cut from the full-resolution frame, so small UI text stays readable.
• Lines up speech with the frames, using whisper.cpp word timing.
• Zoom. Dense frames across any time range, or one full-resolution frame.
• Cheap. A 21-second repro fits in one image, about 2k tokens.

Everything runs locally. Nothing is uploaded.

Give it to your agent

MCP (tools watch_video and zoom_video, images come back inline):

claude mcp add caught-in-4k -- npx -y caught-in-4k mcp

Codex (~/.codex/config.toml):

[mcp_servers.caught-in-4k]
command = "npx"
args = ["-y", "caught-in-4k", "mcp"]

Cursor and others (mcp.json):

{ "mcpServers": { "caught-in-4k": { "command": "npx", "args": ["-y", "caught-in-4k", "mcp"] } } }

Skill: copy SKILL.md to ~/.claude/skills/caught-in-4k/SKILL.md (or your agent’s skills folder). The agent then runs it whenever there’s a video.

By hand:

npx caught-in-4k bug.mov                           # .caught-in-4k/bug/index.md + sheets
npx caught-in-4k zoom bug.mov --from 0:13.8 --to 0:15
npx caught-in-4k zoom bug.mov --from 14.8 --count 1

Does it work?

We ran a blind test on 6 public bug-report recordings from Excalidraw, Zed and VS Code against mcp-video-analyzer. Separate judge agents saw only one tool’s output, with no tool names and no issue text.

To be honest about it:

• In round 1, before our fixes, mcp-video-analyzer caught 1 and we caught 0.
• The fixes (flicker pass, sharp crops) were tuned on these same videos, so round 2 flatters us.
• Both tools are weak on a first pass without the user’s description.

The table above is round 2. Full method, both rounds and caveats are in bench/.

Roadmap

Not built yet:

• [ ] record --do "click Add, type milk": the agent records its own app with Playwright.
• [ ] check: pass/fail with an exit code (flicker, frozen animation, layout jump, blank screen), so agents and hooks can loop until clean.
• [ ] Motion summary for continuous animations: how much moves, where, how smoothly.
• [ ] A Claude Code hook that won’t let the agent stop until the check passes.
• [ ] A GitHub bot that records PR preview builds and comments what it caught.

Requirements

• Node 20+ and ffmpeg (brew install ffmpeg)
• Speech (optional): whisper.cpp (brew install whisper-cpp) and a ggml model in ~/.cache/caught-in-4k/, or set CAUGHT_WHISPER_MODEL=/path/to/ggml-small.en.bin
• URLs (optional): yt-dlp

Develop

npm install
npm test     # renders a narrated bug-repro video, then drives the MCP server over stdio

MIT licensed.