dreamers-laboratory/video-input-to-modern-llm
API developmentGive an LLM eyes on a video without blowing the context window: ffmpeg frame sampling at a chosen resolution, token estimates per model family, budget fitting, contact sheets.
Overview
A model cannot watch a video. It can look at frames, and every frame costs tokens whether or not it mattered. This is a skill plus two scripts that make the choice of frames deliberate: sample with ffmpeg at a resolution you name, estimate the token cost before anything is attached, and fit a budget by trading frame count against resolution. Works as a Claude Code skill, a Codex skill, or plain instructions for any agent that can run a script. Needs ffmpeg and ffprobe on PATH, nothing else. Every frame has its timestamp burned into the corner and listed in manifest.json with its dimensions and estimated tokens per model family. Contact sheets tile the frames into one image for a cheap first look. Then meter what you are about to attach: Estimates use each provider's published sizing rules: Anthropic's pixels ÷ 750 after its own downscale, OpenAI's 512-px tile rule (170 per tile + 85) and 32-px patch rule (capped at 1536 patches). They are estimates. measure_tokens.
README
video-input-to-modern-llm
A model cannot watch a video. It can look at frames, and every frame costs tokens whether or not it mattered. This is a skill plus two scripts that make the choice of frames deliberate: sample with ffmpeg at a resolution you name, estimate the token cost before anything is attached, and fit a budget by trading frame count against resolution.
Works as a Claude Code skill, a Codex skill, or plain instructions for any agent that can run a script. Needs ffmpeg and ffprobe on PATH, nothing else.
Install
git clone
cp -r video-input-to-modern-llm ~/.claude/skills/video-input-to-modern-llm # or ~/.codex/skills/
Use
python3 scripts/sample_video.py demo.mp4 --out /tmp/frames \
--mode scene --resolution low --sheet 4 --token-budget 3000 --model-family claude
demo.mp4: 1920x1080 30.0 fps 00:04:12.40
mode=scene resolution=low -> 15 frames at 512x288 in /tmp/frames/
tokens (frames): claude~2955 openai-tile~3825 openai-patch~2160
tokens (sheets): 1 sheet(s) claude~1842 openai-tile~1445
manifest: /tmp/frames/manifest.json
Every frame has its timestamp burned into the corner and listed in manifest.json with its dimensions and estimated tokens per model family. Contact sheets tile the frames into one image for a cheap first look.
Then meter what you are about to attach:
python3 scripts/measure_tokens.py /tmp/frames/detail
python3 scripts/measure_tokens.py /tmp/frames/detail --exact # real count via Anthropic's endpoint
The knobs
| flag | what it does |
|---|---|
--mode interval --every S |
one frame every S seconds |
--mode fps N |
N frames per second |
--mode scene --scene-threshold T |
frames where the picture changes; lower T finds more cuts |
--mode keyframes |
the encoder’s I-frames, a fast rough pass |
--resolution low|medium|high|full |
long side 512, 768, 1024, or native; never upscales |
--width PX |
explicit long side |
--max-frames N |
cap any mode by thinning evenly |
--token-budget N --model-family claude|openai |
fit the budget: fewer frames first, then lower resolution, never silently over |
--sheet COLS |
build COLSxCOLS contact sheets, capped at 1568 px wide |
--start S --end S |
window the video |
--audio |
also write a mono 16 kHz WAV for transcription |
Token estimates
Estimates use each provider’s published sizing rules: Anthropic’s pixels ÷ 750 after its own downscale, OpenAI’s 512-px tile rule (170 per tile + 85) and 32-px patch rule (capped at 1536 patches). They are estimates. measure_tokens.py --exact asks Anthropic’s token-counting endpoint for the real number without creating a message. OpenAI has no counting endpoint for images, so those columns stay estimates, and some models apply a multiplier on top of the patch count.
The skill
SKILL.md is the part the agent reads. Its rule is two passes: cheap contact sheets to find where to look, then full frames only at the timestamps that answer the question, with the token cost measured and reported before anything is attached.
Test
python3 test/run.py
Synthesizes a 12-second video with ffmpeg, runs every mode, and checks frame counts, timestamps, resolution caps, budget behavior, and sheet math. No network.
Apache-2.0. Built by Dreamers Inc.
Recommended Tools
Try a different keyword or remove a filter.
Install
npx skillfish add dreamers-laboratory/video-input-to-modern-llm