waynejin0918/omni-rewriter

---

中文 · Documentation · Getting Started · Architecture · Adapters · Roadmap · Contributing

News

Scroll the window for older updates. Same compact news as the project site. GitHub may flatten CSS; older items also fold below.

About

Open Agent Harness for image/video prompt expansion (PE): schemas, orchestration, validation, and dialect render — not a bundled PE checkpoint, not a media generator.

Expand ≠ generate. VLM pairwise scoring is post-hoc only. SFT/RL writers are roadmap.

How it works

This is the method. The Agent Harness owns the loop; the Writer LM is called only for Draft / Repair. Validation and render stay deterministic. Output is PE text/JSON — never media.

flowchart LR
  req[RewriteRequest] --> analyze[Analyze]
  analyze --> draft[Draft]
  draft --> validate{Validate}
  validate -->|repairable| repair[Repair]
  repair --> validate
  validate -->|ok| render[Render]
  render --> pe[PE text / JSON]
  writer[(Writer LM)] -.->|structured JSON| draft
  writer -.->|repair JSON| repair

Same path for CLI (omni-rewriter expand) and HTTP (POST /v1/expand). Details: architecture.

Writer LM agents

The harness talks to Writers over one contract: OpenAI-compatible chat + structured JSON. It does not ship a fine-tuned PE checkpoint. How you host the Writer is separate from which model family you pick.

How the two Writer classes connect

• Core agents (closed frontier) — GPT-5.6, Claude Opus 5, and similar. Connect via a vendor API or an OpenAI-compatible gateway (OMNI_WRITER_BACKEND_BASE_URL + model name).
• Terminal / open weights — QwenLM today; MiMo, Kimi, DeepSeek wanted. Serve locally (or on your cluster) with vLLM or SGLang, then point the same env vars at that OpenAI-compatible endpoint.

Three access modes (same protocol)

1. API — hosted frontier agents; no local GPU required for the Writer.
2. vLLM — common path for open-weight Writers (/v1/chat/completions, structured output).
3. SGLang — alternate OpenAI-compatible serve path for open Writers (also used by optional image/video generation adapters outside expand).

Generation adapters stay outside expand — see the compatibility matrix.

Model ecosystem

Community board — left color = category (Video / Image / Unified); right = evidence depth (PE / adapter / unverified / wanted). Prefer small PRs with a title prefix — see CONTRIBUTING.md.

 

 

 

Use the model contribution skill. Evidence-scoped details: compatibility matrix · full backlog.

Try it (no GPU)

Lint a PE envelope. This path does not call a Writer and does not generate media.

pip install omni-rewriter
# or: uvx --from omni-rewriter omni-rewriter validate kite.json

curl -fsSL -o kite.json \
  https://raw.githubusercontent.com/WayneJin0918/Omni-Rewriter/v0.1.0/tests/fixtures/t2va_kite.json
omni-rewriter validate kite.json

In another repo’s GitHub Actions:

- uses: actions/checkout@v4
- uses: WayneJin0918/[email protected]
  with:
    files: prompts/**/*.json

Expand (below) still needs an OpenAI-compatible Writer. Expand ≠ generate.

Reconstruct a clip

Turn a local short mp4 into validated H3 t2va PE. The source stays on disk.

omni-rewriter reconstruct clip.mp4 --pack-only --pack-dir /tmp/pe-pack
omni-rewriter reconstruct --from-observation docs/design/examples/observation_kite.json
omni-rewriter reconstruct clip.mp4

--pack-only needs ffmpeg, not a Writer. --from-observation needs a text Writer. A full mp4 read needs a vision Writer. Generate remains a separate adapter step.

Quick start (expand)

Gallery demos need no GPU. Preferred local path: SGLang Qwen3.6-35B-A3B (language + vision Writer) + SGLang MiniMax-H3 (~30B FL2VA). Hosted API Writers are a fallback. Expand ≠ generate — H3 is only for optional media after PE.

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e ".[cli,server]"
cp .env.example .env

1) Local SGLang (recommended)

Terminal A — Qwen3.6-35B-A3B Writer (OpenAI-compatible chat on :8000):

export OMNI_WRITER_MODEL=Qwen/Qwen3.6-35B-A3B          # HF id or local snapshot
export OMNI_WRITER_SERVED_MODEL_NAME=Qwen/Qwen3.6-35B-A3B
# optional: OMNI_WRITER_TENSOR_PARALLEL_SIZE=4
bash scripts/serve/serve_sglang_qwen_writer.sh

Terminal B — MiniMax-H3 FL2VA via SGLang diffusion (videos API on :30010):

export OMNI_WRITER_H3_MODEL=/path/to/MiniMax-H3/FL2VA  # local H3 checkpoint
export OMNI_WRITER_H3_NUM_GPUS=8                       # match your node
bash scripts/serve/serve_sglang_h3.sh

Point Omni-Rewriter at both, then expand (and optionally generate):

export OMNI_WRITER_BACKEND_BASE_URL=http://127.0.0.1:8000/v1
export OMNI_WRITER_BACKEND_MODEL=Qwen/Qwen3.6-35B-A3B
export OMNI_WRITER_H3_BASE_URL=http://127.0.0.1:30010

omni-rewriter expand examples/requests/t2va_kite.json
omni-rewriter expand examples/requests/t2va_kite.json --output h3
omni-rewriter expand examples/requests/ltx_t2va_workshop.json --output ltx
omni-rewriter validate output.json
# optional generate (needs H3 up): scripts/promo/submit_h3_chunk.py … — see H3 adapters

Lint PE JSON in CI (GitHub Action)

Same checker as omni-rewriter validate (no Writer, no media):

- uses: actions/checkout@v4
- uses: WayneJin0918/[email protected]
  with:
    files: prompts/**/*.json

Local multi-file helper: python scripts/validate_pe_files.py tests/fixtures/**/*.json. See Discussions #4. Marketplace listing copy: docs/pe-validate-action.md.

2) Hosted API Writer (fallback)

No local Writer GPU — use any OpenAI-compatible chat API (H3 generate can still be local):

export OMNI_WRITER_BACKEND_BASE_URL=https://api.openai.com/v1   # or any gateway
export OMNI_WRITER_BACKEND_MODEL=gpt-5.6
export OMNI_WRITER_BACKEND_API_KEY=sk-...

omni-rewriter expand examples/requests/t2va_kite.json --output h3
omni-rewriter expand examples/requests/ltx_t2va_workshop.json --output ltx

Checked-in requests: examples/requests/. Scripts: scripts/serve/. Details: Getting Started, H3 adapters.

Project layers

RewriteRequest
  └─ Agent Harness       analyze · draft · validate · repair · render  (= PE flow)
      └─ PE profile      H3 / Seedance / LTX-2.5 / Seedream / Qwen-Image dialect
          └─ adapter     optional vLLM / SGLang / vendor client  (generate, not expand)
              └─ eval    structural checks · RAW/PE demos under docs/

• Harness: contracts + agent loop (this release).
• Profiles: public prompt dialects for video/image generators.
• Adapters: opt-in generation clients; never called by service.expand.
• Evaluation: structure-first checks; VLM pairwise is post-hoc only.

Documentation

Development

python -m pip install -e ".[dev]"
ruff check .
mypy src
pytest
python -m build

Contributions are welcome across core schemas, dialects, adapters, evaluation, documentation, and future SFT/RL work. Start with CONTRIBUTING.md and ROADMAP.md.

Scope and license

Omni-Rewriter does not attempt to reproduce undisclosed closed-source behavior. It uses public contracts and reproducible examples to help the community close the gap between polished demos, public APIs, and deployable workflows. Untested runtime compatibility is labeled unverified.

Source code is licensed under Apache License 2.0. Third-party models, services, documentation, and names remain subject to their own terms. Security guidance is in SECURITY.md.