zambav/comfyui-agent-toolkit

Run ComfyUI in plain English from your AI agent. Image, video, and LoRA workflows — any install, any model, any agent harness.

Quick start · Worked example · What you can ask · Why it works · Works with any agent · When to use · Repo structure

---

Talk to ComfyUI in natural language

Stop hand-editing JSON workflows. Tell your agent what you want, and it builds the workflow, queues the jobs, watches them run, and reports back.

You:    "batch this entire image folder into cinematic portraits and pick good prompts"
Agent:  "Where are the photos? Where should outputs go? What should I call this batch?"
You:    "D:/photos/portraits, D:/photos/portraits/cinematic, 'portraits_v1'"
Agent:  "47 images found. Running FLUX 2 image edit, one prompt, randomized seed."
         [.. 47 jobs queued, each one blocks on completion, each one verified with /history ..]
Agent:  "Done. 47/47 OK. Output: D:/photos/portraits/cinematic/batch_01/"

No JSON editing. No fire-and-forget queueing. No silent failures. Just: describe it, run it, get the files.

---

Quick start

# 1. Drop the skill into your agent's environment
git clone https://github.com/Zambav/comfyui-skill-public.git
cd comfyui-skill-public

# 2. Read the right entry point
#    - AI agent       -> AGENTS.md
#    - Human operator -> SKILL.md

# 3. (Optional) Copy the production helper into your project
cp scripts/api_lib.py /path/to/your/project/

# 4. Install runtime dependencies
pip install requests websocket-client pillow

Then start issuing ComfyUI tasks to your agent. That’s it.

---

What you can ask

Every trigger goes through the same proven execution path (see Why this works).

---

Worked example

Real session, abbreviated. The agent follows the patterns in scripts/api_lib.py and reference-implementations.md.

User: “Take everything in D:/renders/sci-fi and turn it into a moody cyberpunk cinematic. Use one good prompt. Output to D:/renders/sci-fi/cinematic.”

Agent does, silently:

# 1. Discover the install
from api_lib import comfyui_is_alive, get_object_info
assert comfyui_is_alive("http://127.0.0.1:8188")
nodes = get_object_info("http://127.0.0.1:8188")
assert "Flux 2 Image edit workflow API.json" or equivalent exists  # confirmed via object_info

# 2. Load + patch the base workflow
from api_lib import load_workflow
import copy
wf = copy.deepcopy(load_workflow("workflows/flux_2_i2i.json"))
wf["6"]["inputs"]["text"]    = "moody cyberpunk cinematic, neon rim light, volumetric fog"
wf["198"]["inputs"]["image"] = copy_to_comfyui_input("D:/renders/sci-fi/shot_001.png")
wf["225"]["inputs"]["output_path"] = "D:/renders/sci-fi/cinematic/batch_01"

# 3. Queue, block on completion, verify with /history
from api_lib import queue_prompt, wait_for_completion
import uuid, websocket
client_id = str(uuid.uuid4())
ws = websocket.WebSocket()
ws.connect(f"ws://127.0.0.1:8188/ws?clientId={client_id}")
prompt_id = queue_prompt(wf, host="http://127.0.0.1:8188", client_id=client_id)
job = wait_for_completion(prompt_id, ws=ws, host="http://127.0.0.1:8188")
# -> "OK: shot_001.png"  (only after the server confirms success, not just submission)

The same loop runs 23 times. The agent reports: “23/23 OK. Output: D:/renders/sci-fi/cinematic/batch_01/”.

---

Why this works

This skill is built on a battle-tested execution pattern. Every job — image, video, LoRA — goes through the same loop:

load workflow (once) -> deep copy -> patch 3-5 nodes
        |
        v
queue via POST /prompt
        |
        v
BLOCK on WebSocket until ComfyUI says done
        |
        v
VERIFY with GET /history/{prompt_id}
        |
        v
record result -> next job (one persistent WS, one client_id, whole batch)

---

Architecture

Two layers, both first-class:

+---------------------------------------------------------------+
|  Layer 1: comfy-cli (official lifecycle tool)                 |
|    Setup, server lifecycle, custom nodes, models              |
|    -> comfy install / launch / stop / node / model            |
+-----------------------------+---------------------------------+
                              |
+-----------------------------v---------------------------------+
|  Layer 2: REST + WebSocket API + skill scripts                |
|    Workflow execution, param injection, monitoring, downloads |
|    -> POST /api/prompt, GET /api/view, WS /ws                 |
|    -> scripts/api_lib.py, scripts/template_run_batch.py        |
+---------------------------------------------------------------+

The official CLI is excellent for install and lifecycle. The REST/WS API fills in execution, monitoring, and batch orchestration. Each layer has a clean job; the agent picks the right one.

---

Works with any agent

This skill is agent-harness agnostic. It’s a set of Markdown docs, Python helpers, and demo workflows — not a binary or a platform plugin. Drop it into any agent that can read files and run Python.

Tested and supported:

What the agent does, regardless of harness:

1. Read SKILL.md to confirm the skill applies.
2. Read the matching prompting-guides/*.md for prompt style.
3. Read reference-implementations.md for the node map and patch pattern.
4. Run python3 scripts/api_lib.py (or import from your own code) to talk to ComfyUI.

No agent-specific API keys, no platform lock-in, no vendor handshake. The skill talks to ComfyUI over plain HTTP and WebSocket.

---

Bring your own workflows

The demo-workflows/ folder is just a starting point — three example graphs (FLUX 2 image edit, FLUX 2 T2I, LTX 2.3 video) that happen to work on one specific install. They will not match your node set, your model filenames, or your custom nodes.

The skill works with any ComfyUI workflow you have. A few of the things the included reference docs cover:

• ✅ Any checkpoint loader (CheckpointLoaderSimple, UNETLoader, dual-file loaders, community wrappers)
• ✅ Any text encoder / CLIP loader pair
• ✅ Any VAE loader
• ✅ Any image / video / audio output node (SaveImage, ttN imageOutput, VHS_VideoCombine, SaveVideo, custom save nodes)
• ✅ Any custom node pack installed on the target machine

We strongly encourage you to bring your own workflows. Export any workflow from your ComfyUI in API format (Menu → Workflow → Export (API)), then ask your agent:

“I have a custom ComfyUI workflow at /path/to/my_workflow.json. Help me use it through this skill.”

The agent will:

1. Read your workflow’s API JSON
2. Discover what nodes, models, and encoders it depends on via /object_info
3. Add a node map to reference-implementations.md (or a new file) describing which nodes to patch per job
4. Wire it into a batch script using the same scripts/api_lib.py patterns
5. Hand back a working end-to-end loop

This is the loop the skill is built around: discover → patch → queue → block → verify, on any graph, with any model, on any install.

---

When to use this

Great fit if you…

• Run ComfyUI from an AI agent and want clean natural-language control
• Need the same skill to work on local, remote, and cloud ComfyUI installs without rewriting
• Run large batch jobs (overnight, multi-prompt) and need state tracking, recovery, and progress reporting
• Build LoRAs with structured dataset prep + JoyCaption descriptions
• Want a tested reference implementation you can copy or import, not a doc that re-derives the patterns every time

Skip it if you…

• Only click around in the ComfyUI web UI — this skill assumes agent-driven or scripted execution
• Need a single-machine, single-workflow tool — for that, a one-off script is faster
• Want a GUI front end — this is an API + skill package, not a UI
• Need a closed-source or proprietary ComfyUI alternative — this is MIT-licensed and assumes stock ComfyUI

---

What’s in this repo

---

Common workflows

FLUX image edit (I2I)

You:    "edit these photos to look cinematic"
Agent:  "What's the prompt? One batch or multiple?"
You:    "warm lighting, film grain, photoreal"
Agent:  runs FLUX 2 image edit batch -> outputs next to source images

See FLUX 2 node map.

LTX 2.3 video (I2V/T2V)

You:    "make a video of this image with a slow push-in"
Agent:  "What's the prompt? I2V or T2V? What should the output filename be?"
You:    "slow push-in, subject turns to camera, golden hour, scene_01"
Agent:  stages input, queues job, blocks on completion, returns outputs

See LTX 2.3 node map.

JoyCaption + LLM-driven prompts

You:    "joycaption this folder and generate cinematic prompts for each"
Agent:  checks for existing joycaption.md
        -> if found, asks: use it / update it / start fresh?
        -> if not, runs JoyCaption -> captions.txt -> LLM reads + writes prompts JSON
You:    confirm prompt style
Agent:  queues the batch

See JoyCaption convention.

LoRA training

You:    "train a LoRA on this dataset"
Agent:  structures dataset folders, configures training parameters,
        uses JoyCaption for descriptions, applies dynamic prompting,
        manages checkpoint output, deploys to ComfyUI

See models.md for the LoRA loading + training pipeline.

---

Pipeline overview

LLM-driven prompting pipeline (video)

JoyCaption -> captions.txt (raw scene descriptions, never modified)
      |
      v
LLM reads captions.txt + prompting-guides/ltx-2.3-prompting-guide.md
      |
      v
ltx_video_prompts.json (versioned, unique prompt per image)
      |
      v
Batch runner loads JSON -> queues to ComfyUI

LoRA training pipeline

Dataset prep -> JoyCaption-assisted descriptions -> dynamic prompting variants
      |
      v
Training config + run -> checkpoint management
      |
      v
Deploy selected checkpoint to ComfyUI

---

Repository structure

comfyui-skill-public/
├── SKILL.md                       # Trigger scope + read path
├── AGENTS.md                      # AI agent cold-start instructions
├── README.md                      # You are here
├── setup.md                       # First-run / unknown install onboarding
├── api.md                         # REST + WebSocket API reference
├── workflow-patterns.md           # Programmatic graph building + validation
├── models.md                      # Family-specific loader / LoRA / training guidance
├── dependencies.md                # ComfyUI + Python packages + JoyCaption
├── batch-operations.md            # Queue-and-watch, state, recovery, intake questionnaire
├── reference-implementations.md   # Node maps (FLUX I2I, FLUX T2I, LTX, WAN, Hunyuan) + code patterns
├── cron-jobs.md                   # Scheduler-agnostic monitoring SOP
├── prompting-guides/
│   ├── README.md
│   ├── general-prompting-guide.md
│   ├── ltx-2.3-prompting-guide.md
│   ├── flux-2-prompting-guide.md
│   └── wan-2.2-prompting-guide.md
├── scripts/
│   ├── api_lib.py                 # Production ComfyUI API helper (persistent WS, reconnect, env-var config)
│   └── template_run_batch.py      # Sanitized batch-run starter script
├── docs/
│   ├── Banner_openclaw.png        # Hero banner (this README's top image)
│   └── joycaption-convention.md   # JoyCaption naming, format, and pipeline convention
├── demo-workflows/                # READ ONLY examples — replace with your own
│   ├── Flux 2 Image edit workflow API/
│   ├── Flux 2 image gen/
│   └── ltx 2.3/
├── changelog.md
├── CONTRIBUTING.md
├── SECURITY.md
├── CODE_OF_CONDUCT.md
└── LICENSE

Demo workflows are just starting points. They are tied to one specific ComfyUI install. We strongly encourage you to bring your own workflows — see Bring your own workflows above for how. demo-workflows/README.md is the per-example breakdown.

---

Troubleshooting

---

Project standards

• Security policy: SECURITY.md
• Contribution guide: CONTRIBUTING.md
• Code of conduct: CODE_OF_CONDUCT.md
• Release history: changelog.md

---

License

MIT — see LICENSE