westwaters/pollard-weights

Pollard Weights are models built for a target device’s memory, not for a bit-width chart. Attention, routers and embeddings keep high precision; expert FFNs carry the compression; hot layers keep more bits when you feed the builder a measured routing profile — and the whole build is sized to whatever RAM/VRAM you point it at, minus a working reserve.

Any target, any device — there is no fixed size range. Tell it the target’s budget (--ram/--vram) and it fits that — a phone or a Pi (hundreds of MB), a 16GB Mac Mini, a 24GB 5090, a 256GB server, a 512GB wafer node. Build on one machine, deploy to another — the big-disk box you build on is usually not where the model runs, so --ram is the target’s budget, not the build machine’s (--ram auto only when they’re the same). The 16GB examples below are just a common demo, not a ceiling.

One measured allocation, five output lanes — GGUF (llama.cpp/Ollama/LM Studio), GPTQ (vLLM/SGLang), MLX (Apple), EXL3 (exllamav3), and MX/NVFP4 (Blackwell / Vera Rubin) — pick the runtime, keep the allocation.

How it works

flowchart LR
    HF["FP16 / BF16model"] -->|"optional: smooth · abliterate(pollard-hf-smooth / -abliterate)"| ALLOC
    ALLOC["Pollard measured allocation(sensitivity → KL knapsack)"] --> GGUF["GGUF · llama.cpp"]
    ALLOC --> GPTQ["GPTQ · vLLM / SGLang"]
    ALLOC --> MLX["MLX · Apple Silicon"]
    ALLOC --> EXL3["EXL3 · exllamav3"]
    ALLOC --> MX["MX / NVFP4 · Blackwell / Rubin FP4"]
    GGUF --> V["pollard-verify(real-reconstruction gate)"]
    GPTQ --> V
    MLX --> V
    EXL3 --> V
    MX --> V
    V --> WS["~/pollard workspace(organized + MANIFEST)"]

Contents

• Quick start · Output lanes · Workspace · Workflow
• Features · Proof: 7B / 14B · Measured allocation
• GPU users · Across machines · Command reference · Roadmap · Contributing · Acknowledgements

Quick start

pip install pollard-weights                      # into a venv (or uv pip install / -e )
pollard setup                                    # checks every lane on THIS machine, offers to install what's missing
./install.sh                                     # (alternative) tools + the llama.cpp runtime, one shot
pollard-calc --model Qwen/Qwen3-30B-A3B --ram 16 # what CAN this machine do
pollard-fit  --gguf model-f16.gguf --ram 16      # build a memory-fit GGUF for it
llama-cli    -m model-f16-pollard.gguf           # run it

pollard setup is the installer for everything past the core CLI. It runs a table per lane (gguf, ik, mlx, gptq, mx, exl3, exlr8, draft, draft-train, toolcall, convert) and reports each one as ready, missing X, or not possible here with the reason (MLX needs Apple Silicon; GPTQ/MX/EXL3 need an NVIDIA GPU). “Installed” doesn’t count until it works: each package is imported in a fresh interpreter, and torch runs a small op on the device it would use. A torch wheel without kernels for your card (a cu124 wheel on a Blackwell sm_120, say) is reported as the wrong torch, and a vLLM with no compiled kernels is reported as unusable. After the report it asks Install missing pieces for lanes gptq,exl3? [Y/n]. --dry-run prints the exact commands without running them, --yes installs without asking, and --lanes gptq,mx limits the check to those lanes. Installs go into the active venv, with torch from the PyTorch index that matches your GPU and driver (cu126/cu128/cu130; MPS wheels on a Mac). speculators gets its own venv because it pins transformers<5.17. The llama.cpp runtime is built for this machine: Metal on a Mac, CUDA for the detected GPU arch on NVIDIA. System pieces are never installed for you, and nothing runs under sudo. For the CUDA toolkit, MSVC Build Tools, Xcode command-line tools and git, setup prints a one-line command or link and waits for your OK. pollard-doctor --env prints the same report. Every tool picks its device the same native way: CUDA on NVIDIA (Linux or Windows), MPS/Metal on Apple Silicon (MLX for the MLX lane), CPU otherwise. pollard and pollard release run a fast cached check first. If a lane you asked for is missing something, you get a one-line hint, and a --run stops with the fix command before the build starts.

Nix is optional. It’s one packaging route for the core CLI (Nix flake), and you don’t need it. The GPU lanes’ dependencies and the llama.cpp runtime are outside the flake. With pip/uv, pollard setup handles all of that for you.

⚠️ pollard-fit alone gives you a uniform build sized to your RAM — no quality win. The win over uniform quants comes from the calibration step: measure the model, then allocate on it. See Workflow — run these in order. Start from an f16/bf16 source (requantizing an already-quantized file only loses); the tool refuses to build larger than an already-quantized source.

DRAM is provisioned today as if every weight deserves the same bits and every byte must be resident. Neither is true, and the difference is measurable — for AI models, and for the memory tiers under them (notes/beyond-models.md).

Features

Proof on real models: 7B and 14B

On Qwen2.5-7B and -14B, the Pollard mix — expert/FFN body crushed to 1-bit, attention and residual writers protected — beats the uniform 1-bit trellis quant by ~14% perplexity at the same 1-bit size class, and sits under the 2-bit ceiling. It wins on KL-to-f16, top-1 agreement and chat too, same eval and same imatrix. (WikiText-2, ctx 2048, 145 chunks, ik_llama.cpp.)

At 4-bit, Pollard’s own error-feedback GPTQ (pollard-gptq) recovers ~46% of round-to-nearest’s error — the reconstruction lever an imatrix alone can’t do:

Measured-sensitivity allocation — beats uniform imatrix-IQ, dense AND MoE

pollard-sensitivity points at a model and measures which tensors actually matter — it crushes each group one at a time and watches the KL — plus that model’s own noise curve. Then pollard-fit --sensitivity allocates bits to minimize KL-divergence for your size budget. At matched size it beats uniform imatrix-IQ: +6–27% lower KL on a dense 1.5B (5/5 sizes) and +21–43% on a 40-expert MoE (4/5) — measured against f16 on held-out wikitext, 48K tokens. MoE wins bigger because expert-importance variance is larger, so the allocation has more to exploit. Nothing is baked in: the signals are measured per model (granite’s noise curve runs ~2× Qwen’s). “Uniform at size” is the honest naive-mix baseline — linear interpolation between adjacent measured quants (see notes/e13; do not use log-log, it manufactures fake losses). It loses only at the extreme IQ2_S floor, where nothing smaller exists to compare and nothing’s left to allocate. Regenerate the chart from raw data: python experiments/plot_kl_win.py.

Output lanes — one allocation, pick your runtime

The measured allocation — protect attention / router / embeddings, crush the expert body, spend more bits where a routing profile says it matters — is the same in every lane. What changes is the format, because each runtime has its own fast path. Pick by where the model will actually run:

Low-bit note: for the trellis/error-feedback lanes (EXL3, GPTQ) at low bit, run pollard-hf-smooth on the fp16 model first — it migrates massive-activation outliers that would otherwise collapse the quantizer’s scale (or let pollard-doctor --repair handle smooth→convert→verify). GGUF has this built in via pollard-smooth. Verify any build with pollard-verify.

Choosing:

• Fits one box, want max compression → GGUF (the 1-bit trellis flagship).
• Cluster / “every token counts” serving → vLLM/SGLang via pollard-export. vLLM does real tensor-parallel over the fast network — sidestepping llama.cpp’s slower RPC pipeline-parallel (that GGUF-cluster path still exists — see Across machines — when you’d rather keep the exact GGUF and just pool RAM).
• GPU, strongest low-bit reconstruction → pollard-gptq.

The ~1-bit trellis format is GGUF-only (vLLM/SGLang’s Marlin kernel is 4/8-bit), so the vLLM lane trades some compression for fast distributed tokens — same allocation, higher floor. The agent skill (skills/pollard/SKILL.md) routes a model down the right lane automatically.

Where your builds go — the workspace

Every build lands in an organized home so you never hunt for it. Default ~/pollard (override with $POLLARD_HOME); pass --out on any tool to place a build elsewhere.

~/pollard/
  models/
    Qwen__Qwen2.5-3B/
      Qwen2.5-3B-Pollard-EXL3-4.0bpw/     runtime-ready build (HF-card-style name)
      Qwen2.5-3B-Pollard-GGUF-IQ3_KT.gguf
      calibration/    (imatrix, cal data, sensitivity.json)
      reports/        (pollard-verify / scorecard output)
      charts/         (rendered eval charts + CSVs — pollard-eval --chart)
      MANIFEST.json   (every build: lane, bpw, ppl, verified✓, size, date)
  charts/             (cross-model eval charts land here by default)
  cache/              (downloads + work dirs — safe to delete)

• pollard-ls — list everything you’ve built: lane, quant, size, PPL, and whether it passed pollard-verify. pollard-ls filters; pollard-ls --paths prints full paths.
• Running pollard-verify on a build stamps its verified flag in the manifest, so pollard-ls shows at a glance which builds are known-good.

Workflow — run these in order

The win over uniform quants is a calibration step. Skip it and you get a uniform, memory-fit build (still useful for fitting a model, but no quality edge). Run it and pollard beats uniform IQ at matched size.

# the full winning path, start to finish
pollard-calc       --gguf DeepSeek-V4-00001-of-00005.gguf --ram 128
llama-imatrix   -m model-f16.gguf -f calib.txt -o model.imatrix --chunks 30
pollard-sensitivity --gguf model-f16.gguf --imatrix model.imatrix --eval held.txt --out model.sens.json
pollard-fit        --gguf model-f16.gguf --ram 128 --imatrix model.imatrix --sensitivity model.sens.json

Models too big for one node (300B+ MoE — GLM-5.2, DeepSeek-V4, Kimi): the profiling forward pass (steps 2–3) won’t fit on one box, so pool peers over RPC. Run ggml-rpc-server on each peer, then pass --rpc host:port[,host:port…] to llama-imatrix and pollard-sensitivity. The build (pollard-fit) streams from disk and needs no RPC — it runs on a single node regardless of model size. To run the finished model across peers, pollard-run --rpc ….

Calibrating a big model on a small box (make one, not just run one): the sensitivity sweep measures against f16, which may not fit your RAM. Pass pollard-sensitivity --ram and it drops the base to the highest quant that does fit (e.g. a 24B on 16GB bases on IQ3_S) — a touch weaker than an f16 base, but it runs on your machine. The build still streams from f16, so the output isn’t compromised. (On a big box, omit --ram for the clean f16-referenced sweep.)

Shortcuts and what they cost you:

• No --sensitivity, but --imatrix → a uniform allocation (the imatrix sets IQ-type quality but does not decide the per-layer bits). There is no imatrix-magnitude proxy: magnitude misranks (it says “protect attention” when attention is half as sensitive as FFN, see notes/e13), so a magnitude-ranked build can land worse than uniform — we don’t ship that. Run pollard-sensitivity for the per-layer win.
• No --imatrix at all → uniform build; the imatrix-only IQ2 types are swapped to Q2_K so it can’t crash, and pollard-fit warns that there’s no per-layer benefit. Use this only to fit a model, not to beat a quant.
• Aggressive (IQ2) builds need the imatrix — pollard auto-pins any tensor the base preset would touch but the imatrix can’t cover (exotic tensors like DeepSeek’s compressors), so the build won’t die partway.
• --allow-1bit extends the floor from iq2_xxs down to 1-bit (iq1_m/iq1_s) for models that won’t otherwise fit. Off by default; only used where the budget forces it; needs an --imatrix; warns loudly. Heavy quality loss on small/dense models, but giant MoEs absorb it (a 753B GLM at ~1-bit stays coherent — redundancy).
• Vision / multimodal models (Qwen3.8-VL, etc.): pollard-fit builds the text model; the vision projector is a separate mmproj GGUF. Download it, don’t quantize it, and ship it alongside — run with --mmproj mmproj-….gguf to keep vision. pollard-fit reminds you when the source is multimodal.

The planner in action

Hardware profiles are just numbers — it works the same for an NVIDIA DGX Spark (--ram 128 --rambw 273), an RTX box, Apple Silicon, or a bare CPU server.

Worked example — bracketing the public “Kimi K3 on a CPU” demo from the config alone (the demo’s precision and SSD speed were unpublished; at q8→f16 assumptions the corrected floor brackets the measured 32.7 s/token):

architecture        : MOE  (896 experts/layer, top-16 + 2 shared)
total params        : 2,751.4B  -> 1,582.1 GB @ 4.6bpw
active per token    : 77.3B  -> 44.5 GB reads/cold-token
flash-stream floor  :   0.08 tok/s  (@ 3.5 GB/s sequential)
RAM-bandwidth ceil  :   2.70 tok/s  (@ 120 GB/s)
VERDICT: NEEDS BIGGER TIER — ~1,861 GB RAM for residency

The widely-shared K3-on-CPU benchmark measured 32.7 s/token (0.031 tok/s); the corrected floor band at plausible precisions (f16→q8, 23–44 s/token) brackets it. Treat calculator outputs as assumption-stated estimates, not oracles — and check them, like the community checked ours (see Errata).

Proof of concept: a 66 GB video model on a 16 GB Mac Mini

MiniMax-H3 (33B video+audio DiT, ~66 GB native) running locally on an M4 Mac Mini with 16 GB unified memory — 20-step, upscaled 1664×960 output:

Every row: same scene, same seed, A/B-comparable. Cross-architecture bonus: our FirstBlockCache skip-ratio curve on Apple Silicon reproduces NVIDIA’s GB10 curve — first cross-arch datapoint for the technique.

Not just for models that don’t fit

Local inference is bandwidth-bound: tokens/sec ≈ memory bandwidth ÷ bytes read per token. So every byte the quant mix doesn’t read is time you don’t spend — which means a measured, machine-fit build makes models that already fit faster, not just possible. Measured here: a pruned 4-bit build that was both 4 GB smaller and ~25% faster per step than the naive 3-bit it replaced. Smaller and faster are the same axis when bytes are the bottleneck.

And measured allocation buys quality, not just fit — at matched size against the standard preset, on two very different corpora:

Details and the full evidence chain, negative results included, in notes/e12-both-legs-measured.md.

For GPU users (RTX / CUDA): measured expert placement

Big MoEs on consumer GPUs run with experts offloaded to system RAM (--cpu-moe / --n-cpu-moe N) — but llama.cpp’s built-ins choose blindly (all experts, or the first N layers). pollard-run chooses by measurement: which layers’ experts actually run hot in the decode regime, from a routing profile captured during real generation. The streams stay at full checkpoint precision — placement is lossless by construction.

pollard-run --gguf Qwen3-30B-A3B-Q3_K_M.gguf --profile qwen3-30b-a3b --vram auto --launch

profiles/ ships measured heat for supported models (contribute yours — experiments/README.md shows the capture). Measured results, 16 GB Apple Silicon — and note the shape: measurement’s edge grows as memory gets scarcer (at looser budgets, blind first-N accidentally overlaps the measured split; at tight budgets, knowing wins big):

At the tight budget, measured placement won every paired run (+21% mean vs blind). Variance is real (shared machine); replication on your hardware is exactly what profiles/ wants.

Across machines (pool their RAM)

A build too big for one box runs across several — any number, not just two. llama.cpp has its own clustering (the RPC backend), so this needs no vLLM (and keeps the GGUF you built). install.sh compiles it in (-DGGML_RPC=ON + the ggml-rpc-server binary).

# on every OTHER machine (as many as you have):
ggml-rpc-server -H 0.0.0.0 -p 50052
# on the main machine — comma-separate every peer; layers split across all, RAM pooled:
llama-cli -m model-pollard.gguf --rpc host2:50052,host3:50052,host4:50052

The --rpc list takes as many peers as you add; total usable RAM is the sum across all of them, so you scale by adding boxes. It is pipeline-parallel (each machine holds a slice of the layers, activations hop between them at layer boundaries) — simpler than vLLM’s tensor-parallel and a touch slower per token, but it pools the memory, which is the point when the model doesn’t fit one box. Two DGX Sparks (128 GB each) hold a 167 GB Q4 build this way that neither could alone; add a third and a ~250 GB build comes into reach. A fast link between them (their ConnectX/QSFP, or plain 10GbE to start) carries the activations.

Every vendor works — an RTX box, a GB10 / DGX Spark, an AMD Radeon, an Intel Arc, and an Apple Silicon Mac can all join the same cluster. Each peer runs ggml-rpc-server built for its own accelerator, and install.sh auto-detects which: Metal (Apple), CUDA (NVIDIA — RTX, GB10), HIP/ROCm (AMD), SYCL (Intel), or Vulkan as a cross-vendor fallback that runs off the graphics driver alone; CPU otherwise. Force one with POLLARD_GPU=-DGGML_VULKAN=ON ./install.sh. Throughput tracks the slowest peer and the link, but the RAM adds up regardless of who made the chips.

For the vLLM/GPTQ side of clustering — models too big to even quantize on one box (a 744B is ~1.5 TB in BF16) — pollard-export --shard-plan N prints the contiguous layer band each node owns plus the boundary-handoff contract, and pollard-serve-eval A/Bs the result on the served stack. The full unified-memory runbook (memory-pressure modeling, one-GPU-job-per-node, band-parallel export, byte-accounting, gate hygiene) is in notes/unified-memory-playbook.md.

Wafer-scale (Cerebras): capacity planning

pollard-pack points Pollard’s hot-set ranking at an SRAM machine (Cerebras WSE-3 / CS-3, WSE-3 Turbo / CS-4). On a wafer, weights are 16-bit resident, so the lever is not bit-width but sparsity — the cores skip zeros. Pollard re-casts its sensitivity ranking as REAP-style expert pruning (drop the least-important experts) and forecasts the two numbers that set wafer cost:

pollard-pack --gguf your-moe.gguf --target wse3t --prune-experts 0.5
#   resident: 61.0 GB -> 33.0 GB (-46%)   WAFERS: 2 -> 1 (save 1)   [Qwen3-30B-A3B]
pollard-pack --gguf your-moe.gguf --emit-plan plan.json   # per-layer expert-drop plan

Footprint ∝ total params → wafers; throughput ∝ active params/token. It forecasts capacity, not a tokens/s number (single-stream latency on a wafer is layer-depth bound). MoE-only — a dense model is 16-bit either way and gets no wafer win, and the tool says so. This is an offline planner: Cerebras inference is a closed 16-bit stack, so a Pollard GGUF does not run on a wafer — applying the drop-list and measuring t/s is a partnership track. See notes/wafer-support.md.

Use it with your agent

The repo is written to be agent-executable: hand this README plus experiments/README.md to Claude Code, Hermes Agent, Codex, Cursor, or any coding agent, and ask it to run the calculator on a model you’re considering or to reproduce the routing measurements on one you already have.

→ Point your agent at skills/pollard/SKILL.md — the Pollard agent skill. It encodes the whole decision tree (dense → imatrix, MoE → automap), the exact commands per model type, the runtime export paths (llama.cpp / vLLM+SGLang / wafer), and the guards, so an agent runs your model down the right path for the best result — no wasted multi-hour wrong-tool runs. Everything is argparse’d, stdlib-first, and states its expected inputs. Example: “clone this repo and tell me what Kimi-K3 would do on my machine” is a complete instruction.

The method (what’s in notes/)

1. Measure the hardware floor — cache-bypassed flash curves; block size is a 20–40× lever before any ML begins (notes/e0-hardware-floor.md).
2. Compute the model’s byte-economics — total vs active params, expert pool, cold-token reads (pollard-calc automates this).
3. Find the reuse — routing concentration (MoE), step redundancy (diffusion), depth redundancy (both). This is where “impossible” becomes “viable”: the working set your workload actually keeps hot is far smaller than the file, and the difference is the RAM you don’t need to buy.
4. Fit the quant to the machine — measured per-layer sensitivity → mixed precision summed to your RAM budget. Not “Q4 because Q4 exists.”
5. Verify or it didn’t happen — seed-matched A/B for every optimization; a null test for every cache.

Experiment log: notes/ — from the dense-sparsity verdict that killed the naive version through the fitting reframe that became pollard-fit. Corrections and retractions live in notes/errata.md.

Command reference

Every tool pip install pollard-weights ships. --plan-only (or no --run) previews without building; outputs default into the workspace unless you pass --out.

Plan & orchestrate

Build — output lanes

Precondition — compose across every lane

Measure & allocate

Evaluate & verify

Runtime & workspace

Roadmap

• Routing-reuse index — the online measurement of expert-reuse locality; turns the calculator’s floor/ceiling band into a point estimate for YOUR workload. The number nobody publishes.
• Per-expert residency — today the builder allocates bits per layer and role; the next runtime step pins and streams at individual-expert granularity, with custom Metal/GPU kernels for the hot path.
• Depth-collapse — post-training layer skipping priced in expert-fetches saved (E9); a depth-exited pass doubles as a free speculative drafter.
• Cross-lane calibration tuning — widen the measured EXL3 calibration win (our Calib 3.0 already beats the stock cal) via domain-mix tuning, validated on a 2nd held-out eval, and extend the same lever to the GPTQ/MX lanes.

What this is not

• Not re-hosted weights: harnesses + method + measurements only.
• Not benchmarketing: contended runs are reported separately, approximations are labeled, and retractions stay in the log.

Acknowledgements

• llama.cpp (ggml-org) — the runtime and quantization machinery install.sh builds and pollard-fit drives.
• MiniMax — open weights for the H3 video model used in the proof-of-concept campaign, and molbal for the pruned GGUF builds its final row runs on.
• NVIDIA Sol-Engine — the FirstBlockCache technique in the campaign’s quality mode, via drowzeys’ single-GPU ComfyUI ports.
• ComfyUI — the pipeline the video campaign ran on.
• wafer-ai / gpu-perf-engineering-resources — the curated AI performance-engineering resource list we mined for Pollard improvements: the MXFP4/FP8 Blackwell output lane (OCP MX/FP8 specs, NVIDIA Transformer Engine, Blackwell tcgen05), AWQ/SmoothQuant preconditioning for the allocator, the Compute-Sanitizer + Nsight roofline correctness/perf gate, and KV-cache quantization (KIVI). Distinct from the Cerebras wafer-scale planner.
• Intellectual lineage: Apple’s LLM in a Flash (arXiv 2312.11514) for flash-resident weights on constrained devices, and P. J. Denning’s working-set theory (1968) — this project builds the model-weight-specific instruments those ideas point toward.

Contributing

Pollard is a measurement-first project — the culture is show the numbers. The best contributions come with a reproducible measurement (a KL/PPL delta, a log, a config), and the most valuable of all is someone running a lane on real hardware and sending the logs (see the Errata — several fixes came exactly that way).

• Found a broken build or a wrong number? Open an issue with the model, the command, and pollard-verify output — real reconstruction, never a proxy metric.
• Adding a capability? Prefer a detection rule over a per-model special case, and gate any new quality claim behind a measurement.
• New arch or lane? The tools are parameterized and generic (any CUDA GPU); the workspace + MANIFEST.json make results easy to share.

New here? Start with pollard-calc on a model you know, then pollard-doctor --predict on an fp16 model to see the diagnostics in action.

Is the ranking the model’s, or the corpus’s?

A sensitivity ranking is measured on a calibration set. Jurly (@jurlycat) put it plainly when Pollard was written up: the calibration set is the next bottleneck – rankings could shift with the prompts used to measure them. So the probe measures that too:

pollard-probe --model  --eval calib.txt --stability 4          # 4 contiguous slices of a multi-domain set
pollard-probe --model  --eval wiki.txt --stability-files code.txt,chat.txt   # one slice per domain

Each slice ranks the groups on its own; the profile records the Spearman agreement between slices and how much of the protect set survives from one to the next, and prints STABLE / MIXED / UNSTABLE with what to do about it. An UNSTABLE ranking is not allocated on – it is re-measured on multi-domain text, and the card says which. First measurement, Qwen2.5-0.5B-Instruct across four wikitext-2 slices: STABLE – ffn Spearman min 0.96 (mean 0.98), attn 0.92 (0.95); protect-set overlap 0.71 / 1.00. The ranking is the model’s.

Write-ups by others: Yume_X, “What Pollard actually is” (Sept 2026).

License

Apache-2.0 — see LICENSE. Built on and grateful to the open-source projects in Acknowledgements.