Local AI Coding Pipeline — RTX 5090 + Cloud GPU + Qwen3.6 + TurboQuant

A complete guide to building a local-first AI-powered coding environment that runs on your hardware — plus an optional cloud GPU server for heavier models and enterprise RAG.

What This Is

An AI coding setup spanning two machines:

Local (RTX 5090, 32GB) — Updated April 2026

• Qwen3.6-35B-A3B (primary) — MoE (3B active, 8 routed + 1 shared experts, 256 total). Hybrid Gated DeltaNet + Gated Attention architecture (30 linear + 10 full attention layers). Native 262K context, extensible to 1M. ~177 t/s generation (TheTom fork, turbo3 KV, TURBO_LAYER_ADAPTIVE=2, 262K ctx, one slot). Native tool calling + developer role support (Codex/OpenCode).
• Qwen3.5-35B-A3B (legacy) — MoE (3B active), 188 t/s generation (mainline llama.cpp, f16 KV, 131K ctx).
• Gemma 4 31B — Dense (31B active), 61 t/s generation (mainline llama.cpp, f16 KV, 131K ctx). Best quality/reasoning for single-shot tasks.

Cloud (optional, RTX PRO 6000, 96GB)

• Qwen3.5-122B-A10B — MoE (10B active), Unsloth UD-Q4_K_XL (72GB) + TurboQuant turbo3 KV compression. Better benchmarks, tool use, and reasoning than the 35B variant. Unsloth’s latest GGUFs include fixed chat templates for tool-calling/coding.
• AnythingLLM — RAG knowledge base with workspaces, RBAC, and agent skills
• Mistral OCR MCP — PDF text extraction via Mistral API as an MCP tool for AnythingLLM agents
• Extract API — PDF → OCR → LLM extraction → clean structured-JSON pipeline

The cloud pieces are optional and not local to your desk: the GPU server itself, and the Mistral OCR step, which calls Mistral’s hosted API. The default OpenCode config also enables Context7, a hosted documentation-search MCP server that needs an API key (remove it from configs/opencode.json to drop that dependency). The Mistral OCR MCP server and Extract API source code are not included in this repo; the cloud section below describes how they work.

Both powered by llama-server with Google’s TurboQuant KV cache compression. The local RTX 5090 uses TheTom’s fork (has LLM_ARCH_QWEN3NEXT for Gated DeltaNet support — required for Qwen3.6); the cloud server uses Madreag’s fork (+13-69% faster decode at 32K on SM120 for non-hybrid models).

Tools:

• OpenCode — Agentic coding with file editing, bash, grep, and MCP tools
• Claude Code — Anthropic’s CLI (works with local models via the claude-qwen function, which points at llama-server)
• OpenClaw — Telegram bot bridge to your local model
• Open WebUI — Chat interface in the browser
• AnythingLLM — Enterprise RAG platform (cloud server)

---

Quick Start