Local AI Coding Pipeline — RTX 5090 + Cloud GPU + Qwen3.6 + TurboQuant

A complete guide to building a fully local AI-powered coding environment that runs on your hardware — plus an optional cloud GPU server for heavier models and enterprise RAG.

What This Is

A production-ready AI coding setup spanning two machines:

Local (RTX 5090, 32GB) — Updated April 2026

• Qwen3.6-35B-A3B (primary) — MoE (3B active, 8 routed + 1 shared experts, 256 total). Hybrid Gated DeltaNet + Gated Attention architecture (30 linear + 10 full attention layers). Native 262K context, extensible to 1M. ~177 t/s generation with TurboQuant turbo3. Native tool calling + developer role support (Codex/OpenCode).
• Qwen3.5-35B-A3B (legacy) — MoE (3B active), 188 t/s generation.
• Gemma 4 31B — Dense (31B active), 61 t/s generation. Best quality/reasoning for single-shot tasks.

Cloud (RTX PRO 6000, 96GB)

• Qwen3.5-122B-A10B — MoE (10B active), Unsloth UD-Q4_K_XL (72GB) + TurboQuant turbo3 KV compression. Better benchmarks, tool use, and reasoning than the 35B variant. Unsloth’s latest GGUFs include fixed chat templates for tool-calling/coding.
• AnythingLLM — RAG knowledge base with workspaces, RBAC, and agent skills
• Mistral OCR MCP — PDF text extraction via Mistral API as an MCP tool for AnythingLLM agents
• Extract API — PDF → OCR → LLM extraction → clean structured-JSON pipeline

Both powered by llama-server with Google’s TurboQuant KV cache compression. The local RTX 5090 uses TheTom’s fork (has LLM_ARCH_QWEN3NEXT for Gated DeltaNet support — required for Qwen3.6); the cloud server uses Madreag’s fork (+13-69% faster decode at 32K on SM120 for non-hybrid models).

Tools:

• OpenCode — Agentic coding with file editing, bash, grep, and MCP tools
• Claude Code — Anthropic’s CLI (works with local models via Ollama API)
• OpenClaw — Telegram bot bridge to your local model
• Open WebUI — Chat interface in the browser
• AnythingLLM — Enterprise RAG platform (cloud server)

---

Quick Start

# One-command setup (WSL Kali, WSL Ubuntu, or native Linux)
bash scripts/setup.sh

This installs CUDA, builds llama.cpp (mainline + TurboQuant), installs OpenCode, and generates config files. See scripts/setup.sh for details.

Manual quick start:

# Start the server (pick your model)
./scripts/start-server.sh qwen  YOUR_MODEL_PATH/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf
./scripts/start-server.sh gemma YOUR_MODEL_PATH/gemma-4-31b-it-UD-Q4_K_XL.gguf
./scripts/start-server.sh turbo YOUR_MODEL_PATH/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf

# Start coding
cd your-project && opencode

---

Hardware & Models

| Component | Details |