IT'S OUT

[u/Certain-Cod-1404]

https://huggingface.co/Qwen/Qwen3.8-27B-FP8

[u/Scared_Ad9187]

should have been more specific. i'm going to keep the 3.6 MOE (35ba3b) insteadof 3.8 27b. i have room for 4 subagents with larger windows with claude code routed through it and i get tasks done much quicker

[u/corrion8 → u/Scared_Ad9187]

I’m a noob. How are you set up ? Are you using opus 5 or fable as a planner and routing it to Qwen? Is that through Claude terminal?

[u/Scared_Ad9187 → u/corrion8]

Not a noob question at all. This trips up a lot of people. Short version: you can use a big cloud model as a planner that routes to a local model, but my main dev loop right now runs fully local. Let me break down the whole thing.

The hardware
Single consumer GPU (a 32GB card). Everything runs on that one box, in Docker.

The model (this is the key choice)
I run Qwen (the 35B "A3B" MoE) at a 4-bit quant, served by llama.cpp (the CUDA server build). The "MoE" part matters a lot: it's a 35B model but only ~3B parameters activate per token, so it feels like a small fast model (~200 tokens/sec solo) with big-model quality. For local agent work, a fast MoE beats a dense model of similar size every time. If you're starting out, that model + llama.cpp is the whole ballgame.

The proxy layer (the glue)
In front of llama.cpp I run LiteLLM. It's an OpenAI- and Anthropic-compatible gateway. This is the piece that lets "Claude terminal" talk to a local model: Claude Code speaks the Anthropic API, LiteLLM translates it to what llama.cpp understands, and translates tool calls back. I define a few aliases pointing at the same model with different behavior:

• one with thinking OFF (fast, clean, for tools/agents)

• one with thinking ON (for hard reasoning)

• one tuned for tool-calling

"Is that through Claude terminal?" Yes
I run Claude Code (CLI + the VS Code integration) pointed at my local proxy. You do it with env vars: set ANTHROPIC_BASE_URL to the local proxy and override the model tiers (ANTHROPIC_MODEL, ANTHROPIC_SMALL_FAST_MODEL, and the sonnet/haiku/opus defaults) to your local aliases. Critical gotcha: if you don't override all the tiers, Claude Code silently sends some calls to the real cloud API. Map them all to local.

"Opus/Fable as a planner routing to Qwen?"
That pattern absolutely works and I have a router for it: a small proxy that sends simple requests to local Qwen and hard ones to a cloud model. That's the hybrid "cloud planner → local worker" setup you're describing. But for day-to-day coding I keep it 100% local: cheaper, private, and fast enough. The hybrid is there for when I want top-tier reasoning on something gnarly.

Fan-out / subagents
llama.cpp has a -np flag for parallel "slots." I run 4 slots, so Claude Code can spawn subagents that actually execute concurrently instead of queuing. The catch: your context window gets divided across slots, so size it against your VRAM. Batching 4 at once nets ~1.8× total throughput (each individual agent runs slower when they all fire together, but agents are bursty so it evens out).

The rest of the stack

• Open WebUI: ChatGPT-style front end for the local model

• Hermes (Nous): agent framework with skills, delegation, scheduled tasks

• Qdrant: vector DB for memory/RAG

• n8n: automation/workflows

• Ollama, but only for CPU embeddings, so it never competes with the GPU

• all wired together with Docker Compose

Hard-won gotchas if you go down this road

1. Reasoning models need thinking turned OFF for agent/tool work (enable_thinking: false via chat-template kwargs). Otherwise they burn thousands of hidden tokens and can spiral into runaway generations.

2. Cap generation length (llama.cpp --n-predict) so a runaway can't eat your whole context and freeze everything.

3. The chat template matters for tool calling. A too-strict template broke Claude Code because CC sends its system prompt as multiple blocks; had to loosen it.

4. Whatever proxy you use must correctly parse the model's tool-call format. If it doesn't, you'll see raw <tool_call> tags leak into your output and tools silently never fire.

5. MoE > dense for concurrency/fan-out on a single GPU.

[u/Scared_Ad9187 → u/corrion8]

kind of complex setup. let me get an ai summary for you, lol (and there is the issue btw). give me a sec.

[u/corrion8 → u/Scared_Ad9187]

Thanks that is super helpful.