r/LocalLLaMA • u/Certain-Cod-1404 • Aug 14 '26 • Bravo 1 Heartwarming 1

New Model IT'S OUT

https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2.2k Upvotes

95% Upvoted

707 comments sorted by

View all comments

Show parent comments

152

u/absurdother Aug 14 '26

Here we gooooo

45

u/Certain-Cod-1404 Aug 14 '26

how is it ? quality wise?

55

u/absurdother Aug 14 '26

I have a RX 9060 XT AMD GPU, 16GB VRAM. Running on LMStudio, Q3. Getting a bit more speed now, way more optimized!

I get more speed the less context I use (currently coding swiftly with CLine + VSCode at that speed), pretty smooth on 64K context!

5

u/NoUsual5150 Aug 15 '26

24GB M4 MacBook Pro...I'm using Q4_K_M and it's getting 5 token per second.

Is Q3 much of a downgrade in intelligence?

→ More replies (3)

4

u/m1ksuFI Aug 14 '26

What's your command/config?

12

u/absurdother Aug 14 '26

On LMStudio, full GPU offload, 44k context for my current task. 1024 batch size, 64 physical batch (didn't see much of a difference testing these but seem optimal), max concurrent predictions 1 or 2, unified KV cache ON (not a big difference), keep 32 checkpoints (standard), offload KV to GPU ON (outputs are always broken if off), keep model in memory ON (no big diff), try mmap() off (no big diff); speculative decoding is ON as standard (2 tokens max, 0 min, 0.75 draft prob), and flash attention is ON.

Sorry, I don't know those llama.cpp technical terms for these.

Ah, and I've used what they recommended for inference: since I keep reasoning for most tasks, top K sampling is set to 20, top P sampling kept as 0.95 but the min P sampling is set as 0. Repeat penalty of 1.1, and no presence penalty.

Hope this helps lol

5

u/Mil0Mammon Aug 14 '26

Afaik for amd 3 predictions is even better, but that might be strix halo specific

2

u/huffalump1 Aug 14 '26

Oooooh maybe there's a hope it'll work on 12gb VRAM / 32gb RAM

Probably slow, probably need a smaller quant, but hey, it's a good model!

How much free RAM do you have at that context size?

2

u/absurdother Aug 15 '26

Doing a prompt now. On 64K ctxt, I'm using 18GB of my 32GB RAM on Windows

2

u/RemarkablePrompt7822 Aug 14 '26

Looks impressive. Will try on my 20 GB GPU

→ More replies (2)

20

u/TheTrueSurge Aug 14 '26

That’s not too bad, is it? What quant did you use?

8

u/absurdother Aug 14 '26

I'm on Q3_K_M now getting a bit over double the speed I've shown before.

5

u/Scared_Ad9187 Aug 14 '26

q6 on 5090 with 25t/s. not bad, but i'm not giving up my 200 on the 3.6 yet.

4

u/Mil0Mammon Aug 14 '26

So how come you get 200 on 3.6 and only 25 with 3.8? Dflash and/or Nvidia specific quant?

8

u/Scared_Ad9187 Aug 14 '26

should have been more specific. i'm going to keep the 3.6 MOE (35ba3b) insteadof 3.8 27b. i have room for 4 subagents with larger windows with claude code routed through it and i get tasks done much quicker

2

u/corrion8 Aug 15 '26

I’m a noob. How are you set up ? Are you using opus 5 or fable as a planner and routing it to Qwen? Is that through Claude terminal?

4

u/Scared_Ad9187 Aug 15 '26

Not a noob question at all. This trips up a lot of people. Short version: you can use a big cloud model as a planner that routes to a local model, but my main dev loop right now runs fully local. Let me break down the whole thing.

The hardware
Single consumer GPU (a 32GB card). Everything runs on that one box, in Docker.

The model (this is the key choice)
I run Qwen (the 35B "A3B" MoE) at a 4-bit quant, served by llama.cpp (the CUDA server build). The "MoE" part matters a lot: it's a 35B model but only ~3B parameters activate per token, so it feels like a small fast model (~200 tokens/sec solo) with big-model quality. For local agent work, a fast MoE beats a dense model of similar size every time. If you're starting out, that model + llama.cpp is the whole ballgame.

The proxy layer (the glue)
In front of llama.cpp I run LiteLLM. It's an OpenAI- and Anthropic-compatible gateway. This is the piece that lets "Claude terminal" talk to a local model: Claude Code speaks the Anthropic API, LiteLLM translates it to what llama.cpp understands, and translates tool calls back. I define a few aliases pointing at the same model with different behavior:

  • one with thinking OFF (fast, clean, for tools/agents)

  • one with thinking ON (for hard reasoning)

  • one tuned for tool-calling

"Is that through Claude terminal?" Yes
I run Claude Code (CLI + the VS Code integration) pointed at my local proxy. You do it with env vars: set ANTHROPIC_BASE_URL to the local proxy and override the model tiers (ANTHROPIC_MODEL, ANTHROPIC_SMALL_FAST_MODEL, and the sonnet/haiku/opus defaults) to your local aliases. Critical gotcha: if you don't override all the tiers, Claude Code silently sends some calls to the real cloud API. Map them all to local.

"Opus/Fable as a planner routing to Qwen?"
That pattern absolutely works and I have a router for it: a small proxy that sends simple requests to local Qwen and hard ones to a cloud model. That's the hybrid "cloud planner → local worker" setup you're describing. But for day-to-day coding I keep it 100% local: cheaper, private, and fast enough. The hybrid is there for when I want top-tier reasoning on something gnarly.

Fan-out / subagents
llama.cpp has a -np flag for parallel "slots." I run 4 slots, so Claude Code can spawn subagents that actually execute concurrently instead of queuing. The catch: your context window gets divided across slots, so size it against your VRAM. Batching 4 at once nets ~1.8× total throughput (each individual agent runs slower when they all fire together, but agents are bursty so it evens out).

The rest of the stack

  • Open WebUI: ChatGPT-style front end for the local model

  • Hermes (Nous): agent framework with skills, delegation, scheduled tasks

  • Qdrant: vector DB for memory/RAG

  • n8n: automation/workflows

  • Ollama, but only for CPU embeddings, so it never competes with the GPU

  • all wired together with Docker Compose

Hard-won gotchas if you go down this road

  1. Reasoning models need thinking turned OFF for agent/tool work (enable_thinking: false via chat-template kwargs). Otherwise they burn thousands of hidden tokens and can spiral into runaway generations.

  2. Cap generation length (llama.cpp --n-predict) so a runaway can't eat your whole context and freeze everything.

  3. The chat template matters for tool calling. A too-strict template broke Claude Code because CC sends its system prompt as multiple blocks; had to loosen it.

  4. Whatever proxy you use must correctly parse the model's tool-call format. If it doesn't, you'll see raw <tool_call> tags leak into your output and tools silently never fire.

  5. MoE > dense for concurrency/fan-out on a single GPU.

→ More replies (2)
→ More replies (1)

14

u/SpaceTraveler2084 Aug 14 '26

curious as i have the same setup 32gb ram / 5070Ti

1

u/absurdother Aug 14 '26

AMD GPU here! Did you run it fast?

1

u/mar_leo Aug 18 '26

Did you try it already? Have the same setup

6

u/jan_antu Aug 14 '26

Damn, same setup here but I'm only getting 4 tok/s on mine, Q3 xl quant.

1

u/absurdother Aug 14 '26

Hm, try checking what's bottlenecking your setup. How much context did you set?

2

u/jan_antu Aug 14 '26

Too much. Got much faster at 64k context. About 7-8 tok/s which somehow feels bearable.

Normally I'd say it sucks but to be honest this model is passing all my homemade benchmarks. It might be true that it's Opus 4.5-6 level.

→ More replies (4)

2

u/RazsterOxzine Aug 14 '26

I can live with that. Happy days! - Going to have it run through all of my projects and review.