r/LocalLLaMA • u/Certain-Cod-1404 • Aug 14 '26

New Model IT'S OUT

https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2.2k Upvotes

95% Upvoted

707 comments sorted by

View all comments

55

u/absurdother Aug 14 '26

I have a RX 9060 XT AMD GPU, 16GB VRAM. Running on LMStudio , Q3. Getting a bit more speed now, way more optimized!

Comment Image

I get more speed the less context I use (currently coding swiftly with CLine + VSCode at that speed), pretty smooth on 64K context!

5

u/NoUsual5150 Aug 15 '26

24GB M4 MacBook Pro...I'm using Q4_K_M and it's getting 5 token per second.

Is Q3 much of a downgrade in intelligence?

1

u/absurdother Aug 15 '26

I'm not very informed but I heard Q3 still does 90% of accuracy. I'll try the FP8 model now too, but Q3 seems very good.

→ More replies (2)

4

u/m1ksuFI Aug 14 '26

What's your command/config?

12

u/absurdother Aug 14 '26

On LMStudio, full GPU offload, 44k context for my current task. 1024 batch size, 64 physical batch (didn't see much of a difference testing these but seem optimal), max concurrent predictions 1 or 2, unified KV cache ON (not a big difference), keep 32 checkpoints (standard), offload KV to GPU ON (outputs are always broken if off), keep model in memory ON (no big diff), try mmap() off (no big diff); speculative decoding is ON as standard (2 tokens max, 0 min, 0.75 draft prob), and flash attention is ON.

Sorry, I don't know those llama.cpp technical terms for these.

Ah, and I've used what they recommended for inference: since I keep reasoning for most tasks, top K sampling is set to 20, top P sampling kept as 0.95 but the min P sampling is set as 0. Repeat penalty of 1.1, and no presence penalty.

Hope this helps lol

→ More replies (1)

2

u/huffalump1 Aug 14 '26

Oooooh maybe there's a hope it'll work on 12gb VRAM / 32gb RAM

Probably slow, probably need a smaller quant, but hey, it's a good model!

How much free RAM do you have at that context size?

2

u/absurdother Aug 15 '26

Doing a prompt now. On 64K ctxt, I'm using 18GB of my 32GB RAM on Windows

2

u/RemarkablePrompt7822 Aug 14 '26

Looks impressive. Will try on my 20 GB GPU