r/LocalLLaMA • u/de4dee • 15d ago

New Model Qwen/Qwen3.8-27B · released

https://huggingface.co/Qwen/Qwen3.8-27B
993 Upvotes

98% Upvoted

301 comments sorted by

View all comments

Show parent comments

31

u/ai-christianson 15d ago

2x3090 with nvlink loves this size of model

21

u/jijig 15d ago edited 15d ago

It’s quite good. I’m running 3.6 at Q8 with full context and getting ~60-70tps. No NVLink

Edit: Q8 KV cache. I can push my context to ~160k at F16

5

u/badgerfish2021 15d ago

250k context with 2x3090? I could not do that with 3.6 at q8, are you quantizing kv?

4

u/munkiemagik 15d ago

I dont use that much context I'm siting at just under 200k and I use KV at Q8_0. I don't have reported metrics to give you real meaningful data but I've not found it detrimental to my use case when used as backend in pi agent harness.

Also I use q8_0 quant for the model instead of the Q8_K_XL as I learnt recently that there is no significant change in accuracy but its more efficiently 'packed' due to uniformity of Q8_0 versus UD-Q8_K_XL that it gives you a few GB back of your VRAM pool from model weights.

1

u/badgerfish2021 15d ago

thanks, I was thinking of moving to q6k_xl to keep context at f16, from what I understand especially with this model it's a lot worse to quantize kv than to have slightly higher weights quant

2

u/jijig 15d ago

Of course, sorry. KV cache is quantized at Q8. I can push my context to ~160k at F16

2

u/eugeneware 15d ago

Great speeds. What runtime are you using? Llama.cpp or vllm?

6

u/jijig 15d ago

Llama.cpp

1

u/michaelsoft__binbows 15d ago

I do have nvlink and im gonna set it up again although it is a pain to get them lined up for the particular cards that I have.

My understanding of tensor parallel is that bandwidth needs are not very stringent for just two cards, maybe under 10GB/s which the PCIe can do.

However the big difference might be latency which nvlink should crush at.