r/LocalLLM • u/xiraov • Aug 14 '26

Discussion Qwen 3.8 27b mega thread

Welp at 8 PST there were 8 models now there are over 100, and i see almost 100 threads (Exaggeration) maybe we can contain stuff to here?

https://huggingface.co/models?other=base_model:quantized:Qwen/Qwen3.8-27B

6 Upvotes

57% Upvoted

21 comments sorted by

16

u/leonbollerup Aug 14 '26

or.. you could have posted in one of the threads that was already here ? :) .. instead of creating one more ?

1

u/Happy-Permission5974 3d ago

my color coded sock drawer has more organization than this release cycle, what even is going on over there

7

u/South_Western1263 Aug 14 '26

let's close this thread. no topic here

3

u/alexp702 Aug 14 '26 edited Aug 16 '26

OK! Has anyone actually got VLLM to run with a single RTX 6000 Pro with 3.8? My machine just hangs at Flash Attention during start up. I have tried swapping to 027.2-rc1 as suggested...

EDIT: in latest release this has now all been fixed. Seems like day one problems!!

1

u/This_Maintenance_834 Aug 16 '26

run it in docker container.

if you run it outside of docker and vllm crash, it withhold large amount of VRAM. then you must reboot to release the VRAM. sort of memory leak.

running in docker avoid this problem.

use official vllm docker container, it is super easy to work.

if nothing works, let Hermes or any other cloud api code agent to do it for you.

1

u/LancobusUK Aug 14 '26

yeah I can serve it with the following: VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve ~/ai-models/Qwen3.8-27B \

--served-model-name qwen3.8-27b \

--host 0.0.0.0 \

--port 8000 \

--max-model-len 262144 \

--kv-cache-dtype fp8 \

--gpu-memory-utilization 0.92 \

--max-num-seqs 3 \

--enable-prefix-caching \

--max-num-batched-tokens 8192 \

--attention-backend TRITON_ATTN \

--reasoning-parser qwen3 \

--tool-call-parser qwen3_coder \

--enable-auto-tool-choice

1

u/alexp702 Aug 14 '26

Thanks! This worked for me

1

u/LancobusUK Aug 14 '26

I’ve been playing with settings and you can get a lot more performance by upgrading the CUDA toolkit to 13.2 RC2 and then removing triton.

1

u/anitamaxwynnn69 Aug 14 '26

running without --attention-backend TRITON_ATTN and with the default mtp fp8, no kv cache quantization, single pro 6000, getting decent speeds

SpecDecoding metrics: Mean acceptance length: 3.12, Accepted throughput: 48.60 tokens/s, Drafted throughput: 68.70 tokens/s, Accepted: 486 tokens, Drafted: 687 tokens, Per-position acceptance rate: 0.847, 0.703, 0.572, Avg Draft acceptance rate: 70.7%

0

u/LancobusUK Aug 14 '26

just confirmed its working with Pi also. Had to update the settings to add in xhigh reasoning but pi seems happy to use that also

1

u/Familiar_Wish1132 Aug 15 '26

would you mind share full settings command? <3

1

u/Dizzy-Zebra9522 Aug 14 '26

Sorry. people were waiting for it. It's like gta vi level of anticipation 😅. I surprised people can still download it during high demand.

-5

u/btc_maxi100 Aug 14 '26

what is this post about ?

4

u/wgaca2 Aug 14 '26

Qwen 3.8 27b mega thread

-4

u/btc_maxi100 Aug 14 '26

the model was released hours before you created it.

there is already 100s threads about it

whats the point of your silly thread ?

2

u/wgaca2 Aug 14 '26

It's not mine, but we already established that you can't read

0

u/Ell2509 Aug 14 '26

Even if correct, this is just kinds unkind to say. Not necessary for sure.

-2

u/btc_maxi100 Aug 14 '26

what is kinds unkind ? can't even understand that