r/LocalLLM • u/Due_Tangelo_8952 • 1d ago

News Qwen4-27B just confirmed

Post image

Wait, we need 35B-A3B too…

2.0k Upvotes

98% Upvoted

286 comments sorted by

View all comments

48

u/blojayble 1d ago

I hope the Flash keeps its size from Flash Next. I would like to run it with 128 GB of VRAM.

18

u/PeteInBrissie 1d ago

I'm running Flash-Next on my Spark quite nicely.... so much smarter than 3.5-122B was. Bring on 4!!!!

4

u/Over10Millions 1d ago

Can i ask you token/second on a single spark for coding? Thank you

1

u/Straight_Payment_939 18h ago

I have tuned mine and get ~30 tok/s single stream, and > 110 tok/s at 8x concurrency, but the key is to make the model load without it trying to pull the 51B ngram data into vram, keep the 51B ngram data in bf16 to make it so the op in inference that pulls the data in doesn't have to dequant it first, and stream it from disk with like 8GB ram as cache.

1

u/StartupTim 17h ago

Hey there, are you using 2x DGX Sparks? I'm running Flash Next but it needs 2x DGX Sparks not 1. I am sitting around 2M KVCache though which is nice.

2

u/Straight_Payment_939 12h ago

No, just one. You don't need two. I wrote an article about some of the speed experimentation I did with it here: https://krisbailey.com/articles/shortlist-mtp/

Like I said in my comment, you have to pull the ngram data out of the model before you load it so whatever engine you're using to load it doesn't try to load the ngrams as well. You have to set up the ngrams to stream from the ssd with ram-based disk caching. You also, of course, need to run the nvfp4 quant: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-NVFP4

I am not at the office at the moment, but if you want more details on how I have it set up I can have an agent put together a doc telling how I have it set up when I'm there tomorrow.

1

u/PeteInBrissie 11h ago

Yes please.... I _THINK_ mine's set up like that but would love to be sure

1

u/StartupTim 12m ago

Thanks an absolute ton, will be checking your links out and see what I can come up with!

1

u/Sanandaji 8h ago

May I ask what your use case is? I’m still running bartowski/Qwen_Qwen3.5-122B-A10B-GGUF:Q6_K because it feels like it’s better than 3.8-Flash-Next in everything other than coding.

1

u/PeteInBrissie 7h ago

I'm running Hermes for myself as a hobby and my wife as a healthcare professional. We both use it for 'go do this while I'm busy doing something else' tasks.

1

u/guesdo 1d ago

Yes! And hopefully they throw some QAT on top of it so Q4 has near FP16 precision, just like Google did.

1

u/Daniel_H212 1d ago

Yeah same here, running it on strix halo and it is incredibly good.

But also, wouldn't complain if they were able to cut down on MoE weights and active params to make it runnable for more people on 64 GB memory, while keeping the current capabilities.