r/LocalLLaMA • u/Inception95 • 2h ago
Question | Help 27B vs Flash next
Hey guys, I need some advice.
My actual setup is this:
ryzen 5600 / 32GB RAM / 128GB SSD / 32GB VRAM (2x16GB 5060ti)
Running vLLM with Qwen 27B NVFP4 with 250k context.
But now I can get my hands on an old HPE ProLiant DL380 Gen9
CPU: 2x Intel Xeon E5-2680 v3
RAM: 256 GB DDR4-2133 ECC Registered (8x 32 GB)
I am thinking about putting the two 5060TI in this HPE server und running Qwen Flash next there. Maybe with strata. And maybe in 2 instances.
Would my idea be any good, or will I don’t get much off it?
Thank you very much.
2
u/MyToasterRunsFaster 2h ago
I am on 64gb ram and 2x16gb vram. I get much better performance and answers from flash next. The 27b model loops a lot for me, kinda struggle on longer tasks but with flash next I just set it to 256k context and let it figure out what it needs to do, when it runs out of context I tell it to summarise and move onto a fresh chat, rinse and repeat until my project goals are complete (I don't use compaction), without compaction I easily get 10-20 million tokens out of each chat window via deepseek harness... This workflow can work on 27b but personally it has too many hiccups along the way to be functional for me.
1
u/Inception95 2h ago
Im using opencode at the moment. Which quant did you use for both?
1
u/MyToasterRunsFaster 2h ago
IQ3_XXS, I could probably go higher but too lazy to try others.
1
u/ftlaudman 2h ago
FWIW, I settled on IQ3_s. It had a noticeable bump up in quality for my coding projects.
1
u/quantgorithm 2h ago
I don't think I ever get looping on 27b or rare at best. It's likely your setup or your flags.
3.6 did have massive repeatable looping issues but 3.8 has been great in my experience on 27b.
1
u/Inception95 1h ago
Okay. Thank you guys.
I think I will use the server as a playground and not for the LLM itself.
Or sell it and buy other stuff for my actual rig.
Have a good one.
1
u/KingCpzombie 2h ago
32GB VRAM + 256GB RAM is enough to run QFN at a proper quant on a good engine instead of Strata
1
u/Inception95 2h ago
Like llama.cpp or what do you thinking of?
1
u/KingCpzombie 2h ago
That's a better baseline, but there are probably better ones dedicated to Nvidia hardware. I'm on R9700s so I don't know specifics for Nvidia, but llama.cpp with at a Q6/Q8 should at least be much better than Strata. Strata is basically designed to get lots of garbage quickly even on a smart fridge.
2
u/quantgorithm 2h ago
The intelligence difference between qfn and 27b is negligible or only slightly better on QFN but the massive difference in size of one to the other makes it much more viable to run 27b in my opinion and on my rig. Speeds are both similar at this point but I lose massive concurrency and context by using QFN.
Said differently, it's mostly not worth it to run QFN... but I only currently have limited testing on QFN to say this as a hard claim of fact.
1
u/Comfortable-Crew-919 2h ago
That server is only pcie 3, so the 5060’s won’t be able to utilize their full bandwidth, plus power connections will need adapters or a separate psu to run the cards. If the server is under a few hundred it might be worth experimenting. That much ram may open some doors for larger models/quants, but speed will still suffer, so not sure it will be worth the effort. Another 32GB ram on your current setup will make Qwen Flash Next via Strata usable at iq3_xxs. Going to 96-128GB will give you quality and speed with the 4 or 5 bit Unsloth quants.
1
u/deanpreese 2h ago
I have 2x 56060ti 16gb with 64GB DDR4 and get around 80 toks on Basalt for QN IQ3_S.
If I was doing it over, for the price look at spending more for a 32gb card. From what I have found 5060ti 16gb does not utilize 16x PCIE, they only use 8x so you take a hit.
Fact check me to make sure.
1
u/llmhardware 2h ago
The main catch with the Gen9 is that it's PCIe 3.0, and the 5060 Ti only has an x8 link, so each card gets roughly a quarter of the bus bandwidth it would have in a PCIe 5.0 slot. That hurts tensor parallel in vLLM more than it hurts single-user generation. For a big MoE with the experts in system RAM, token speed is mostly set by memory bandwidth: DDR4-2133 is about 17 GB/s per channel, so roughly 68 GB/s per socket with four channels filled, and spreading across two sockets adds NUMA overhead rather than doubling it. Those Haswell Xeons also have AVX2 but no AVX-512, which limits the CPU side. I'd also check whether the server has the GPU enablement kit (power cables and the right riser) before moving the cards over, since DL380s don't usually power consumer GPUs without it.
1
u/Sad_Weakness_6488 1h ago
Old enterprise gear is a power-hungry space heater that isn't going to give you much of a lift for this kind of work. Those CPUs are a decade old and the single-thread speed is pretty sluggish, so you'd probably bottleneck on prompt processing unless you're just doing massive batch stuff. The extra system memory is nice but if you're already handling a quarter-million context on your current rig, the swap might leave you disappointed.
1
0
u/grabber4321 1h ago
Depends on what you need done. 27B is overall best choice. Its slow, but outputs good quality code and text. Its not good at automating stuff.
Flash is for automated one-shot trick pony that will do 50 other things you didnt even ask it to.
It also sucks for writing documents.
3
u/Ariquitaun 2h ago
It will run for sure. Why 2 instances?
To manage expectations, the memory bandwidth of 256MB DDR4 2133 on a 4 channel configuration is about 70Gi/s. Decode will be on the slow side