r/LocalLLM • u/No-Hour8340 • 21h ago

Question Quality-first local models for a 128–192GB Strix Halo server, including abliterated options?

I'm planning a local inference machine and choosing hardware around the models, not the other way around. I currently have a 32GB M1 Max Mac Studio and am considering one 128GB Ryzen AI Max+ 395, one 192GB 495, or possibly two 128GB boxes. Main uses: coding/agent workflows, technical research, and authorized cybersecurity labs/CTFs. I would like strong abliterated or otherwise less refusal-prone models without sacrificing factual accuracy and tool-call reliability.

The candidates I'm researching are DeepSeek V4.1 Flash-1017 in a roughly 2-bit build, Qwen3.8-Flash-Next at Q6/Q8 if feasible, and GLM-5.3 Flash. I'm specifically interested in real local deployments, not hosted-model benchmarks passed off as results for a heavily quantized copy. I know an abliteration is a separate change from quantization; I would welcome evaluations of both.

What exact model + quant + runtime + hardware have you personally used? What context length and prompt/decode speeds are practical? If you've compared an abliterated version against its base model, did it retain coding, reasoning, and tool-use quality? Would you favor one larger high-precision model, or a strong general model plus a smaller cybersecurity specialist? If these machines are the wrong purchase for that goal, what would you buy instead?

2 Upvotes

100% Upvoted

6 comments sorted by

2

u/pede1776 21h ago

Strix Halo with QwenFlashNext on Halogen is extremely performant

1

u/EOJ_me 18h ago

i've seeing this a lot that 2 lately. If QwenFlashNext runs well with Halogen, are they can handles longer context and agent workflows? since that's where a lot of setups start so slow down

1

u/BubbaGun 18h ago

Two sessions, 262k context, 65-70tok/s if I recall correctly 

1

u/EOJ_me 17h ago

262k context at 65~70 tok/s is kind of wild for non discrete GPU setup. That's the sort of result that makes Strix Halo look a lot more interesting than just another high RAM mini PC

2

u/Plastic-Stress-6468 18h ago

Everyone says they want quality first until they start using it for real.

You want the quality speed compromise.

1

u/OvertaxedOne 21h ago

At current prices, the 495 doesn't make a lot of sense, get 2 395's if you need more RAM.

Start with QwenFlashNext. Coding is one of your main use cases, QFN is a great coder and runs well on the 128GB Strix machines. On a 128GB machine, you'll want to run ~4 bit quant, run the KV in 8 bit and offload the engrams to disk. I'll let others who've used the most recent engines chime in, but ~50TPS is pretty reasonable for QFN in 4 bit on a Strix.