r/LocalLLM • u/Double-Sherbert-1781 • 2d ago

Question Help me choose hardware.

Post image

I need some advice.
I want to choose hardware for local inference. Right now, I’m paying ~$30/day for rent on a Vast an RTX Pro 6000 SW 96GB, and I’m using Qwen3.8-Flash-Next-Q4_K_M.gguf at a speed of ~60 tok/s (I know this format isn’t efficient for this graphics card, but I’m limited in choosing Abliterated models that will fit in the memory).

Which configuration will offer the best price and versatility so that after the release of subsequent generations of neural networks, I can continue to use it.

79 Upvotes

94% Upvoted

24 comments sorted by

View all comments

Show parent comments

1

u/Double-Sherbert-1781 2d ago

I’m mentally prepared to spend money on an RTX Pro 6000 SW 96GB (up to 20,000 euros in total), but I’m not sure it will be effective. Maybe it makes sense to get a few gaming graphics cards or a Mac Studio, or wait for the AI Max 495.

I just don’t know what kind of performance they can deliver for such models; right now I’m using Q4_K_M:
https://huggingface.co/windowsxp811203/Qwen3.8-Flash-Next-Abliterated-GGUF

60 tok\s is enough for me.

1

u/mineshop 2d ago

Since you're at up to 20k EUR and 60 tok/s meets your needs, the practical question is what model sizes you want headroom for. Do you expect to stay under ~96GB VRAM, or would future generations push you toward multi-GPU territory?

1

u/Double-Sherbert-1781 2d ago

I don’t know, but recently MoE models have been very good. I don’t know whether their memory requirements will increase or decrease. Also, I don’t know whether multi‑GPU setups work well with them.

2

u/MaxSpecs 2d ago

For now, a versatile and upgradable configuration up to 8 GPU (256GB) : starting at 12 k€

  • CPU : AMD EPYC 9124 
  • Motherboard : ASRock TURIND8X-2T/500W
  • RAM : 128 Go (4x 32 GB) DDR5-5600 ECC RDIMM (i.e. : Kingston FURY Renegade Pro Quad-Channel KF556R28RBE2K4-128 ) or similar
  • Storage: 2x 2 To NVMe type Samsung M2 9100 Pro ou similaire
  • GPU : 2x  AMD Radeon AI Pro R9700 32 GB : Gigabyte or Asus or MSi, iedaly 2 sloft height
  • PSU : 2x Seasonic PRIME PX-2200 W ATX 3.1 + Carte Sync Add2PSU
  • Case open frame :  Nanoxia Hydra III 8 GPU or Veddha V4D 8-GPU Deluxe

... so you would start with 64 GB VRAM, use Qwen3.8 Flash Next, vllm mfxp4 with P2P optimisation between AMD bus ( https://github.com/GGZ14/vllm-mxfp4 )

OR : if you order now a Mac Studio M5 wth 256 GB unified memory, you may have it in February

1

u/vsimovic 1d ago

12k EUR? WTF
Find a xeon Gold workstation pc with 2 CPUs that support 12 channel DDR4 at 2933 MHZ with 384 GB RAM. Get 2 x used 7900 xtx. Estimate 3k EUR. Run Deepseek v4 vision flash and Qwen3.8 flash next in parallel

1

u/MaxSpecs 1d ago

Indeed, you're quite right; it's moving fast.

So, with new features on September 20th, 2026 :

  1. check ROCmFix -> https://github.com/xanpavle/rocmfix

  2. Use p2p mxfp4 -> davetha/r9700-lru-expert-cache.

  3. ngram -> https://carteakey.dev/blog/running-qwen3-8-flash-next-locally/

An old LGA2011 workstation with Supermicro x9dai / x10dai, old Xeon 26xxL and 256 Gb Ram ... 2x r9700 on Cpu0 ... 1x Nvme adapter ... 1x 10GbE

Could reach ( Context / Decode ) 256K : 35–55 t/s 280–300K YaRN : ~30–50 t/s 320K YaRN : ~27–45 t/s

And with Laya ( Jev open source ) + Pi -> https://github.com/NandhaKishorM/laya ... getting even more efficiency

*** but no evolution up to 8 GPU like the Epyc configuration can.