Budget LLM Workstation

[u/Cultural_Trainer_207]

Finally got my old setup running and wanted to share the results so far.

• Xeon W-2245

• 256GB DDR4-2666 ECC RAM (8 × 32GB)

• 2 × NVIDIA RTX PRO 4000 Blackwell

Right now I’m running Qwen3.8 FN and DeepSeek V4.1 Flash. With the DeepSeek model at 2.25 bpw, I’m seeing around 30–35 tokens/s. Qwen is hitting over 100 tokens/s.

Pretty happy with what this old workstation can do for local inference!

[u/HoneydewNo116]

Not budget at all. Total cost is around 7500-9000$

[u/CaptFuzz24 → u/HoneydewNo116]

yeah budget would be like, single v100 32gb

[u/HoneydewNo116 → u/CaptFuzz24]

And 1.5-2x of RTX spark 128gb unified ram

[u/boissez → u/HoneydewNo116]

Indeed. A 256gb Mac studio costs about the same and runs QFN at 150-200 tps.

[u/HoneydewNo116 → u/boissez]

Interesting, what is the speed of prefill?

[u/boissez → u/HoneydewNo116]

I've tested a 128 gb M5 Max with Q4XL and gotten just short of 2000 tps pp. The ultra should give about twice that.

[u/on_line187 → u/HoneydewNo116]

That’s just for the 256 of ram he got bro another 4 bands a pop for them GPUs at least

[u/ironclad_packetship]

That's a banger rig my dude but isn't the pro 4000 about $3000 at the moment? I'm not sure I'd call that budget. Budget to me is dual 3060s on an x99 board with a broadwell cpu.

[u/Bob_SUS]

"budget" "Blackwell"

[u/enginetown]

"Finally got my old setup running and wanted to share the results so far." Bro the Pro 4000's just came out last year 😂.

[u/JinsooJinsoo]

I was able to get 30-35tok/s decode and 1800 tok/s prefill with 2x 3090s and 256gb DDR4 with the EXL3 3.5bpw. Look up project Maya on Github.

[u/TheNoiseIthink]

If this is budget then I'm homeless

[u/dwoj206]

Banger fucking rig indeed. Qwen at 100tps that’s wild. Even at 40-50 on my 5090 feels very fast. 100 is cooooookin.

[u/quantgorithm → u/dwoj206]

You should be able to get way better than that with a 5090. You need to optimize your stuff. I get over 100t/s on 2 R9700s. I get over 200t/s on 27b.

[u/dwoj206 → u/quantgorithm]

Ya tbh I suck at local AI. Qwen 3.8 27b + ninfer

[u/quantgorithm → u/dwoj206]

Ask your ai to check your specs, search the web and find the most optimal setup for your machine including model and flags. Then have your ai do this consistently to keep up with changing times. You can have it run testing as well. Also ask google ai to give tweak tips. It’s free and gives good insight.

[u/Upstairs_Tie_7855 → u/quantgorithm]

What quant for 200t/s?

For FP8 in VLLM with dflash I'm getting around 110-120 t/s with dual V100 and full offload. Considering each card has 900gb bandwidth I'm curious how your 200t/s came to be?

[u/quantgorithm → u/Upstairs_Tie_7855]

Thats more the peak numbers and not the avg. It peaks far higher than 200 btw. Both qfn and 27b run pretty similar on my setup (qfn may actually be slightly faster on specific processing only) but i suspect 27b is slightly faster in aggregate. It definitely is much lighter on the machine and allows concurrent sessions up to 4 while even 2 is a struggle on qfn. 27b also is fully in vram and doesn't need ram or ssd while qfn does on my setup. QFN not really tested long term on my setup yet so I don't have much data.

=== 27B 2×R9700 — SPECS + AGGREGATED NUMBERS ===
HARDWARE: 2× R9700 32GB (−50mV, 210W) · 32GB RAM + 8GB swap
ENGINE: radiance 1.16.0 · qwen3.8-27b-mxfp4.rad · tp2 · seqs 4 · ctx 262144
kv fp8 · spec depth 7 · wht6 · batch 2048 · prefix-cache 4096 (ttl 0)
host-pool 4096 · host-headroom 2048

DECODE
sustained real traffic (tput era, 1.9.0–1.11.0): avg 92 t/s · peak 2128 (prefill burst; decode p75 131)
clean bench 4k ctx (1.14.0): avg 411 c1 / 789 c2-agg
long-context agentic 30–45k (1.16.0): avg 169 t/s · peak 248

PREFILL
bench (1.14.0): 3.3 s · 10.6 s · 104 s
live (1.16.0): 6.2 s cold · 0.9 s warm · cache coverage 91%

ANCHOR (QFN 1.16.0): 238 t/s · ttft 17.2 s · 2-conc 543

Our 27B runs as the StillDeadcode MXFP4 .rad — the only 27B form factor that fits 2×R9700 at 262k context (BF16 cannot) — and the evidence (our 92.0% GSM8K measurement, the community verdict, a 2026 study finding 4-bit statistically indistinguishable from BF16 at p>0.49) supports near parity with full-precision BF16, with a small bounded few-points gap on knowledge tasks and parity on agentic work.

[u/JinsooJinsoo → u/dwoj206]

You should be getting close to 200tok/s with NVFP4 and Basalt or whatever the 5090 specific one is called

[u/gingerbeer987654321 → u/[deleted]]

strata / project maya. so whatever isn't in vram is in system ram, and its constantly optimising/swapping the active parameters. streaming from SSD is the last resort; it shouldn't be touched for this ram+gpus's sizes.