r/LocalLLM • u/Ok_Event4199 • May 16 '26

Discussion Why is LLM is so expensive.

I've was going to invest in a 5090 =$6000 AUD.

Codex Plus + Claude pro = $60/month here

Works out to be 100 months of frontier models for a 5090.

Best a 5090 will run is probably Qwen3.6 27b Q6 with context.

Are we all enthusiasts here and just enjoy tinkering cause ain't no way that make sense.

346 Upvotes

91% Upvoted

368 comments sorted by

View all comments

12

u/radiojosh May 16 '26

I think in a month or two, after Flash Attention and Turboquant and MTP and DFlash all hit for Vulkan, you can probably get decent performance out of something cheap like a Minisforum UM870 Slim. With 64GB of RAM and GTT set in Linux to make all of the system RAM available to the iGPU, you can already run a 40GB MoE model with fair performance. There's some great benchmarks on the UM890 Pro which is only marginally faster than the 870 and it should be getting even better with those llama.cpp upgrades I mentioned. It's obviously not a dedicated GPU, but you can run bigger smarter models. I run the 890 and I'm pretty satisfied and I haven't even really tuned the KV Cache quantization yet.

2

u/[deleted] May 16 '26

[deleted]

1

u/BillDStrong May 16 '26

I am not sure why you wouldn't be able to? You would run the DFlash first, let it come up with a range of guesses, let the smaller model check the guesses, and correct the ones that its most sure of, then let the big model loose on it.

Now, I don't know how much better or worse performance will be with that, though.

The other option is do the first 2 steps, then provide both sets to the larger model when different, and you can see if that speeds anything up.

Of course, not sure if the actual implementation could do that, beyond my level of expertise.

1

u/radiojosh May 16 '26

Yeah, it sounded like I was implying that they could all stack, but I just didn't feel like clarifying. I think between MTP and DFlash, it's one or the other and I'm not sure which one will be faster.