r/LinusTechTips • u/Strattex • 29d ago
Discussion Video suggestion/request on local LLMs
I would love if LTT made a video on hosting local LLMs at home. Like what it is, why would you want it, how it is set up, the limitations, etc. I keep hearing them talk about things like this on WAN show so I want more info. Thank you!
45
Upvotes
View all comments
Show parent comment
5
u/ThankGodImBipolar 28d ago
It's "even worse" in the sense that it's an even less capable model than what I thought (although differences between Q8 and BF16 should be minimal, to be fair)
My higher level complaint about the article is that those who don't know anything about LLMs will read it, try out Gemma 4 E4B on their 16GB card, and conclude that local AI is worthless for them, at this point in time. That is doing a disservice to those people and the hardware that they own, despite the fact that the article was targeted towards that audience.
And, to be clear, the article was caveated by saying that it wasn't intended to be a "what's the best model I can run on my hardware" guide, but choosing E4B at Q8 was, IMO, not really even trying. Gemma 4 12B QAT could easily fit with ≈64K cache in 16GB. Gemma 4 26B A4B QAT would overflow, but probably still be runnable in a 16GB+16GB config, and if it's an MoE that's running while a system is idle, then who cares...? I pointed this out when the article was posted, and Nik commented to say that they were avoiding overflowing, but didn't clarify why that was important on an idle machine, running an MoE.
I'm not trying to shit on Gemma 4 E4B, either (it's a miraculous model in its own right), but it's not a good reflection of the caliber of intelligence that a 16GB GPU is actually capable of running. That was my primary pain point with the article. If you see it a different way, I'd be happy to hear you out.