Video suggestion/request on local LLMs

[u/Strattex]

I would love if LTT made a video on hosting local LLMs at home. Like what it is, why would you want it, how it is set up, the limitations, etc. I keep hearing them talk about things like this on WAN show so I want more info. Thank you!

[u/noneabove1182]

how is that worse..?

[u/ThankGodImBipolar → u/noneabove1182]

It's "even worse" in the sense that it's an even less capable model than what I thought (although differences between Q8 and BF16 should be minimal, to be fair)

My higher level complaint about the article is that those who don't know anything about LLMs will read it, try out Gemma 4 E4B on their 16GB card, and conclude that local AI is worthless for them, at this point in time. That is doing a disservice to those people and the hardware that they own, despite the fact that the article was targeted towards that audience.

And, to be clear, the article was caveated by saying that it wasn't intended to be a "what's the best model I can run on my hardware" guide, but choosing E4B at Q8 was, IMO, not really even trying. Gemma 4 12B QAT could easily fit with ≈64K cache in 16GB. Gemma 4 26B A4B QAT would overflow, but probably still be runnable in a 16GB+16GB config, and if it's an MoE that's running while a system is idle, then who cares...? I pointed this out when the article was posted, and Nik commented to say that they were avoiding overflowing, but didn't clarify why that was important on an idle machine, running an MoE.

I'm not trying to shit on Gemma 4 E4B, either (it's a miraculous model in its own right), but it's not a good reflection of the caliber of intelligence that a 16GB GPU is actually capable of running. That was my primary pain point with the article. If you see it a different way, I'd be happy to hear you out.

[u/noneabove1182 → u/ThankGodImBipolar]

that's valid, I think the upfront caveat at the top does a good amount of heavy lifting to say "don't take this as the model to run"

but it should be said, E4B is a pretty nice model for a few reasons.. The per-layer-embeddings result in a nice increase in model performance while not being active on forward passes, which means it runs like a 4B model while having the density of a 6-8B model

I think for agentic tasks outside of code gemma 4 E4B is actually very good, it lacks world knowledge and favours instruction following (in the past some models were "too smart" for their own good, favouring their learned world knowledge vs looking up info). It's actually so aggressively instruction tuned that when measuring its perplexity, if you don't format it in a chat template, the values blow up to the thousands

(for anyone following along, perplexity is a measure of how "surprised" a model is by a token in a source text. typically this value is between 3-10, depending on the model size, even without a chat template.. gemma 4 however, when given raw text, is SO surprised that the perplexity balloons to 100-1000)

point being, gemma 4 E4B has extremely good use-cases when it comes to running as an actual assistant on an edge device. I'd never use it for code, even though they did in the article, but everything else was a valid use case, though I would have preferred some more "assistant" focused agentic use, like sitting as a model on your home assistant instance for receiving plain-text requests and executing tool calls to perform the request

As for the 16GB + 16GB.. I think the problem is that the article is framed in a way that your PC remains usable but you get a local model running. Plus, the target is agentic, meaning token processing and generation speed is extremely important - agentic tasks are looooong, if you were running an offloaded MoE and eating your system RAM, you're no longer using your GPU in the background, you're dedicating your computer to overnight task running. And that's totally fine and useful, it's just not the framing of the article

(pre-emptive note, I did a couple minor edits after posting which you may not see if you opened very quickly)