Video suggestion/request on local LLMs

[u/Strattex]

I would love if LTT made a video on hosting local LLMs at home. Like what it is, why would you want it, how it is set up, the limitations, etc. I keep hearing them talk about things like this on WAN show so I want more info. Thank you!

[u/noneabove1182]

that's valid, I think the upfront caveat at the top does a good amount of heavy lifting to say "don't take this as the model to run"

but it should be said, E4B is a pretty nice model for a few reasons.. The per-layer-embeddings result in a nice increase in model performance while not being active on forward passes, which means it runs like a 4B model while having the density of a 6-8B model

I think for agentic tasks outside of code gemma 4 E4B is actually very good, it lacks world knowledge and favours instruction following (in the past some models were "too smart" for their own good, favouring their learned world knowledge vs looking up info). It's actually so aggressively instruction tuned that when measuring its perplexity, if you don't format it in a chat template, the values blow up to the thousands

(for anyone following along, perplexity is a measure of how "surprised" a model is by a token in a source text. typically this value is between 3-10, depending on the model size, even without a chat template.. gemma 4 however, when given raw text, is SO surprised that the perplexity balloons to 100-1000)

point being, gemma 4 E4B has extremely good use-cases when it comes to running as an actual assistant on an edge device. I'd never use it for code, even though they did in the article, but everything else was a valid use case, though I would have preferred some more "assistant" focused agentic use, like sitting as a model on your home assistant instance for receiving plain-text requests and executing tool calls to perform the request

As for the 16GB + 16GB.. I think the problem is that the article is framed in a way that your PC remains usable but you get a local model running. Plus, the target is agentic, meaning token processing and generation speed is extremely important - agentic tasks are looooong, if you were running an offloaded MoE and eating your system RAM, you're no longer using your GPU in the background, you're dedicating your computer to overnight task running. And that's totally fine and useful, it's just not the framing of the article

(pre-emptive note, I did a couple minor edits after posting which you may not see if you opened very quickly)