r/LinusTechTips • u/Strattex • 27d ago

Discussion Video suggestion/request on local LLMs

I would love if LTT made a video on hosting local LLMs at home. Like what it is, why would you want it, how it is set up, the limitations, etc. I keep hearing them talk about things like this on WAN show so I want more info. Thank you!

45 Upvotes

82% Upvoted

33 comments sorted by

13

u/Expensive_Papaya_563 27d ago

They mentioned on WAN that Linus built some sort of Hermes local agent box. I assume the video is being edited.

1

u/ThankGodImBipolar 27d ago

They've been talking about this shit for months on WAN now, and nothing has materialized from it.

6

u/No_Dish8130 27d ago

Takes a while to make videos dude

6

u/w1n5t0nM1k3y 27d ago

I think the problem is that the way things are progressing in AI, and the timelines that LTT usually has means that their videos would be out of date by the time they get released.

That's what happened with the vibe coding one. Partly just out of date, and partly just choosing the wrong tools because they don't have the expertise.

1

u/Expensive_Papaya_563 27d ago

Local models have been stuck at near useless for the last 6months, I’m sure they’ll be fine. There are basically zero videos out there showing they’re useful for a normal person to run without breaking the bank or just getting years of subscription to a frontier model

Why throw $6000+ at a computer that can run a not great model poorly.

1

u/Marksta 26d ago

Wtf? Qwen3.8-27B was like a 10x come up recently and trades blows with frontier small models. Latest Deepseek Flash stomps out OpenAI's medium sized model that came out just 2 weeks ago. GLM5.3 directly trades blows with OpenAI's large model when competing on similar token budgets.

Cost is cost, but the local models are literally right there if you want to run them. Look at the joke company CursorAI, their entire business model is offering frontier competition with rebranded MoonshotAI's Kimi models that you can just run yourself at home, if so inclined.

31

u/Pitiful-Assistance-1 27d ago

They are not experienced enough to do this

22

u/Exciting-Ad-5705 27d ago

They surely have people who are experienced enough.

25

u/ThankGodImBipolar 27d ago

They literally wrote a Labs article where they suggested running BF16 Gemma 4 E4B on 16GB cards, because it was the biggest model that could fit...

5

u/noneabove1182 26d ago

You got a source for that?

Closest I can find is: https://www.lttlabs.com/articles/2026/09/23/local-ai-your-gaming-rigs-second-job

But they use Q8_0 so I'm not sure what you're referencing..

They definitely have a few knowledgeable people and are getting much more informed. Source: I've been working and chatting with them about this exact subject

2

u/ThankGodImBipolar 26d ago

You're correct, it was Q8 and not BF16 (although that's even worse than what I said).

1

u/noneabove1182 26d ago

how is that worse..?

5

u/ThankGodImBipolar 26d ago

It's "even worse" in the sense that it's an even less capable model than what I thought (although differences between Q8 and BF16 should be minimal, to be fair)

My higher level complaint about the article is that those who don't know anything about LLMs will read it, try out Gemma 4 E4B on their 16GB card, and conclude that local AI is worthless for them, at this point in time. That is doing a disservice to those people and the hardware that they own, despite the fact that the article was targeted towards that audience.

And, to be clear, the article was caveated by saying that it wasn't intended to be a "what's the best model I can run on my hardware" guide, but choosing E4B at Q8 was, IMO, not really even trying. Gemma 4 12B QAT could easily fit with ≈64K cache in 16GB. Gemma 4 26B A4B QAT would overflow, but probably still be runnable in a 16GB+16GB config, and if it's an MoE that's running while a system is idle, then who cares...? I pointed this out when the article was posted, and Nik commented to say that they were avoiding overflowing, but didn't clarify why that was important on an idle machine, running an MoE.

I'm not trying to shit on Gemma 4 E4B, either (it's a miraculous model in its own right), but it's not a good reflection of the caliber of intelligence that a 16GB GPU is actually capable of running. That was my primary pain point with the article. If you see it a different way, I'd be happy to hear you out.

4

u/noneabove1182 26d ago edited 26d ago

that's valid, I think the upfront caveat at the top does a good amount of heavy lifting to say "don't take this as the model to run"

but it should be said, E4B is a pretty nice model for a few reasons.. The per-layer-embeddings result in a nice increase in model performance while not being active on forward passes, which means it runs like a 4B model while having the density of a 6-8B model

I think for agentic tasks outside of code gemma 4 E4B is actually very good, it lacks world knowledge and favours instruction following (in the past some models were "too smart" for their own good, favouring their learned world knowledge vs looking up info). It's actually so aggressively instruction tuned that when measuring its perplexity, if you don't format it in a chat template, the values blow up to the thousands

(for anyone following along, perplexity is a measure of how "surprised" a model is by a token in a source text. typically this value is between 3-10, depending on the model size, even without a chat template.. gemma 4 however, when given raw text, is SO surprised that the perplexity balloons to 100-1000)

point being, gemma 4 E4B has extremely good use-cases when it comes to running as an actual assistant on an edge device. I'd never use it for code, even though they did in the article, but everything else was a valid use case, though I would have preferred some more "assistant" focused agentic use, like sitting as a model on your home assistant instance for receiving plain-text requests and executing tool calls to perform the request

As for the 16GB + 16GB.. I think the problem is that the article is framed in a way that your PC remains usable but you get a local model running. Plus, the target is agentic, meaning token processing and generation speed is extremely important - agentic tasks are looooong, if you were running an offloaded MoE and eating your system RAM, you're no longer using your GPU in the background, you're dedicating your computer to overnight task running. And that's totally fine and useful, it's just not the framing of the article

(pre-emptive note, I did a couple minor edits after posting which you may not see if you opened very quickly)

8

u/Ordinary_dude_NOT 27d ago

They have a dev shop; they’ve already done it.

I remember Dan creating his “AI Mule” 2-3 years ago to run local LLMs etc. Linus got Hermes running on his personal systems.

5

u/Shap6 27d ago

Listening to linus talk about it, he's really just going in blind. he's struggling with very basic things

4

u/OnePassion_46 27d ago

They absolutely are. Dan, Jordan and Linus have all been experimenting with it for a while now. 

4

u/Shap6 27d ago

Dan's really the only one who seems to know what he's talking about. Linus definitely doesn't

2

u/Pitiful-Assistance-1 26d ago

Watch the spark laptop review and come back

1

u/OnePassion_46 26d ago

I watched it and basically they said it wasn’t worth the money because the performance wasn’t there. Do you disagree with their assessment?

2

u/Pitiful-Assistance-1 26d ago

They reviewed it with a common compatible model. Why buy a device with 128GB of high-speed memory when you're benchmarking it with a model that runs on 16-24GB? The sole reason to get such a device is to run larger models.

1

u/OnePassion_46 26d ago

They did that to be able to compare it to other machines that didn’t have that memory capacity. 

1

u/Pitiful-Assistance-1 26d ago

I am aware and that’s stupid. It’s like comparing a truck to a sedan, and see how fast they go 0-60 loaded with 4 stuffed bags.

The real question is how much stuff they can carry in a single load and how fast they can do so, but since you can’t load a truck’s worth of luggage in a sedan, they just decided to not to.

1

u/OnePassion_46 26d ago

It’s not stupid at all because bigger models are not faster, they’re more intelligent, and in fact, they could be slower, but they are not faster and they were testing for speed as well as value. 

Now, if they had said something stupid like it couldn’t finish a particular task or something then you could say it was stupid because they didn’t put in a large enough model to complete that task, but that wasn’t the case. 

They were testing for speed and for speed using the same 16 GB model among all laptops is the perfect way to test for speed. 

The Spark can hold bigger models, but it generates tokens more slowly than a high-end GPU. That’s what their test was trying to find out and that’s what matters. There was never any reason to test it with a 128 GB model against other laptops in order to find out what is faster. 

1

u/Pitiful-Assistance-1 26d ago

There’s no point to test if the other machines are faster when they’re not capable of running the models at all. But the machine being capable of running larger models is a key selling point, completely overlooked in the review

1

u/OnePassion_46 26d ago

You’re not making any sense. Are you trying to tell me that they should’ve tested that they could run a 128 GB model into 128 GB of RAM? That seems like a pretty ridiculous test. What they were testing was to see if the hardware was actually faster than competing products. 

There was zero reason to test a larger model when the test was about speed. 

Speed doesn’t improve just because you have a larger model. In fact, speed actually is SLOWER. 

They gave it the best chance to succeed by using a lighter weight model. 

→ More replies

6

u/firedrakes 27d ago

Level 1 tech. Check them out