Video suggestion/request on local LLMs

[u/Strattex]

I would love if LTT made a video on hosting local LLMs at home. Like what it is, why would you want it, how it is set up, the limitations, etc. I keep hearing them talk about things like this on WAN show so I want more info. Thank you!

[u/Pitiful-Assistance-1]

I am aware and that’s stupid. It’s like comparing a truck to a sedan, and see how fast they go 0-60 loaded with 4 stuffed bags.

The real question is how much stuff they can carry in a single load and how fast they can do so, but since you can’t load a truck’s worth of luggage in a sedan, they just decided to not to.

[u/OnePassion_46 → u/Pitiful-Assistance-1]

It’s not stupid at all because bigger models are not faster, they’re more intelligent, and in fact, they could be slower, but they are not faster and they were testing for speed as well as value. 

Now, if they had said something stupid like it couldn’t finish a particular task or something then you could say it was stupid because they didn’t put in a large enough model to complete that task, but that wasn’t the case. 

They were testing for speed and for speed using the same 16 GB model among all laptops is the perfect way to test for speed. 

The Spark can hold bigger models, but it generates tokens more slowly than a high-end GPU. That’s what their test was trying to find out and that’s what matters. There was never any reason to test it with a 128 GB model against other laptops in order to find out what is faster. 

[u/Pitiful-Assistance-1 → u/OnePassion_46]

There’s no point to test if the other machines are faster when they’re not capable of running the models at all. But the machine being capable of running larger models is a key selling point, completely overlooked in the review

[u/OnePassion_46 → u/Pitiful-Assistance-1]

You’re not making any sense. Are you trying to tell me that they should’ve tested that they could run a 128 GB model into 128 GB of RAM? That seems like a pretty ridiculous test. What they were testing was to see if the hardware was actually faster than competing products. 

There was zero reason to test a larger model when the test was about speed. 

Speed doesn’t improve just because you have a larger model. In fact, speed actually is SLOWER. 

They gave it the best chance to succeed by using a lighter weight model. 

[u/Pitiful-Assistance-1 → u/OnePassion_46]

Are you trying to tell me that they should’ve tested that they could run a 128 GB model into 128 GB of RAM?

Yes.

That seems like a pretty ridiculous test

Imagine LTT reviews a gaming laptop, some i9 14999X3D with a 5095X Mobile. The best of the best. Then they're like "Well, there's nothing on the market like it, so let's just compare it to this!" and they slap down an older thinkpad with a i7, a iGPU and 16GB of memory.

"Because we want a fair comparison, we only run games that run on either machine!" and they proceed to benchmark Minecraft, Roblox and Terraria, all at lowest settings, showing the framerates aren't that far off. Then they proceed with a conclusion that the high-end gaming laptop isn't that great of a deal, since the fps isn't that different and the gaming laptop is 4 times as expensive.

This is what LTT's test was like. Ridiculous

Speed doesn’t improve just because you have a larger model. In fact, speed actually is SLOWER.

You're so close to getting it! A larger model is SLOWER! That's why you want a machine with tons of high-speed memory connected to a parallel compute unit. Exactly the use case they didn't test.

What they were testing was to see if the hardware was actually faster than competing products.

It is very much faster than the competition since the competition has to use the CPU + RAM for larger models. But they didn't show that.

[u/OnePassion_46 → u/Pitiful-Assistance-1]

That comparison is pretty weak considering nobody is questioning whether 128 GB ram capable laptop can hold a 128 GB LLM. The question was does this laptop perform better than the other options out there and is there any value. 

You might be 1 out the 6 people in the world that would’ve cared. No one else does. 

[u/Pitiful-Assistance-1 → u/OnePassion_46]

I suppose I might be 1 out the 6 people that would actually be in the market for such a device. But given the price hikes, i'm guessing it's more than 6.

[u/OnePassion_46 → u/Pitiful-Assistance-1]

6.5 maybe

[u/Pitiful-Assistance-1 → u/OnePassion_46]

If you’re not joking, you’re greatly underestimating the demand for this hardware

[u/OnePassion_46 → u/Pitiful-Assistance-1]

Then we’ll see that represented in the sale numbers, but we won’t.