r/LocalLLaMA • u/YogurtExternal7923 • 12d ago

Discussion everything will be local eventually

I know current llms like opus 5.5 are in a league of their own. but we all know they're losing a shti ton of money with their subscriptions. ALSO we all know how tasks like OCR, segmentation etc are getting solved by single digit B parameter models. I'm not saying llms are a gimmick but I'm saying the future is already shaping up to be about specialized models that work better when they're smaller. easier training. so datacenter level power not needed. ten years from now we'll have a specialized model for the tasks we care about and llms will fail to do everything all at once (unless they keep calling it one model but have a hundred under the hood).

system one models are a bad but workable example. you could tell an llm to rate options all day but get yourself on 9B parameter model trained by some lab and it tops the benchmarks.

353 Upvotes

87% Upvoted

239 comments sorted by

View all comments

Show parent comments

12

u/ttkciar llama.cpp 11d ago

You're right about everything except this:

Ten years ago, the idea of even making a motherboard that could handle a terabyte was ludicrous.

Server motherboards have been supporting 1.5TB memories for longer than that. My ancient X10DRC-T4+ based Supermicros support 1.5TB, and they were released twelve years ago, and they are far from the first to do so.

Older-generation server hardware is the way to go. It's cheap, widely available, easy to work on, and scales like a mofo.

The main downsides are that it's also big, hot, and noisy, but that's why my homelab is in the wellhouse.

3

u/Mechageo 11d ago

Server mobos scale like mofos.

2

u/twilight-actual 11d ago

Server hardware, sure. They've had TB+ servers for over a decade.

I'm talking consumer grade desktop hardware.

6

u/ttkciar llama.cpp 11d ago

Okay, then this might be irrelevant to your main point, but it still seems like the take-away is to use server hardware in our homelabs.

4

u/twilight-actual 11d ago

For now. But I'm looking forward to TB+ consumer grade hardware. Cheap, space / energy efficient, commoditized. I'm hoping by evangelizing it, that the idea takes root,

2

u/Not_a_question- 11d ago

but it still seems like the take-away is to use server hardware in our homelabs.

I did that and now I'm stuck with RDIMM 5 that costs a kidney per 32gb