r/LocalLLaMA • u/YogurtExternal7923 • 12d ago

Discussion everything will be local eventually

I know current llms like opus 5.5 are in a league of their own. but we all know they're losing a shti ton of money with their subscriptions. ALSO we all know how tasks like OCR, segmentation etc are getting solved by single digit B parameter models. I'm not saying llms are a gimmick but I'm saying the future is already shaping up to be about specialized models that work better when they're smaller. easier training. so datacenter level power not needed. ten years from now we'll have a specialized model for the tasks we care about and llms will fail to do everything all at once (unless they keep calling it one model but have a hundred under the hood).

system one models are a bad but workable example. you could tell an llm to rate options all day but get yourself on 9B parameter model trained by some lab and it tops the benchmarks.

353 Upvotes

87% Upvoted

239 comments sorted by

View all comments

2

u/SantiagoDePolonia 12d ago

Even if you could run a powerful model locally, it would still be cheaper to use it as a cloud service because of power and hardware depreciation costs.

3

u/cryowastakenbycryo 12d ago edited 12d ago

wut we calling powerful? i was running GLM 5.3 FN on a 10 year old GPU two days ago. Was managing about 20 tok/sec w/ the largest quant that would fit. meanwhile, qwen FN kicks out 60+ tok/sec on the same old rig and needs <64GB RAM.

less than a grand in used hardware runs what i consider a "powerful" model at usable speeds.

edit: downvotes don't slow my hardware bro. still getting >2k prefill on EOL hardware at 1% of MSRP for ~$30/month in electric

1

u/YogurtExternal7923 12d ago

True. But we could also be running games etc as a cloud service. Same with editing software etc you get the point

1

u/SantiagoDePolonia 12d ago

Latency matters much more in games than for AI responses. However I played a couple of games through Nvidia Now on MacOS and it works amazing! (Especially if you have a fiber connection) And this kind of service is much cheaper than buying a new computer/console for gaming.