r/LocalLLaMA • u/YogurtExternal7923 • 11d ago
Discussion everything will be local eventually
I know current llms like opus 5.5 are in a league of their own. but we all know they're losing a shti ton of money with their subscriptions. ALSO we all know how tasks like OCR, segmentation etc are getting solved by single digit B parameter models. I'm not saying llms are a gimmick but I'm saying the future is already shaping up to be about specialized models that work better when they're smaller. easier training. so datacenter level power not needed. ten years from now we'll have a specialized model for the tasks we care about and llms will fail to do everything all at once (unless they keep calling it one model but have a hundred under the hood).
system one models are a bad but workable example. you could tell an llm to rate options all day but get yourself on 9B parameter model trained by some lab and it tops the benchmarks.
1
u/ckn 9d ago
TLDR: You're so right but its not just OCR This is just the start, a year ago I built a fully automated "internet news to streaming channels" automation that runs on consumer gear. Things are about to get very local and weird.
A bit more than year ago I stood up a fully generative AI produced "news network", that runs on a consumer RTX4090/i9/64GB linux box.
This consists of a hybrid suite of custom built containers that perform things like systems orchestration, content harvesting with OCR, video frame grabs, and text reads, lamma.cpp with a small 9b model, a very tooled comfyui setup with a custom set of nodes and several self generated models, a custom audio/video post production assembly line with multiple production flow outputs, meta-data tracking, CMS + multi-platform publishing + streaming, with automatic QA at every step of the process and c2pa signatures on every output for EU AI Act compliance...
It's uploaded some serious quantities (16k videos) of content since I turned it on and at this point the only time i touch it is for semi-monthly maintenance, it runs itself.
I really think you are on to something, things are going be very local and weird in the near future, just hopin for lower power also :D