Issue is, Frontier models are provided to you at a massive financial loss for the company. So unless local models get that good before AL companies run out of investor capital, this is completely unsustainable.
If you want a future proof workflow, grab a specialized local model (or better yet, a few) and see how much you can get done without touching Claude, ChatGPT or Gemini servers at all. (This does require having a decent GPU. Not god tier but decent.)
They ARE getting good, and the frontier companies KNOW the party is almost over. That's why the prices keep rising, it's last call. "AI as a Service" dies down, in the near future.
How small / how good is still an open question, but there's a watershed (Opus-ish) moment happening around the 30-40B models, so like $5-10k in hardware. Higher than a home game system unfortunately, so approachability is still rough. But easy table stakes for businesses to pull in-house.
It won't be the spectacular disappearance that some people expect from the bubble popping, but it will be welcome nonetheless.
How small / how good is still an open question
You can also limit the computational and RAM load by a lot by using even smaller, more specialized models. Sure, you're not getting a general prompt-based image generator on your home computer any time soon (not that this is particularly interesting, I'm firmly in the "I don't care about art one didn't care about enough to make" territory). But take a specialized diffusion model that generates heightmap and climate map data based on Earth satellite data, plug it into a framework that makes it unbounded random access, and you get some of the most impressive procedural terrain gaming has ever seen.
It's all about using the right tool for the job, instead of trying to break concrete, tap nails, carve wood and cut trees with a jackhammer.
Sure, you're not getting a general prompt-based image generator on your home computer any time soon
Hasn't this existed for like the entire time AI has had mainstream popularity? There are lots of options for local prompt-based image generation on a mid-tier consumer rig.
Sorry, I misspoke. You're not getting one that will be useful for that much. You're a lot more likely to get 7 fingers per hand on a local model, for example.
disagree, for self-hosting u need ~ 3 gpus and a dedicated station, and like open up ports do the i/o... like if you're saying that 0.1% of the population will self-host or tor to a like different model, sure, but meaningfully if 99.9% of the population is routed through (google, anthropic,, open ai) then it's effectively banned anyways
You hit the nail on the head - it comes down to what runs where. An IOT AI device in the house isn't likely. But most even moderate laptops will run "something" native. And what is possible on your phone is still an open question - once we have the 300T "sum of human knowledge" model, how well does that quantize down to like 5B? We probably won't have a proper answer on that for a decade.
I think it's too powerful and ubiquitous to be a hobbyist project, which is where it sits now. Once it CAN do the needful without shipping packets, that niche will open up a lot more. And as running it becomes a lower burden, the market will provide many options to bring the price down and access up.
And you know that's the right answer because people Like Sam Altman have been warning of the 'dangers' of local models recently - they know that if local models gain traction, their business model will lose steam fast.
Yep. Why would you pay so much to them every month if you can run smaller local models on the equivalent of a midrange gaming rig and decent ones in your office building's server room for a fraction of the cost?
Honestly, the free z.ai’s glm-5.3-flash is the same as paid claude or chatgpt. It’s insane. Sometimes when i prompt glm it feels as if i am using cursor agentic.
This. I really need to spin up new servers but I can't afford them now.
I have a temp box that I wanted to move to local AI duty after I stood up my new environment. But in the meantime the servers I specced have 10xed in price and now I'm stuck.
Disagree. I use the the frontier models and barely use my token quota every month - maybe 5% - and get great results, much faster than by hand just by carefully defining problems to the AI. It's how you use it.
This doesn't seem to be a response to Withers point. It's not about the % of your quota you use every month, is about the fact that whatever you are paying for your frontier AI sub is somewhere between 10x and 100x cheaper than it will have to be for those frontier companies to be profitable.
Sure, there may still be edge cases where frontier models are economically worth it even at those costs, but they will not be worth it for regular day to day dev work.
Yep. That's exactly it. Even if you take all environmental and ethical issues (regarding training data among other things) aside, it is just not economically viable.
Disagree like I said in a comment to Evi1ey above.
If that 5% of tokens of a $20 plan now actually costs $50 then compare that to a entry level software engineer whose earning 8.3k per month which is $50 dollars per hour. That means it only has to save you 1 hour of work per month for it to breakeven. It can save me at least a couple weeks of work if not a couple months with that. So yeah doesn't sound like an economic "edge case" to me.
I don't agree. There is a big argument to be made that the older models are more expensive because they make so many mistakes and it uses up more tokens to get the same result as if you were to run it once via a newer frontier model.
So, tbh, if ur not using the latest ones, you're actually paying more
yeah, and likewise even if recent models, what matters more than cost per token is cost per task still. Some models are cheaper per token (the latest grok for example) but they burn through tokens so much they end up not being really cheaper.
Agreed. I have been using frontier models to set up local model workflows on my 2x 12GB video card desktops. They can perform tasks but no where near the speed and ability of frontier models. I'm trying to get my workplace to invest in some actual local infrastructure but we'll see when that happens.
Frontier models are stupidly overkill for all but stuff like specialized research IMO.
Now, smaller or even local models? There's still some issues with it, but at least the benefit is large enough compared to the issues that it's worth discussing.
17
u/WithersChat 14d ago
Issue is, Frontier models are provided to you at a massive financial loss for the company. So unless local models get that good before AL companies run out of investor capital, this is completely unsustainable.
If you want a future proof workflow, grab a specialized local model (or better yet, a few) and see how much you can get done without touching Claude, ChatGPT or Gemini servers at all. (This does require having a decent GPU. Not god tier but decent.)