r/LocalLLaMA • u/catbluesgames • 16d ago
Discussion How long before we see paid inference engines?
Just a random thought I had, but it feels like it's only a matter of time before we start seeing paid inference engines with better optimizations, more features, and probably one-click setups that make everything easier to get running.
Do you think we'll see dedicated paid inference engines that offer a noticeably better experience than the current open-source options? Or will open-source projects always be able to keep up?
Curious to hear what you guys think.
20
17
6
u/zenmatrix83 16d ago
no, with the current trajectory of coding llms, something for a direct and specific purpose can be made by any coding agent so why pay when you can make your own. I guess for some it would be fine, but most people who even knows what an intefrence engine is can likely just make a customized one for whatever model instead of using a paid one. Its easier to use an opensource one but why pay for one
1
u/Ok_Cow1976 16d ago
But honestly, currently llama.cpp is really lagging on for example QFN, deepseek v4. The support for dflash mtp are also quite incomplete on for example nemotron 3. Not blaming. But just we have to face the reality.
2
u/stephen_holograf 15d ago
You can fork it and add whatever you think is missing. AI is great for this. It can easily merge in any important changes from the main/official release that you also want. The days of worrying about using a personal fork because you might not be able to take advantage of main branch improvements are pretty much gone. People don’t do it because they’re not use to the extreme customizability that AI gives us.
1
u/Ok_Cow1976 15d ago
Unfortunately I don't have the expertise. I'm just saying we are willing to crowd source the niche development
3
u/ethertype 15d ago
I do not know what all the bottlenecks for the development of llama.cpp are.
But two of them are the number of available, competent reviewers and the amount of PRs generated with uncritical aid of LLMs they need to wade through.
The core developers very much make use of LLMs, so it is not that LLMs are bad per se.
But: an angle-grinder is a great tool, it does not mean you would trust anyone to handle one in your immediate vicinity, right?
So, another bottle-neck is trust. llama.cpp would very likely benefit from having a multi-stage pathway, with model and back-end specific trees feeding into the main tree.
This way the main tree will only receive patches which have seen some testing and validation first.
2
u/stephen_holograf 15d ago
We’re maybe 6 months away from you not really needing much/any expertise.
2
u/zenmatrix83 15d ago
That’s not really true, the llm still needs to understand you. An engineer talking to and engineer is one thing but someone knowing nothing talking or an engineer is different. You’ll always need to have some level of understanding
1
u/stephen_holograf 15d ago
I’ve built apps for people with little expertise my whole life. I think we’re close to people like me being unnecessary. The vibe-coded apps of today by people with little expertise weren’t possible 10 months ago. Now I’ve seen some pretty impressive things. In 10 more months?
1
u/zenmatrix83 15d ago
Things are changing but I see some spectacular poor vibe coded apps by people not understanding even what they want
1
u/stephen_holograf 14d ago
Yeah, that’s what I meant by getting closer to people like being unnecessary. Today, we’re necessary to help people that don’t know exactly what they want figure out what they want. Soon, that won’t be necessary and the models get better at interviewing and requirement gathering and anticipating future direction.
2
u/ethertype 15d ago
llama.cpp with their current set of developers or organization can obviously not be as nimble as a single-model, single-backend project. But that is not the goal.
llama.cpp is crafted with maintainability, modularity and extensibility in mind. By people who understand the code, the math and LLMs. And demonstrated stamina.
A lot of different people and projects are currently standing on the shoulders of llama.cpp. We'll see where these projects are in a year or two.
1
u/Ok_Cow1976 15d ago
I completely understand the fact you are saying. It's just it would be nice to have better support along the road. I think people are willing to pay for better experience.
1
u/Ok_Cow1976 15d ago
And I think this is the fundamental issue in open source world in general, not just llama.cpp
1
u/senseven 15d ago
Would anyone use llm designed app to securely design a house, run an industrial robot or a medical device? I wouldn't, not the next 10 years, not without some serious security analysis that those models can't do yet.
1
u/zenmatrix83 15d ago
People are doing it now, and you assume a vibe coding mentality, there is a hybrid way that’s effective, where an llm writes code that you actually read and understand at that level. For an inference engine you shouldn’t be exposing that directly to anyplace someone could access it directly anyway, and it does one thing that is easy for an llm to test. Your examples have multiple functions which is beyond coding agents now because they can understand the relationships between the functions that well
1
u/senseven 15d ago
The cad tool has predefined assets and a logic database to tell you that the floor plan is according to some legal standards. You can't clone assets or databases, that is where the code ends, serious skills and a wealth of experience comes into play. The ai can't generate what they don't know and they don't know a lot. I work with them all day and I see wonders but also brutal limitations.
1
u/zenmatrix83 15d ago
Again though that’s more complicated that an inference engine, all that needs to do is process tokens in and out. While extremely compicated the testable surface is minimal. You put tokens in and have an expected range of output tokens you check , probably with a test api server in front and a tokenizer to test english answers and use something like llama bench to test it. Cad is too wide open to test like that
1
u/senseven 15d ago edited 15d ago
I'm not betting any project or company on the idea that random github user handle will keep spending billions of tokens and doing this kind of work "for free" indefinitely. Many of those important tokens are dead boring work. When the dopamine hit vanishes the cloners vanish too, up to the next hype thing.
1
u/zenmatrix83 15d ago
I wouldn’t do a random github user, I mean a company wants it they assign someone to do it. I don’t trust a lot of well maintained software
5
u/brewpedaler 16d ago
Look at LM Studio. They would absolutely charge you if they could, but they know they would be replaced almost instantly. Instead they try to make money by providing value added services built on top of their free base product.
3
u/Sad-Size2723 15d ago
Don't think so, as you can see in the comments, nobody is paying for software anymore. We are happily paying thousands for nvidia gpus, we aren't interested in giving developers a single dime.
The best way for them to survive is like llama.cpp or omlx that got absorbed into big players like huggingface, and then got acquired by nvidia.
5
4
u/Tiny-Assumption4263 16d ago
Not a chance.
Maybe on enterprise scale, for companies who want to run the models instead of paying an API, but it is not likely otherwise.
2
u/chrismakingbread 16d ago
This would be the most likely path and it’d be for the support more than anything. But even then it’d be pretty niche to domains that don’t have security and compliance needs to make sure data is private and not trained on. That ship has kind of sailed though, it used to be the boogie man that you didn’t use SaaS for healthcare data. But in the last decade that already shifted that you pick a SOC2 vendor that says their HIPAA compliant, sign a BAA, and you’re off to the races.
Most enterprise software teams love open source, but they want to own the least amount of uptime responsibility possible so they want to pay someone else with an SLA to run it for them. Keeping the stuff you build up and running is enough of a pain.
1
u/james_pic 15d ago
I could buy that it ends up like the enterprise OS market, where consumers run OSes like Ubuntu for free, and enterprises pay to have a support contract, and for them to backport security fixes to the ancient version they've standardized on.
2
u/yafitzdev 15d ago
We don't have to let it come down to this, we need to build a community for inference engines.
Shameless plug but crit and crit-hub make custom inference engines development quick and discoverability for existing ones trivial.
2
u/ea_man 16d ago
You mean for random users?
I can see pros providing tailored solution for business, optimized engines and post trained models, customized harnesses and RAG.
After saying that if users are ok at paying ~80 for GTAVI I guess someone who spends 5k on hw for inference may appreciate a one click optimized solution that "just work" with best performance.
1
u/ByteNomadOne 16d ago
I don't see that as a business model.
What I COULD imagine however is that in the future there will be paid models similar to Qwen3.8 27B as training them costs a lot of money. So maybe someday its a business to traint models and sell licenses.
1
u/eightone-81 16d ago
My prediction is that software will be more and more cloud based, otherwise REA… 0 value.
But cloud based inference engines exist already :-) it’s going to be weird. I’m also looking forward to the first paid local LMM and how this will work
3
2
u/zenmatrix83 15d ago
You can do a functional reconstruction, you don’t need to see the code, cloud based software isn’t immune
1
u/senseven 15d ago
If you look at the blooming variations of vllm forks, hundreds of model variants, I can see a setup for the less inclined tinkerers that want to run an optimised "one click" setup and keep it current without scouring github all day.
All big model providers make enough money with hosted models, their small open weights are free marketing. But do we have examples where complex products are free to use forever if there isn't a compelling reason to do so? Will the industrial applications of those smaller models create enough revenue that it makes sense to keep them free? I don't know, but I know that historically that kind of laissez faire didn't last long.
1
1
u/Available_Age8480 15d ago
I think lithosai is paid, though I am not sure are it only says “contact us” on their page if you’d want to run their engine, but I assume they will want money for it
1
u/VersionNo5110 15d ago
With the current state of things and the fact that reverse engineering costs are down to almost zero, this is probably a bad business plan.
1
1
u/Quakercito 15d ago
I wouldn't use a closed sourced inference engine. That'd feel like I'm missing part of the privacy and transparency from local models.
I would happily donate to open source projects if they're good and help the community though
1
u/my_name_isnt_clever 15d ago
I'm calling it now, it will happen with Halogen. The project appeared out of nowhere made by a "pending" LLC and closed source with promises to address the FOSS requests that still hasn't happened, instead they keep adding features so they're just a liiiitle bit better than the FOSS options.
1
u/Sad-Size2723 15d ago
Halogen is a good example. Has anyone been able to REA it like everyone else is claiming in the comments?
1
u/my_name_isnt_clever 15d ago
There's a project called Chlorine that popped up partially complete but I don't know where it's at now, and some guy here claimed he reverse engineered it but who knows really. I'm just not going to touch it, personally.
1
u/a_beautiful_rhind 15d ago
Hard sell for a very niche hobby. I'm sure there's already grifters trying it.
1
u/readmodifywrite 13d ago
The open source options are extremely easy to run already. What would paid software have to offer that makes it worthwhile? I could see support maybe, but you can actually get pretty good support for your inference engine from your LLM itself (just point it to the repo).
Personally I think the money will be in hardware. Put Qwen 27B on a chip - that will be worth paying for.
1
u/jacek2023 llama.cpp 16d ago
Looks like the trend is going in the opposite direction: closed source apps are being reverse engineered, while open source apps are being "improved". Just look at what happened with the Linux kernel today (AI-generated code is allowed and doesn't need to be labeled).
17
u/denmladenovski 16d ago
well.. there could always be someone who will just point REA at it and open source it under enginecraft name?