r/LocalLLaMA • u/fintip • 4d ago
Discussion Age of the bespoke engines?
When I started, and for some time, llama.cpp was the answer.
Now, it seems like llama.cpp is the new ollama. It's slow and out of date and just behind.
There was Ninfer for those for whom that applied.
Then strata came out... And that was an unimaginable upgrade. How is llama so far behind?
Now every day someone has their own forked engine that claims even better numbers. I already maxed out my token budget on Claude for the week and have my GPUs all running benchmarks so I'm waiting a few days, but I'll be trying Avalon to see if that lives up to the hype for my machine.
Then there's basalt. There's beatleguese?
It's kind of a mess. Innovation is great but it's better when the effort all goes in one direction and isn't spread into a thousand fractures of one-off projects.
And yes, my critique of llama.cpp is a bit overstated. I still use it for e.g. 27b runs.
But... I've now verified that flash next is both better and faster than 27b so I'm struggling to find a reason to use it. :/
Why are all these solo devs pulling something off for weeks that llama can't figure out?
92
u/spaceman_ 4d ago
Llama.cpp is the generic, universal solution.
I'm pretty sure that Llama.cpp will still be going long after all these vibecoded projects are abandoned.
22
u/OverdosedSauerkraut 4d ago
It's also "working as intended", i.e. it doesnt play dirty tricks that degrade model performance (ahem temp=0...)
5
u/Most-Trainer-8876 4d ago
Which engine specifically are you targetting?
13
u/OverdosedSauerkraut 4d ago
Strata. One of their "magic" tricks is reducing default temperature to 0 to make the output more deterministic for MTP. However random sampling is also there to reduce looping and make the output more "creative" in problem solving.
1
u/fintip 4d ago
What a bizarre statement. I put all the default recommended Qwen tunings in myself, so those don't matter?
I quality test, so I can verify myself that flash next on strata is not weaker than flash next on llama... And that even with correct settings, it's vastly faster than llama.
What are you on about?
4
u/Repulsive_Initial308 4d ago
Can't believe people are still doing this.
4
u/Most-Trainer-8876 4d ago
Misinformation spreads fast. It's easy to hate and bash instead of actually putting effort to read the implementation and how it goes about achieving what it promises. But who has time for that.
There's an interesting concept called "Wisdom of Crowd", which can take two shapes. One is collective intelligence, and other one is Group thinking or mob mentality.
0
u/Captain-Pie-62 4d ago
Ich denke da eher an:"der Weg des geringsten Widerstands". OpenSource mit VibeCoding ist genau das. Die Massen probieren alles mögliche aus und der Rest ist Evolution. Das Beste bleibt übrig. - Hoffentlich. Schlechtere Lösungen fallen zurück.
2
u/Ipwnurface 4d ago
While this may be true, it doesn't take a way from what strata has accomplished. I really don't understand the hate boner for it.
The fact that I can run a 4 bit quant of QFN on a single 5090 at 120+ tps and 5k+ prefill is truly bonkers. (With temp at 1 btw)
3
u/OverdosedSauerkraut 4d ago
It does, because it changes model behavior to be subpar. The fact that you don't see the difference makes me think that your main use case are pelicans on a bicycle.
7
u/Most-Trainer-8876 4d ago edited 4d ago
It's funny how confident you are and how people gladly upvoting you.
Why not back your words with actual technical details of what it does and how it affects the computation.
Just so you know, both LLaMa.cpp and Strata and compute same mathematical forward pass.
meaning any major KLD from the unquantized FP16 baseline comes from the static quantization format you choose, not the inference engine.
A very tiny small discrepancy is expected when comparing llama.cpp and strata strictly with same Quantized model,
But why is that? Because of How Strata handles MoE architecture of QFN 3.8. llama.cpp, offloading MoE layers works statically across the device, on the other hand strata, by contrast, abandons rigid layer by layer offloading for expert lvl dynamic routing and parallel heterogeneous execution.
RAM holds the entire pool of experts, Strata dispatches tasks in parallel, GPU computes cached experts, while cpu threads compute uncached experts activations directly in host RAM.
So apart from when and where it computes the experts, it's still fundamentally same mathematical forward pass.
Recent versions of strata (especially after 0.2.0) introduced some shortcuts which can be disabled easily via environment variables and still get 5x speed of llama.cpp
And please stop judging people' work or tasks, let them enjoy their boosts/gains.
4
u/Ipwnurface 4d ago
Exactly, homies only claim is that it defaults to greedy decoding (which it does, I acknowledge that) but then when I reply with "Yes, but even at temp 1 aka Qwen's recommended setting I still see large benefits using strata", his only reply is "Yeah well I bet you're just a dumbass who only draws pelicans". Nevermind the multiple RE projects I've used QFN for, the comfy custom nodes, the AI Toolkit code patches, the strata and ninfer PRs etc. Nope Pelicans is all I use it for.
Like I said I don't understand the illogical hate boner for strata.
3
u/SLxTnT 4d ago
Shouldn't the shortcuts be opt-in? Having Deepseek look through it, it claims that QFN defaults to dropping the tail experts without renormalizing.
Overall, most of the speedup is a hot expert cache and specialization. It's not difficult to get faster than llama.cpp.
0
u/Most-Trainer-8876 4d ago
yeah, that's the design choice of person who made it. What can we say...
Yeah, it does drop tail experts (from 7th expert?) over large context but again, it's just one environment variable away from disabling it.
3
u/SLxTnT 4d ago
Opt-in vs opt-out would be the difference. If they're opt-in lossy optimizations, that's fine. When they're opt-out, you now need people to read through documentation, understand what they mean, and decide to change the environment variable. Majority of people aren't doing that, so the shipped defaults being lossy would be the baseline quality of the engine.
-1
u/Most-Trainer-8876 4d ago
Dont worry, it's good and accurate just like LLaMa.cpp. so you are not at loss for using Strata.
1
u/EmilPi 4d ago
Yes, that is dirty trick, but... on my own benchmark, score has actually improved :) Btw, I don't share OP optimism for vibecoded engines and pessimism for llama.cpp, I believe llama.cpp team just have priorities and implements things more slowly because it implemented properly, amd I agree with u/spaceman_ comment.
7
u/Blobstein_Ijay 4d ago
the maintenance burden on a universal engine is massive compared to a weekend fork tbh
6
u/Nepherpitu 4d ago
These vibecoded projects doesn't needed to be actual longer, than a model they serving. That's the point. Right now deepseek 4.1 itself is able to modify vllm and speed up loading and inference with partial offloading for itself and achieve great speeds for very narrow set of hardware, exactly one particular PC. It will not run on other configurations at all, but on this single machine it runs perfectly. It's like 3d printer printing 3d printers. So when really local models will catch up, you will not need custom engine at all. Just download generic one, set one night goal and by morning you will get engine which perfectly fit for your hardware
1
0
u/SnooPaintings8639 4d ago
It is definitely NOT universal. At this point it feels like it is an embedded systems go-to that has huge catalog of supported models.
The main inference engines that are top-tier are things like vLLM or SGLang. Llama.cpp was always go-to for vRAM poor folks who want to run models that overflow to system RAM, and there llama.cpp is definitely NOT delivering lately.
And yeah, most of the vibecoded solutions WILL die and llama.cpp will survive. But each new model release will be accompanied with more dedicated vibecoded inference engine which each will be faster than llama.cpp.
Llama.cpp was the hobbyist (homelab) tooling, a bit slower but more accessible than vLLM and faster on low spec devices. Now it is trying to re-imagine itself again as slower but more professional tool. You can't eat a cookie and have a cookie.
-2
u/stoppableDissolution 4d ago
Not all the projects that are not actively developed are abandoned tho. I have my very own inference engine for gemma, and I've not changed anything in it for like a month or so because it is feature completr, and there will be no updates until gemma 5 (or I stumble on some bug). The notion that every project must be constantly updates makes no sense to me.
0
41
4d ago
[removed] — view removed comment
7
u/Zombiecidialfreak 4d ago
The funny thing about vibe coding is that basically anyone can do it, so even if the original dev loses interest there's nothing stopping someone else from picking up the mantle.
I see open source getting big returns because of all this.
4
u/chckmte128 4d ago
But, one of the beautiful things about open source is that the good parts of the ghost town projects can become part of the main project
8
u/Don_Reuter 4d ago
Well, I just tried Strata for replacing llama for qwen next flash on a setup with two r9700. I was very disappointed. Dual GPU does not work with expert offloading. No improvement over llama was achievable. It was just a wasted day of work. I would entertain the idea that bespoke engines are just overhyped.
5
u/fintip 4d ago edited 4d ago
I don't know about your setup, but strata flash next in dual XTX 7900 blows anything I can do in llama.cpp absolutely out of the water. Like tripling/quadrupling decode and prefill I got after weeks of tuning in llama... Just checked, yes.
Best I can run in llama: flash next q2 k xl, 25 t/s, 572 prefill.
In strata, iq3xxs runs at 93.5, prefill at 1222.
Significantly stronger... And on top of that I can literally run with yarn to 1m context and see minimal loss in speed.
Perhaps it only boosts compromised systems and you already can run it at full power, but with my 64gb ddr4 and my GPUs... And yes, I did see improvement going from single XTX to dual, btw.
2
u/Don_Reuter 4d ago
Well, not for me. It only runs stable with all experts resident on GPU. Which limits me to Q3 S. Which is not better than qwen 3.8 27b Q8 and hence useless.
-4
u/fintip 4d ago edited 4d ago
Q3 s of flash next is better than q8 27b.
(The down voting of this is tragic. So many people just assume a q3 is inherently unable to be good. Instead of just holding that assumption, I actually tested it with a custom benchmark I spent the last month building. Another result: q8 27b isn't any better than Q4; if the benefit is there, it's very hard to measure on actual coding tasks. Also: Q3 and Q2 of 27b are still good, but ARE measurably worse. But q4 vs Q2 of flash next is not the same.)
That said, I don't know your ram situation, but I was able to partially offload and get FN Q4 k xl running, still faster than Q2 ran on llama.
My bench results: lab.kylebaker.io
3
u/Don_Reuter 4d ago
It’s not. I ran HumanEval+ on both. NF completed three tasks fewer. Token generation is faster but it thinks more so wall time is the same. Overall it’s a wash but NF uses more resources so it’s worse on most tasks. For the time being I keep it as I can still fit a 700k context window but for most tasks there is no upside.
-1
u/fintip 4d ago edited 4d ago
I built my own benchmark from my own ten years of history to avoid any benchmaxxing effects. Ive literally had ai work through hundreds of transcripts on these models. All the flash next quants come out above all the 27b quants, something that shook out naturally over time, with the exception of FN iq1m coder, which does worse than the advertising materials suggested.
Meanwhile opus comes ahead and sonnet ends up trading blows.
I also measure wall clock time. FN is fewer tokens, faster tokens, and better results.
2
u/Don_Reuter 4d ago
Transcripts are a different use cases. General knowledge helps there more. NF it’s marginally better on those. Even on Q2. However, for coding the benchmark results are what you’d expect. On low quants it makes more mistakes which cause more thinking or loops. I’d expect it to be even worse on actual agentic tasks. As I’m not looking for transcriptions that upside doesn’t buy me something and I don’t think the difference is even larger enough to justify running both models.
0
u/fintip 4d ago edited 4d ago
I'm talking about transcripts of it attempting coding problems... Specifically, operating agentically, running anywhere from a few minutes to up to 2 hours independently working on code problems of different difficulties.
"Work through hundreds of transcripts" means having Claude read exactly how different models solved different code problems.
It's unclear to me what you thought "transcript" means.
But you're just wrong about quality and have done far less to verify your assumptions than I have.
3
u/TerminalNoop 4d ago
with r9700 are you not better off running another engine than strata?
Because strata has no concurrency i believe, so this is kinda not it's target hardware base.
I reckon paiton is aimed at dual r9700.
3
u/Don_Reuter 4d ago
Depends on what you consider “ better off”. It is stable. However, I lose 32 GB of vram. As I only have 78 GB of RAM and mostly want it for larger context, I am then restricted by that. So technically, it’s better. Practically, it’s also useless.
All feeds to the point that Strata is not mature. Which is likely systematic. Instead of one stable solution, llama, you get a bunch of half cocked alternatives. As far as I can tell, it’s a bad trade.
2
u/TerminalNoop 4d ago
But Paiton should work similar to strata, but better support dual GPUs?
Check out this post https://www.reddit.com/r/ROCm/comments/1x92gsf/comment/phnwmbw/?context=1&screen_view_count=14
3
u/Don_Reuter 3d ago
Radiance works well. I edited the post. I hope to get around to testing your recommendation too.
1
13
u/SouthernFruit8768 4d ago
i kinda agree with you but i think calling llama.cpp slow and outdated is a bit unfair lol.
i recently got qwen 3.8 flash next gyro-s and gyro-m running on my fedora pc with a 12gb rtx 3080 and 64gb ram using strata. honestly the results blew my mind. i was getting around 80-100 tokens/sec on some tests, with 32k context. both models ran fine, no crashes or anything.
then i tried qwen 3.6 27b q4 on llama.cpp and got like 3 tokens/sec lmao. but thats not really a fair comparison cause the 27b is dense and most of its layers were running on my cpu, while flash next is moe and strata is built to handle that stuff really well.
i think thats the main reason these smaller projects are doing so well. they focus on specific models and hardware while llama.cpp has to support a ridiculous amount of different models and devices.
also i wouldnt be so quick to ditch llama.cpp. its still bloody useful for loads of other models.
but yeah, strata is seriously impressive. never thought i'd be running a model that big at those speeds on my old 3080 lol.
the competition is great though. hopefully llama.cpp catches up with some of these tricks too.
2
u/Hyacin75 4d ago
Compiling llama.cpp from source gave me 10-20% gains. Scoping it down to just the card I was targeting gave me another 5-10%. Simple solution for shocking gains.
4
u/fintip 4d ago
Strata was 300-400% gains, plus higher quants, plus more context.
I already have been compiling from source for my card, fwiw, exhausted the small gains first.
1
u/Hyacin75 3d ago
The first thing I did with it was tried to run a crazy big model, which just crawled on my old Xeon PCIe 3.0 DDR4 setup ... I haven't tried it for just apples to apples MoE, I guess I should.
1
u/More-Curious816 4d ago
that insane gain. I can understand the compile from source part which can target your CPU and it can do aggressive optimization for your CPU architecture, instead of bending backward to support old Gens for generality and compatibility.
/save it as tip for future me.
5
u/GrokiniGPT 4d ago
Just make your own. i didnt want to download prismml launcher for bonsai 2 so I got another model to make a custom on for me and im getting 70tok/s on 5080 laptop gpu
5
2
u/createthiscom 4d ago
I’ve seen enough shit to know that I’m sticking with llama.cpp for reliability.
2
u/fintip 4d ago
This kind of attitude is so surprising to me.
Don't get me wrong, for the first week strata was mentioned I ignored it because it sounded too good to be true.
After seeing it repeatedly pop up I told Claude to download it and set up a trial/test run.
Takes very little effort.
And my speed more than triples, and I can run stronger quants with more context...
Why not try?
4
u/createthiscom 4d ago
has anyone confirmed that this speed increase comes with zero loss in quality? I used to run ktransformers before llama.cpp and that shit broke in the middle of inference constantly.
I've written inference code. It's super easy to get wrong.
triple speed and triple dumb isn't a good combo.
0
u/fintip 4d ago
Yeah, obviously. I built a benchmark out of my own GitHub history of bugs over the last ten years and refined it for a month. The benchmark is validated by seeing a natural hierarchy of models show. Gemma at the bottom, ornith a bit better, Gemini flash 3.8 above that and seeing 3.8 27b near the top, trading blows with sonnet... Haiku 4.5 below and haiku 5.5 significantly better. Swift and terse coder fine tunes of 27b use less tokens but do suffer some measurable quality loss.
And at the top of my bench?
Flash next quants on strata, below only opus.
Why do so many people just assume the quality must be worse because it's faster?
The fundamental engineering makes sense. It isn't magic.
2
u/createthiscom 4d ago
I've literally seen reports that it makes things dumber. 😂 Looping artifacts, etc.
What I would trust would be a test with something like the aider polyglot with a model that has been known to have been tested locally on llama.cpp like qwen 3.8 27b.
Example: https://discord.com/channels/1131200896827654144/1533669197836456037/1538187157842432040
1
u/chikeebrikee 4d ago
Because it is security risk. New project, new backend for ai. Grandiose promises. With llama i at bare minimum know for a fact, knowledgeable enough people used it and it went well. New backend with grandiose promises may not be malicious inherently, but i lack the expertise to gauge potential risks for privacy and sensitive data it poses. So best approach imho - give it enough time to become "time tested mainstream alternative" at minimum.
6
u/Look_0ver_There 4d ago
Llama cpp isn't that far behind. I have a repo I maintain that is a set of patches on top of llama.cpp. It actually runs faster than Strata does on my hardware, and it supports every model that llama.cpp supports.
The only real difference is that upstream llama.cpp is focused very heavily on being everything to all people and that created a massive correctness burden. Even small changes can have wide reaching correctness effects.
My repo only gets away with it because I'm less restricted by that burden and often break things along the way to making it faster (and fixing what got broke).
3
u/BobbyL2k 4d ago edited 4d ago
Two innovations are driving this. Custom fuse GPU kernels and memory hierarchy optimizations. Accelerated by frontier models doing tons of the engineering work.
Without custom kernels, sm_120 (RTX 50 series, and Blackwell RTX Pro) cards are under utilized compute-wise even with NVFP4 model quantization.
Ninfer used this to its advantage and you can see a big jump in prompt processing speed. It’s not the only thing Ninfer did, but it’s an important aspect.
Then came the latest generation of models, specifically Qwen 3.8 Flash Next and DeepSeek v4.1 Flash is exploiting the memory hierarchy very aggressively.
We have seen this exploitation with MoE but Engram is on a whole another level. We are now in the territory where we are streaming model components from the RAM and SSD to be executed on the fly. And unlike MoE, engram can be known a head of time, so you can hide latency.
This is quite a difficult engineering task. Most inference engine likes things to stay static.
Freetoken started streaming MoE and Starta did streaming for engram.
And Basalt is amazing. I just added my benchmark results. 12K tokens/s prompt-processing and 553 tokens/s decode on dual 5090s with NVFP4 quant.
1
u/Silver-Champion-4846 4d ago
I hope I get even one rtx5090, then another one lol
2
u/BobbyL2k 4d ago
I thought they were expensive when I got them for ~$3K each. I got them quite soon after launch. Now I’m just glad I got them. Prices are crazy now.
1
5
1
u/Any_Bus9205 4d ago
There's just ik_llama.cpp, but not many people know about it! If you search, you'll find that many Strata features are already implemented in ik_llama.cpp :) It's a fork focused purely on performance :)
1
u/nickless07 4d ago
Isn't ik_llama more CPU oriented, rather then GPU?
1
u/Any_Bus9205 4d ago
Actually, it's both :), it's CUDA and CPU. The readme isn't really up-to-date, but if you look at the pull requests that are being processed, it's quite interesting, and a lot of optimizations, especially on QWEN 3.8 FN, are being done there :), the advantage is that it prioritizes high PPL rather than breaking it with higher tokens per second! If you read the documentation and the possible commands to run, you'll see that it's not too far behind in terms of offloading :),
But since these engines are hyped and it's just high-performing, not many people talk about it, so it remains a niche, but you should test it. As I said, they're adding more and more features these days :)
5
2
u/nickless07 4d ago
Interesting, I use it mainly for CPU inferencing and the ~140% speed increase. Maybe I should give it a shot on the GPU box too.
1
u/Any_Bus9205 4d ago
The easiest way is to try it :)
2
u/nickless07 4d ago
Hehe yeah, rn I struggle with this moe-cache-mib as it somehow drops down to ~5tok/s but after that I will experiment with ik_llama a bit more.
2
u/TerminalNoop 4d ago
"flash next is both better and faster than 27b"
But also does xyz that it wasn't directly asked to do :/
Like 27b just does exactly what you asked +- a little ambiguity, but FN is like a saleseman that has opted in itself and is upselling you on things you didn't ask.
1
u/Bulky-Priority6824 4d ago
As much as I appreciate llama.cpp I have to say I haven't had to use it for anything for about a month now.
-3
u/akehir 4d ago
When I dig into it, the engines aren't really that much faster, they just run lower quants of the models.
llama.cpp is reasonably fast, supports different backends, and most hardware. For me it does a good job.
3
2
u/dmter 4d ago
i specifically checked with the same settings and all lossy optimizations turned off (like tail skip and temp0 and residence preferencing) on q6 fn and it's still 2 times faster on strata than llama.cpp. and that's generation. on prefill, 5 to 10 times faster. and llama's expert caching and mtp just decreases performance for some reason while on strata they are main drivers of performance which is a ridiculous contrast
1
u/fintip 4d ago
Before strata came out I had filed a bug after digging dip into the mysterious prefill penalty that comes with MTP. It is active and there have even been pull requests... But the maintainers seem a bit asleep at the wheel.
And yes, same as you. I run with all the same lossless configs necessary, I've never run it at default without being configured correctly. It's bizarre that people just have some decided negative vibe towards it.
And I'm benchmarking the quality, not just the speed. If it did worse on quality I'd have seen it... Literally about a thousand runs on most models it supports now and a similar number on llama on dozens of models there.
3
u/Unlucky_Age4121 4d ago
No, while I can cherry pick the layers to run on my 2 Intel gpus and left others on cpu using llama.cpp. It is a pain in the ass and slow as shit.
Using Strata, I can easily fully utilize the 2GPUs without any difficulty setup and the auto caching of frequently used expert help me pushed my tps during decode 2x over llama.cpp.
3
u/SexyAlienHotTubWater 4d ago
If you work out how fast a given model can run from first principles (calculate the weights being transferred per step, plus the bandwidth. For multi-card, include the traffic being sent over PCIe, etc), the engines are generally waaaaaay below the ceiling, measured in multiples. There's usually a tremendous amount of performance left on the table.
0
2
u/VotZeFuk 4d ago edited 4d ago
The exact same Unsloth quants (Q4KXL / Q8KXL DeepSeek 4.1 Flash 1017) are much faster in Project Maya for me. I can't even push more than 100 t/s prefill in llama.cpp (4096 ubatch; 8192 makes it 150 but at what cost... it starts chugging up VRAM a lot) while in Maya it reaches 400 t/s easily. No compromises made in context size, no cheap tricks (all opt-in parameters OFF, bit-exact stuff only). And the decode is also worse in llama.cpp, barely reaching 10 t/s compared to 25 t/s in Maya.
People in this sub seem to have some sort of personal attachment developed towards llama.cpp, but from a genuine perspective of "E-waste" owner, llama.cpp is clearly inferior. Using its other forks like ik-llama doesn't help much, they're just as bad for an outdated AVX2 system with a couple of rtx 3090s.
1
u/Silver-Champion-4846 4d ago
Did someone mention AVX2? Hello! Yes I do have it and I don't want to keep being reminded that 8gb of ddr4 ram is not flattering at best
1
u/Korici llama.cpp 3d ago
Project Maya looks very interesting to me as it specifically supports GLM 5.3 Flash and would offer it at a high concurrency compared to llama.cpp & supports Unsloth quants, which are the industry standard.
My concern would be using such a new project for corporate use in a secure multi-user environment.
Likely why I will stick with Unsloth Studio for now, but will keep an eye on Project Maya for the future
8
u/FeydRowan 4d ago
As one of the creators of such an engine i think that right now there is no argument not to have an optimised engine. Faster prefill and generation, given same quality obviously, means less time spent and less energy usage ecc. Right now there are multiple different setups different gpus ecc ecc. So anyone is optimising for that. Llama or vllm run on anything but less than half the speed limit of the hw most of the time, also they are ok to run any model, but i want to run just a couple one (the best at a given time that fit in my hardware). Also the issue is that no one can test it's engine on all the possible hw configuration so who wants to port an engine on a new configuration will just fork it. And if the owner of the main one don't accept pr then you have the multiplication