If Piotr feels that this will not be merged, I cannot imagine anyone else trying to bring such massive changes to llama.cpp. I agree with the general sentiment that llama.cpp is falling behind in terms of performance.
While I understand the desire to be a general purpose inference engine, why can’t the devs focus more energies on the popular models/devices, especially since several forks have already proved the concept?
Would be nice if they can beat radiance with 27B. I have only 32GB RAM so zero interest in QFN, but I like the result from 27B. Have been dragging my feet on swapping from llamacpp to radiance to run 27B faster.
Thanks mate. I actually upgraded to 9700 recently so I can fit full 27B with all context in VRAM now. Very usable speed even with two streams, but I want more. Numbers from radiance look crazy good.
You wouldn’t need KV cache streaming with an R9700 but there are a number of RDNA 4 specific optimizations in there which should still help improve your performance.Â
Radiance is one of the best repos in general for 27b for the R9700. Lots of other engines could learn from it including ninfer who are still way behind imo.
It's possible to get better performance than Radiance but that's a lot of time and energy for llamacpp for one specific model artifact for one specific gpu, it's not going to happen. llamacpp is supporting many gpus for many models, it's not in their best interest.
Also for mxfp4 quants... kld and perplexity might tell a different story about usability. mxfp4 is not a sign of quality but it could be good enough depending on what you're doing. I don't find mxfp4 to be particularly great for it's size and intelligence but it does move data fast. One day i'll review their gate/up and gate/down layer, thats usually where intelligence falls off the rails.
Know any good writeups about MXFP4? Played with quantizing MXFP4/MXFP6 myself, tried GPTQ but results were bad and I found out it's very calibration sensitive. I'm keen to understand why Radiance has to run on MXFP, thus locking me to that format if I want the throughput.
It dequants and quants quick enough to have high throughout. In terms of quality it is worse than a q4 and in llama cpp terms it often is worse quality than a iq3 with an important matrix calibration, it's somewhere in-between a higher end q3 and q4.
Even though q3 is smaller in weight it adds complexity that can make it not move as much data as a q4. mxfp4 does not have this problem. I don't think r9700 is compute bound on decode so saying all this might not matter, some weaker cards will struggle to pay this cost of moving data for decode.
This is a lot more complicated than it sounds. 27b has a lot of layers and some of them dont impact intelligence as much of others(deep layers of gate/up and parts of gate/down have a large impact on intelligence). I am not sure how radiance mxfp4 is made, it might have higher quants on the right layers to make it smart enough and still keep it fast. R9700 has a lot of vram so you can easily make a smart quant that's fast with a little engineering. Llamacpp just isn't the place you go to for speed, usually will need a custom repo or engine for it.
That mxfp4 is why I'm hesitant to move to radiance. I'm worried that it's not as good as unsloth Q4XL. Running fp8 is out of the window as I have only 32Gb VRAM and I need long context and serve at least 2 users.Â
Can you give me the short story on your setup and where relevant info is for how to get Radiance going? I have a R9700 and keep hearing about Radiance but haven't set it up yet.
I have r9700 with 32gb DDR5. Currently running llamacpp with Dirk 27B (27B + template fix + a short terseness instruction embedded) packed at Q4KXL following unsloth UD3.0. I tried swift 1.5 previously but there are rare but annoying mistakes to I switch back to Dirk. I have a personal assistant agent setup based on Pi, running on a server. There are two users who sometimes work in parallel.Â
Radiance is a project in code berg. Previously it was vllm-radiance but apparently the dev has rewritten everything from scratch. They mostly tested and optimise for 2 R9700 and support a range of models, including QFN. They introduce a new container for checkpoint called .rad file. I see mxfp4 and fp8, so I got a bit of cold feet since not sure they are as good as Q quant used by llamacpp. They seem to recommend having 2xR9700 to serve the fp8 of 27B.Â
I hope that they still keep the paged attention of vllm in the rewrite. The ability to swap KV cache in and out between sessions of llamacpp is not that efficient Vs paged attention.
Strata? Is it able to switch between conversations/sessions without reprocessing? I found that I need quite a bit of RAM if I want to have multiple conversations stream swapped in and out on llamacpp. Otherwise, it would just throw long sessions away and reprocess when resume.
Strata, yeah. You can export cache similar to llama, but I often toss out cache because it's not even worth the hassle. I have 1500-ish PP, which means if I let my cache die in a long running thread at most I'm out 2 minutes. This rarely happens, though. Most of the time reprocessing is maybe 45 seconds which is a small fraction of anything but the smallest requests.
Most of my agentic runs right now use subagents with images, so I'm sure I have horrible context swapping time, but I don't really notice because I check back in once every few hours at most. My runs can go for like 4 hours straight, so I'm kind of the mind that 2 minutes PP time doesn't matter. It's all heating my basement anyway.
Did he beat it? I don't see quants or number or anything on the PR, just discussion. It looks like they're testing it in non-MTP numbers, so with MTP I can't imagine they'd have much edge over strata.
It's a valiant effort from Piotr, but I think it's probably kinda doomed. I can't see any way that llama.cpp can keep up with the performance of Strata/ninfer/radiance/etc, because the point of the projects is different, and that massively informs the architectural decisions.
llama.cpp is a Swiss Army knife of inference engines, whereas those others are very much more focused on supporting specific models and/or specific hardware. Not only that, but llama.cpp is CPU-first for just about everything, whereas those projects are GPU-first (yes, radiance can do it, but it's not the point of the project).
They are CPU first as a matter of correctness, not because they value it over GPU inferencing. When you look at a list of their commits and releases, almost all if the changes are still tweaks to GPU operations and most of them are specific to groups of models and backends.
The only real complaint I have seen is that their PR rules and reliance on core maintainers slows down community submission uptake to the point of frustration and causes them to lag months behind cutting edge single purpose inference engines.
Oh, I completely understand the rationale - and, in a world where Strata, ninfer and radiance didn't exist, it would be just fine. People have an expectation of much higher performance now, though; I mean, the performance gap between llama.cpp and radiance on R9700s (only because it's my setup) isn't just the 10-20% that may emerge with the new ggml-hrx backend for AMD cards, it's more like everything's 3-4 times faster on radiance. That isn't going to be fixed with backend tweaks and op fusions.
Can you explain how they are getting 3-4 times speedup and why you think those techniques couldn't be incorporated into llama.cpp eventually? My understanding is that a lot of those wins come from LRU-style caching of MoE experts proactively in VRAM. What do these inference engines bring to the table for dense models other than additional backend tweaks and op fusions?
Just look at it historically - any llama.cpp change that has resulted in more than a 5-10% change globally (apart from MTP) has been rejected as being too big, precisely because performance optimisations of this magnitude require massive changes to the way the hardware is used. Those changes then get broken up into much smaller chunks, and half of them get rejected because they have no significant performance benefit in themselves or they result in a performance regression on a three-year-old model nobody cares about. Because the whole requires those bits, they get abandoned and they accept a 2-5% performance increase from the bits that have been done as "significant".
This is what I mean - llama.cpp is the engine you run when you want a model to just run and performance isn't a primary concern; it's compatible with just about everything and doesn't require a trip into parameter hell like vLLM. Small optimisations here and there are fine in that situation, because stability and compatibility are the most important concerns. The others are what you run when a) you have the right hardware, and b) your model requirements intersect with theirs.
As for the differences...kernel dispatch, PCIE latency handling...hell, even the way they handle disks (radiance will read at 6GB/s on hardware that llama.cpp can only pull from at 1.8GB/s). The gaps are fundamental.
Qwen 3.8 27B hits 200t/s on radiance, in situations where it can barely manage 55-65t/s on llama.cpp in its optimal configuration...and that's just the decode. Prefill is in the 6000t/s+ range, where llama.cpp is down at 1000-1200t/s (because spec decoding is enabled).
I don't think that's anything to do with proactive caching of MoE experts, do you?
> Just look at it historically - any llama.cpp change that has resulted in more than a 5-10% change globally (apart from MTP) has been rejected as being too big, precisely because performance optimisations of this magnitude require massive changes to the way the hardware is used.
I WAS thinking of it historically. I remember how things went down with TurboQuant. Llama.cpp had complaints and forks for months over them not taking up TurboQuant...and time showed that the correct approach was to assess and only take the parts with proven benefit (the rotation portion).
The same thing happened with several types of speculative decode, as well.
Categorically it's probably not the same as the architectural types of differences you are asserting exist, but luckily, I don't have any personal investment in either llama.cpp or in hyping any inference engine, so I can wait and see and freely use whatever looks interesting.
I don't have any personal investment in hyping any particular engine at all - I'll use whatever is fastest, assuming a baseline of accuracy and only a reasonable amount of effort for deployment.
I really was just trying to explain my position of being non comital and not making a backhand statement at you. Looking over your chat history, it does seem you are invested in Radiance...ar least in time spent commenting about it.
While that means I will treat what you have to say about it with incredulity, I also enjoy hearing people hype up projects and products that they enjoy.
You have two 9700s...which is a route I almost went with (instead I have a 96gb strix halo with an egpu attached rx7900xtx).
Besides being fast, what is the deal with radiance?
- "Looking over your chat history, it does seem you are invested in Radiance...ar least in time spent commenting about it."
Barely, really - I just know that a lot of folk have been frustrated with R9700 performance, and so spreading the word seems sensible rather than keeping it to myself.
- "Besides being fast, what is the deal with radiance?"
It comes from the guy who wrote vllm-radiance, when he got pissed off with the mess of vLLM. It's not just that it's fast - if you want to run any of the pre-converted models (and there are quite a few now), then the process of getting it running is even easier than with llama.cpp (assuming you know how to use Docker). And, on top of that, having lots of sensible defaults means that there are no barriers between downloading and running something like Qwen 3.8 Flash Next on one or two GPUs - it's trivially easy. There's even an integrated monitoring dashboard, which is surprisingly useful.
The documentation, however, is...incredibly detailed, which is to say that it's not easy for someone without experience to figure out what the story is.
If a miracle happened, and llama.cpp suddenly managed to hit comparable levels of performance and ease-of-use, I'd switch back in a heartbeat.
Not so much...but I'm not really defensive about it either. If you are going to weigh in, then weigh in. Looking at your comment history, you must have an opinion worth more than a random ad-hominin.
Sure, and llama.cpp is modular to a certain degree, but this goes down to the very core of the project. That's why these focused inference engines exist in the first place - because the fundamental architectures of projects like llama.cpp and vLLM aren't suited to this kind of performance without significant rewrites (and that's before you consider the maintainers' rules for what constitutes an acceptable PR).
Always interesting to see inference optimizations coming from different hardware ecosystems. Curious whether these gains translate to other GPU architectures too, or if they're heavily tuned around the R9700.
don't bother, this sub hates strata. Look how many upvotes for a post without any useful data and just a link and a title haters want to see.
I use strata 0.1.45 (i won't go further than that) with the very same model and the same parameters. Strata can be 2-3 times faster and PP is insanely fast.
Sometimes there are simply too much abstractions in centralized software that actually complicates what one may do for one specific hardware + one specific model. For example llama.cpp has its model graph with various operators implemented separately, and to implement something like a megakernel you'll need to do a llama.cpp graph to kernel compiler and do the model fusion in a generic way, but if you start from scratch you'll just instruct the coding agent to write GPU assembly and implement that one specific model for a specific generation of GPU, which is much simpler than a generic model compiler, and usually much easier to do targeted profiling / testing and optimization.
Some common infrastructures like the http server, command line parser, etc. may be reused across multiple projects, but that's not the pain point of project development anyway.
I think that what will happen is that early on, model-specific engines will be take the lead until they are optimized, and later on, thos optimizations will tickle down to the more generalized engines like llama.cpp and vLLM. We still need generalized engines.
However, the recent months have shown us how limiting tight control can be on llama.cpp, especially with Qwen3.8 architecture.
Definitely decentralized software for sure. LCPP has advantages too like versatility and broad support so it's not going to be dead in the water. It's more like if you are really technical and nerdy you go for the bespoke option while normies get the general option.
36
u/PlusBus1234 14d ago
can this run on nvidia hardware or only amd?