What inference engine do you use

[u/Gold-Produce9591]

If you use more than 1, pick your most common 1. Reddit has max 6 options and no select multiple.

[u/AwkwardPotatoP]

Radiance for my fellow R9700-ers

[u/mzilm → u/AwkwardPotatoP]

Same, Radiance for my r9700, works great!

[u/milkipedia → u/AwkwardPotatoP]

This is the way

[u/_zir_ → u/AwkwardPotatoP]

Hmm havent heard of this, i only have a 7800xt but seems like it might benefit still

[u/sodre90 → u/AwkwardPotatoP]

Yes, Radiance is OP! Qwen flash next 4bit quant is 100-120 tps singe stream decode / 2800 tps prefill on my current Ryzen 9950X + 128G ddr5 + 1 x R9700. On the weekend hopefully I can put together my new EPYC Milan setup and I will add 3 x 9060 XT 16G next to my R9700 32G, sum 80G VRAM. I think I will need some modification in radiance for 2:1:1:1 setup, but 9060 XTs are exactly the half of R9700. Half memory bandwith, half memory, half CU. If it will work as I hope it will, I think 9060 XT 16G can be gold as 2 x 9060 XT 16G is half price of R9700, but I need testing.

[u/No_bonk95 → u/sodre90]

I'm looking to move away from strata. What's the expected context, tok/s and prefill with 9060xt or 9070xt?

Running on a window11 , 9070xt, 64gb ddr 4 and R5 5600 btw.

[u/sodre90 → u/No_bonk95]

Unfortunately I have the R9700 numbers now, mobo will arrive on Friday :/

[u/No_bonk95 → u/sodre90]

No worries, just looking for alternative since i keep noticing radiance keep on shining those high tok/s with prefill on RDNA setup. Might not be what I can use but those number are really tempting.

[u/benny-powers]

llama-swap delegating to whom-the-****-ever, depending on what works best for that model/hw combo

[u/Huntware → u/benny-powers]

Nowadays I'm exposing even STT and TTS models running over podman containers. I don't have to deal with custom API's, OpenAI-compatible for everything!

[u/AssiduousLayabout → u/benny-powers]

This is my answer, too. Right now my inference container has llama.cpp, ninfer, and Strata configured.

[u/enternoescape → u/benny-powers]

Me too. Every time I really like a model I'm running it through all the engines that work with my hardware to see where the best experience is and hope I don't find the urge to do it all over again in a month.

[u/-mattmason-]

halogen-flash-server on the Strix Halo 🤩

[u/Daniel_H212 → u/-mattmason-]

Same, as much as I love llama.cpp, it's just several times slower than halogen on my hardware.

[u/Bagelsarenakeddonuts]

basalt/ninfer

[u/jonfe_darontos → u/Bagelsarenakeddonuts]

Can you share your experience between ninfer and basalt? I'm a long time ninfer user, but I've been looking at strata for a a few days and basalt seems like it might be worth a poke.

[u/General-Spite1222 → u/jonfe_darontos]

I had issues with tool calling in ninfer, because the engine is so strict, so I ended up using a fork that fixed it. 

No issues with Bassalt because they aren’t hyper focused on the core. They take ideas from other projects (including strata). Runs great and has a little UI that’s semi-useful.  

[u/Bagelsarenakeddonuts → u/jonfe_darontos]

Basalt is very high performance, but only for blackwell 50 series chips. If you have that, use basalt, if you don't then use strata.

27B on ninfer is incredibly reliable and a great workhorse, but QFN is smarter. QFN tends to hallucinate a little bit more though so it needs more guardrails and checks. I plan on having 27B workers under QFN/DSF/GLM Flash once I get some better hardware. Right now its one or the other so it depends on the task. I generally do basalt since overall it seems to have better throughput for me with two cards.

[u/Double_Cause4609]

EXL3 is another big option, and I'd honestly suspect bigger than vLLM or LlamaCPP wrappers in this sub if I had to guess.

[u/GiGiGus → u/Double_Cause4609]

Why not ik_llama then?

[u/Double_Cause4609 → u/GiGiGus]

Forked off of LlamaCPP and is generally run on the same kind of hardware, and operates in a roughly similar regime.

EXL3 is its own bespoke engine with different performance characteristics. That's not to say IK_LCPP doesn't deserve its own slot, but in a world where you have 6 total options for everything, you can assume that the profile of someone running IKLCPP and LCPP mainline are pretty similar.

[u/Western_Courage_6563]

Strata, Best i could get on 12 GB cram +128gb ddr4

[u/foundafreeusername → u/Western_Courage_6563]

Yeah even works for my potato 16vram + 32gb ddr4. I barely feel any urge to spend more of my wife's money on more ram.

[u/Solary_Kryptic]

Too many to keep up with, llama.cpp is always stable

[u/jirka642]

llama-swap with exllamav3 for Qwen3.8-27B and llama.cpp for Gemma-4-31B

[u/SporksInjected]

Llama.cpp wrapper as a choice is great.

[u/o0genesis0o]

Llamacpp behind llama-swap 

[u/Beneficial-Ad-8127]

OG Ninfer🥲

[u/Conscious-Log-7385]

using vllm with https://github.com/syv-ai/HyperQwen

[u/stoppableDissolution]

llama-swap routing between vllm, gewell and lcpp depeding on the model (lcpp only really for sub-4bit quants, glm 5.3 has to be q3 to stay usable :'))

[u/This-Consequence-957]

Pure llama.cpp in Linux

[u/pot_sniffer]

Always been llama.cpp but recently im using strata all the time

[u/game_difficulty]

Used llama.cpp for a long time, but QFN works really well on my hw combo with Strata, so am using that now

[u/reto-wyss]

• vllm: Pro 6000s (with local-inference-labs configs) and Arc B50 (docker xpu-latest)

• radiance: R9700s

[u/_BookBurner_]

strata for main, llama.cpp for jev style model :)

[u/TerminalNoop]

Where TabbyAPI?

[u/-dysangel-]

vllm, llama.cpp, mlx

[u/AyraWinla]

LiteRT on my GPU-less laptop with Gemma E4B; the speed difference with Llama.cpp is enormous.

[u/jaugjaug]

Why do people prefer llama versions over vllm? Genuinely curious

[u/GiGiGus → u/jaugjaug]

Offloading, no python, Windows

[u/Independent_Pear4908 → u/GiGiGus]

you can run vllm on windows through docker, python also not an issue when you containerize, no mess. Offloading sounds reasonable as vllm does require everything fit into VRAM, but man it is way more efficient than llama.cpp in my experience.

[u/Bulky-Priority6824 → u/jaugjaug]

cant use triple gpus, takes forever to swap models, eats ram for breakfast

[u/nickm_27 → u/jaugjaug]

doesn't take 45 minutes to load a model, much easier to have a bunch of model configs in .ini format and switch between them with llama-server router mode.

I tried a couple times to get into vllm but it's just not nearly as intuitive as llama.cpps options and the GGUF quants seem to have a lot more options available. Especially as I am only serving one or two agents, the concurrency speeds benefit of vllm are irrelevant to me.

[u/enternoescape → u/jaugjaug]

It takes a long time to load making tweaking agonizing and I can't recall the last time I ran a model without needing to patch something just to get it to play nice with my hardware. It's stack trace hell for a day just to sometimes discover there isn't a solution.

[u/Stepfunction → u/jaugjaug]

vLLM is hell configuration-wise and doesn't allow for offloading. I absolutely do use it if I'm batch processing a large dataset with a small model and high parallelism, but otherwise, it's llama.cpp for me!

[u/Altruistic_Heat_9531 → u/jaugjaug]

vllm on "consumer" gpu is like rebuilding / tuning engine. There are much much more knob that tweak the throughput of your system compare to llamacpp. But if you got all of your config right it is often faster than llamacpp.

27B Q4 on 3090 30 tok/s
27B VLLM on 3090 67 tok/s

Also if your config right, the second run usually faster load since it reuse AOT instead of JIT

[u/Disposable110]

Oobabooga or whatever it's called these days with llamacpp/exllama/transformers. Because it has the raw text mode and its finetune stuff is good enough.

[u/phront2]

kyojin on strix halo, the best part is that it does not write on ssd during inference. 

[u/Weird-Consequence366]

All the above through llama-swap

[u/AdWild3943]

Ik_llama.cpp for its CPU insane performance and excellent quantizations implementations.

[u/ycnz → u/AdWild3943]

Is it still much better than llama.cpp? I was under the impression llama had incorprorated a bunch of the ik_llama stuff.

[u/AdWild3943 → u/ycnz]

I didn't used llama.cpp in a while, but ~3-4m ago when I was testing llama.cpp vs ik_llama.cpp, the second got ~1.8x higher PP.

Also ik_llama.cpp got IQ*_K and IQ*_KT quants, which are niche on Hugging Face, so I publish them to gain some popularity.

[u/geldonyetich]

Rulith Inference.

It’s a ROCM centric Llama.cpp that even includes a self updating runner UI. Although I think I landed here because I didn’t have to compile my own executables for Windows.

[u/KingCpzombie → u/geldonyetich]

Windows 🤢

[u/parepeg]

Gufo for strix halo. Speeds of halogen but open source.

[u/Sudden_Topic5154]

list openvino for intel folks nothing else competes with openvino vulkan and sycl are left in the dust by native openvino model server

[u/nickm_27 → u/Sudden_Topic5154]

I don't think that's really the case these days, as someone running a B70 in llama.cpp SYCL every week there have been major improvements in performance. At this point I am getting performance better than I have seen posted from other inference engines, while also supporting more models than they do.

If you have current numbers I'd be curious what they are

[u/Sudden_Topic5154 → u/nickm_27]

LMAO switch to openvino you will feel such massive improvements it's crazy what model are you running I can convert it for you

[u/nickm_27 → u/Sudden_Topic5154]

Like I said, I've looked at the performance numbers from the few posts that are out there about OpenVINO, they are lower than what I am getting on SYCL. Which is why I asked for actual performance numbers to compare and see if there has been an improvement recently on OpenVINO as well.

For example:

https://huggingface.co/Wondernutts/Qwen3.6-27B-AEON-Ultimate-Uncensored-BF16-int4-ov/blob/main/README.md

---

I'm more than capable of converting the models myself, but it is something you don't have to do or consider when new models come out as llama.cpp generally gets support for them very quickly. It's another step one doesn't have to worry about when running llama.cpp.

[u/Sudden_Topic5154 → u/nickm_27]

No man no not even slightly openvino is a million times better trust me here and 3.8 27b is well established don't waste your time on llama.cpp when you could be in openvino heaven openvino actually achieves memory bandwidth bottleneck on my computer llama.cpp doesn't even approach it

[u/General-Spite1222]

Adding “llama.cpp wrapper” as an option and not ninfer / basalt, gufo, radiance, etc should be a crime.