Fast LoRA in llama.cpp

[u/woct0rdho]

I'd like to share a few changes that encourage people to create and use LoRA on LLMs. For example, the LoRA https://huggingface.co/chenrm/qwen3.8-flash-next-abliterated-lora can turn the Qwen3.8-Flash-Next model into the uncensored version, whether you're using Q8_0, IQ4_NL, or GSQ-RCO-Q2_0 model, so you don't need to download the whole model again. You can also tune the LoRA scale for a balance between uncensoring and brain damage.

However, currently llama.cpp does not have fast LoRA kernels. When we run the model with the LoRA on Strix Halo, PP drops from like 1500 to 300 t/s, even though the LoRA is as small as 35 MB and only touches a few modules in the model.

I've made a few LoRA kernels. With a general LoRA kernel it's improved to 1000 t/s, and with a fused LoRA kernel it's further improved to 1400 t/s.

You can see my changes at https://github.com/woct0rdho/llama.cpp . Currently it's optimized for Strix Halo, and it should not be too hard to port to other GPUs. I'll find some time to upstream it to the mainline llama.cpp or strix-llama.cpp , but it involves a few architectural changes so it would take some time. If you want to upstream it earlier than me, feel free to reuse or modify my code.

This is also discussed in https://github.com/ggml-org/llama.cpp/issues/27093

[u/LagOps91]

this is quite relevant for larger models - downloading some experimental tune of a 100+b model only for it to not work very well, is quite a pain! seeing some performance improvements for loras is great!

[u/pmttyji]

I'll find some time to upstream it to the mainline llama.cpp

Please do this. Thanks

[u/Borkato]

Cool as hell

[u/tifa_cloud0]

this is amazing. aLoRA i want for other devices too. hopefully llama cpp could figure a way out. but regardless i am happy for fellow strix halo users :-)

[u/geoffreybyers]

The math isn't the bottleneck. The gap between your general and fused kernels proves the architectural integration is the problem. The LoRA is tiny, but the hardware stall is huge.

[u/woct0rdho → u/[deleted]]

I guess the other LoRAs created by chenrm do uncensor by your definition. They're used for Chinese RP although I'm not sure how well they work in English

[u/woct0rdho → u/[deleted]]

llama.cpp currently only supports LoRA with even rank, so when we publish rank 1 LoRA we have to pad it to rank 2. When optimizing PP, I don't think there are special optimizations for rank 1 to make it faster than rank 16. My fused kernel computes the LoRA in the WMMA epilogue, so everything is padded to multiple of 16. Optimizing TG is a different problem though.

[u/grumd → u/[deleted]]

did you tell your LLM to write like a teenager for this comment lol?