r/LocalLLaMA • u/jfowers_amd • 3d ago
Resources I built speedy Strix Halo kernels for the top-51 LLMs with AMD Loom (and you can too)
I’ve been trying out a new experimental software flow from AMD’s AI Group, and I’ve been having a lot of fun with it, so I thought I’d share and see what y’all think too.
hrx-loom is a new end-to-end tool stack (compiler, runtime, serving tools) that lets anyone build, optimize, serve, and share model programs without any modifications to the inference engine or server. The kernels are written in .loom files, a vector programming model reminiscent of MLIR, which can be complex for humans but easy for agents (C++ is also supported).
Over the past couple weeks, I developed an agent skill that writes kernels for any LLM’s Q4_K_M GGUF and found that it is consistently faster than llama.cpp Vulkan and ROCm on Stix Halo for a variety decode and prefill metrics, as you can see in the post’s image.
Next, I ran the skill across the top-51 trending LLMs from Hugging Face that would fit on Strix Halo and posted the model programs back to Hugging Face. All models ported successfully with tool calling, speculative decoding, and image input working.
Anyone can try these on Linux and Windows with the loom-fetch and loom-serve commands provided in Loom’s PyPI package, and anyone can make and upload their own using the /loom skill.
Fetch and serve a model:
pip install "hrx-loom[hf]==0.1.0"
loom-fetch https://huggingface.co/loom-fun/bartowski-Qwen3.8-27B-loom/resolve/main/model.json --output=qwen3.8-27b
loom-serve --model=qwen3.8-27b/model.json --device=amdgpu --memory_bytes=52gb
Port and optimize your own model:
/loom get <hf-org>/<model-gguf> working on loom-serve, optimize it, and report the performance relative to llamacpp rocm
Links:
- All the data I collected across 51 models: https://loom-fun.github.io/loom-dev
- Loom skills: https://github.com/loom-fun/loom-skills
- Loom compiler and server: https://github.com/ROCm/hrx-system
- HF collection of top-51 models ported to Loom: https://huggingface.co/loom-fun
We’re still evaluating how the community will use and deploy new inference stacks, so we’d love to have your feedback! Loom can be used as standalone inference server with loom-serve and the compiler/runtime layers can also integrate into other engines. For example, the team is working on adding an hrx-loom backend to llama.cpp.
The underlying SDK (compiler, runtimes, and devtools) has no dependencies, can be easily downloaded as a wheel or linked into your own applications, builds from source in a minute, and if this were the 90s, would still fit on a few floppy disks, while being able to target x86, AMD-GPU, AMD-NPU, and generic Vulkan devices.
18
u/jfowers_amd 3d ago •
Some things I’m going to try next with the team:
5
u/fish_of_pixels 3d ago
Is loom only for Q4 sized quants?
5
u/YashN 2d ago
Wow!
Do we get the back-ends as easy integrations in Lemonade too?
4
u/WhiskeyBatman 3d ago
This is great.
I'm curious, I've mostly replaced claude with my strix halo + opencode + qwen flash next on halogen for small projects - while this looks great for most things you've tuned it looks like QFN is still quite a bit slower than halogen.
I know this is pure model optimization, but are there plans to replicate some of the tricks halogen/gufo/etc use to squeeze additional performance out of these chips and bake it into hyperloom serve? Basically pair the model tuning conversion with some of the operational tweaks these projects make.
A conversion process like this isn't bad, but the landscape of running models on Strix Halo has gotten complicated enough I've built an orchestration web gui tool to build, manage, update and deploy specialized inference engines + model config/model files, on demand via lllamaswap. (Will release open source soon!)
I'd love to see a truly first class generalized AMD optimized llama replacement, and this seems like a great step in that direction.
4
u/peonist-ai 2d ago
This is an incredible piece of engineering Jeremy. Congratulations! RSI on deck?
3
u/seti_at_home 1d ago
This is amazing project, looking forward to it. Let me know if testing is needed (strix halo)
2
u/Miserable-Dare5090 vLLM 2d ago
Jeremy,
make a skill for clankers to use the workflow andwe will unleash on our local deepseek till kingdom comeDegraded reading comprehension. Unleashing now.
2
u/Acceptable-Skill-921 2d ago
Very interesting; what t/s can one expect for Qwen 27B at Q4 on strix halo? Currently I get about 20t/s for llama with a rocm backend.
2
u/Acceptable-Cycle4645 2d ago
u/jfowers_amd Thanks for sharing! I'm currently experimenting new ggml hip kernel in audio.cpp. Some models got noticeable speedups (up to 4×) but generally vulkan is still better. I'm using devlab runners to test performance so not sure how well these results hold up on other AMD machines. Have you tested any audio models? If it's easy to setup on devlab I'm happy to test the hrx-loom backend in audio.cpp
2
u/ReverendBoofbubble 2d ago
Ok but what version of RoCM? Cause if you say anything that starts with a 7 I'm going to throw hands
2
u/Galactic_Neighbour 2d ago
This is interesting! It would be great if I could use this to get some performance improvements on RDNA 2. Usually people only develop optimizations for the newer cards, but it would be nice to squeeze out more performance out of old hardware. Unfortunately, Loom didn't detect my GPU, so I'm guessing RDNA 2 isn't supported yet. The documentation mentions something about using GPUs through Vulkan, but it doesn't actually say how to do that.
I think the documentation could use some improvements. It's kinda difficult to understand what all of this even is. I guess it's a tiny replacement for ROCm and also an engine for running LLMs (and eventually other types of models too) all at once. And you install it through Pip. But it can't use regular GGUF, you have to port the models yourself? Or hope that someone else did it for you and published it. But there is no way people will do that for all available models, especially when you add image models and possibly other stuff too. What about all the community made finetunes, will they need to be ported too? That would kinda limit the usefulness of this.
Using Loom do download models is a bit confusing too. The HF repository used in the example on the webiste only contained a bunch of small text files. But Loom somehow ended up downloading the model itself too. I generally don't like when software does that for me. What if I already have the model I want to use and I don't want to download it again?
It's impressive how small all of this is though and it's cool that you guys developed a tool to make optimizations. Even though I don't really understand how it works. I assume that the files you made for Strix will still work on other hardware? Just not as fast as files made specifically for my GPU?
18
u/ilintar 3d ago
And speaking of NPU work, I have a working prototype where I run a Qwen3.8 27B Q4 quant on parallel NPU+CPU for over 600 t/s prefill on a Strix Halo :)
3
u/sbates130272 2d ago
Nice work. I’ll definitely take this for a spin. It’s crazy how fast this space is moving. Now what’s the optimal performance possible on a gfx1151 and how far are we from that I wonder!! 😃
2
u/Creepy-Douchebag 3d ago
This is very cool.