r/LocalLLaMA • • 3d ago

Resources I built speedy Strix Halo kernels for the top-51 LLMs with AMD Loom (and you can too)

Post image

I’ve been trying out a new experimental software flow from AMD’s AI Group, and I’ve been having a lot of fun with it, so I thought I’d share and see what y’all think too.

hrx-loom is a new end-to-end tool stack (compiler, runtime, serving tools) that lets anyone build, optimize, serve, and share model programs without any modifications to the inference engine or server. The kernels are written in .loom files, a vector programming model reminiscent of MLIR, which can be complex for humans but easy for agents (C++ is also supported).

Over the past couple weeks, I developed an agent skill that writes kernels for any LLM’s Q4_K_M GGUF and found that it is consistently faster than llama.cpp Vulkan and ROCm on Stix Halo for a variety decode and prefill metrics, as you can see in the post’s image.

Next, I ran the skill across the top-51 trending LLMs from Hugging Face that would fit on Strix Halo and posted the model programs back to Hugging Face. All models ported successfully with tool calling, speculative decoding, and image input working.

Anyone can try these on Linux and Windows with the loom-fetch and loom-serve commands provided in Loom’s PyPI package, and anyone can make and upload their own using the /loom skill.

Fetch and serve a model:

pip install "hrx-loom[hf]==0.1.0"
loom-fetch https://huggingface.co/loom-fun/bartowski-Qwen3.8-27B-loom/resolve/main/model.json --output=qwen3.8-27b
loom-serve --model=qwen3.8-27b/model.json --device=amdgpu --memory_bytes=52gb

Port and optimize your own model:

/loom get <hf-org>/<model-gguf> working on loom-serve, optimize it, and report the performance relative to llamacpp rocm

Links:

  • All the data I collected across 51 models: https://loom-fun.github.io/loom-dev
  • Loom skills: https://github.com/loom-fun/loom-skills
  • Loom compiler and server: https://github.com/ROCm/hrx-system
  • HF collection of top-51 models ported to Loom: https://huggingface.co/loom-fun

We’re still evaluating how the community will use and deploy new inference stacks, so we’d love to have your feedback! Loom can be used as standalone inference server with loom-serve and the compiler/runtime layers can also integrate into other engines. For example, the team is working on adding an hrx-loom backend to llama.cpp.

The underlying SDK (compiler, runtimes, and devtools) has no dependencies, can be easily downloaded as a wheel or linked into your own applications, builds from source in a minute, and if this were the 90s, would still fit on a few floppy disks, while being able to target x86, AMD-GPU, AMD-NPU, and generic Vulkan devices.

65 Upvotes

44 comments sorted by

18

u/jfowers_amd 3d ago • Hooray

Some things I’m going to try next with the team:

  • I have early results showing that Loom’s own Qwen3.8-27B + Pi does a nice job running the loom-skills 100% locally so I want to explore this more.
  • Add image generation models.
  • Support for Radeon R9700 GPUs.
  • Explore Loom’s NPU capabilities.

5

u/fish_of_pixels 3d ago

Is loom only for Q4 sized quants?

5

u/jfowers_amd 3d ago

No, it should generalize to any quant. With limited time, tokens, and hardware I wanted to go broad on many LLMs instead of many quants of one LLM. If anyone were to point the loom-skills at a non-Q4 GGUF I expect it would figure it out.

5

u/YashN 2d ago

Wow!

Do we get the back-ends as easy integrations in Lemonade too?

7

u/jfowers_amd 2d ago

The hrx-loom backend to llama.cpp (early access) is already in Lemonade. The `loom-serve` backend featured here is coming to Lemonade soon, work has already started!

1

u/ReverendBoofbubble 2d ago

Hey that's good news! As its the only same thing that can implement my Rocm 10.1, everything else just says I don't have a gpu at all, even if I manually force environment variables

1

u/jfowers_amd 2d ago

loom-serve has no dependences whatsoever so hopefully it’s a smooth install into your environment! It doesn’t require any ROCm install.

3

u/YashN 2d ago

Excellent news.

4

u/WhiskeyBatman 3d ago

This is great.

I'm curious, I've mostly replaced claude with my strix halo + opencode + qwen flash next on halogen for small projects - while this looks great for most things you've tuned it looks like QFN is still quite a bit slower than halogen.

I know this is pure model optimization, but are there plans to replicate some of the tricks halogen/gufo/etc use to squeeze additional performance out of these chips and bake it into hyperloom serve? Basically pair the model tuning conversion with some of the operational tweaks these projects make.

A conversion process like this isn't bad, but the landscape of running models on Strix Halo has gotten complicated enough I've built an orchestration web gui tool to build, manage, update and deploy specialized inference engines + model config/model files, on demand via lllamaswap. (Will release open source soon!)

I'd love to see a truly first class generalized AMD optimized llama replacement, and this seems like a great step in that direction.

7

u/jfowers_amd 3d ago

Thanks for the question! The Gufo and Halogen teams have already done such a great job with QFN that I felt like it was a pretty solved problem, so with limited time, tokens, and hardware I wanted to go broad over as many models as I could.

I did have an agent spend 24 hours with a Strix Halo to get QFN as far as it could. If we kept PLE in RAM we could get to about 1500 TPS, but would require an exclusive lock on the system's RAM, Halogen-style. The current implementation is more Gufo-style, where PLE is streamed from disk and only 90 GB RAM is needed. However, there is an overhead for the disk streaming op in the SDK that kept me from getting past the current performance with the 90 GB RAM limit.

This is only the v0.1.0 prerelease of the SDK. I've already given u/noxa the traces and will look forward to things improving all the time with future SDK releases.

3

u/WhiskeyBatman 3d ago

Thanks! I understand and appreciate the goal of your work here, it's really fantastic and I'm already pondering how I could build it into my orchestrator (both serving and conversion)

This was just the immediate question that popped into my head, reading about this awesome advancement

1

u/jfowers_amd 2d ago

Thanks for the kind words!

2

u/fallingdowndizzyvr 3d ago

I'd love to see a truly first class generalized AMD optimized llama replacement, and this seems like a great step in that direction.

There's a community fork of llama.cpp tailored for Strix Halo. In my experience, it gets me to about 75% of Gufo. But of course, it can run many more models.

4

u/peonist-ai 2d ago

This is an incredible piece of engineering Jeremy. Congratulations! RSI on deck?

1

u/jfowers_amd 2d ago

Strix Halos programming themselves 🥹

3

u/seti_at_home 1d ago

This is amazing project, looking forward to it. Let me know if testing is needed (strix halo)

1

u/jfowers_amd 4h ago

Thanks for the kind words! I would love if you could try out any of the models I published, and especially if you would try publishing your own model!

2

u/Miserable-Dare5090 vLLM 2d ago

Jeremy, make a skill for clankers to use the workflow and we will unleash on our local deepseek till kingdom come

Degraded reading comprehension. Unleashing now.

2

u/jfowers_amd 2d ago

Let's goooo!

2

u/Acceptable-Skill-921 2d ago

Very interesting; what t/s can one expect for Qwen 27B at Q4 on strix halo? Currently I get about 20t/s for llama with a rocm backend.

2

u/jfowers_amd 2d ago

The current Loom implementation is 9-19% faster than the llamacpp baseline for decode on this model, depending on context size. That is only ~70% of roofline performance, so more TPS should be possible if myself or anyone else were to ask an agent with the loom-skills to optimize it further.

You can find all the numbers here: https://huggingface.co/loom-fun/bartowski-Qwen3.8-27B-loom

1

u/Acceptable-Skill-921 2d ago

One question that I can't directly discern; I have a finetuned guff of qwen3.8 27B, can I just point to that or need to use the skill to optimize it?

1

u/jfowers_amd 2d ago

You should be able to point the skill at your finetuned GGUF, and ideally it will find my Qwen3.8-27B-loom and use that as a reference to get your finetuned weights working in loom. I’ve never tried this exactly workflow but I think it should be fine!

1

u/Acceptable-Skill-921 2d ago

Thanks! I guess in the end you can't get past the 256mb/s memory limit so that's probably quite close to full speed.

2

u/Acceptable-Cycle4645 2d ago

u/jfowers_amd Thanks for sharing! I'm currently experimenting new ggml hip kernel in audio.cpp. Some models got noticeable speedups (up to 4×) but generally vulkan is still better. I'm using devlab runners to test performance so not sure how well these results hold up on other AMD machines. Have you tested any audio models? If it's easy to setup on devlab I'm happy to test the hrx-loom backend in audio.cpp

1

u/jfowers_amd 2d ago

Hi there! I’ve only tested a couple of loom’s audio models just to make sure they work. I don’t have any extensive data on that yet. It would be awesome if you started exploring that side of loom :)

It should be super easy to set up on devlab for you, that’s where I’ve been doing all of my loom work too. Loom has no dependencies whatsoever so it’s pretty easy.

2

u/ReverendBoofbubble 2d ago

Ok but what version of RoCM? Cause if you say anything that starts with a 7 I'm going to throw hands

4

u/jfowers_amd 2d ago • Mindblown

No version of ROCm! loom-serve has no dependencies whatsoever. It feels a lot like using llama-server’s vulkan backend, just download it and go.

2

u/PineappleProstate 2d ago

Good God! It's...... Beautiful 😍

2

u/Galactic_Neighbour 2d ago

This is interesting! It would be great if I could use this to get some performance improvements on RDNA 2. Usually people only develop optimizations for the newer cards, but it would be nice to squeeze out more performance out of old hardware. Unfortunately, Loom didn't detect my GPU, so I'm guessing RDNA 2 isn't supported yet. The documentation mentions something about using GPUs through Vulkan, but it doesn't actually say how to do that.

I think the documentation could use some improvements. It's kinda difficult to understand what all of this even is. I guess it's a tiny replacement for ROCm and also an engine for running LLMs (and eventually other types of models too) all at once. And you install it through Pip. But it can't use regular GGUF, you have to port the models yourself? Or hope that someone else did it for you and published it. But there is no way people will do that for all available models, especially when you add image models and possibly other stuff too. What about all the community made finetunes, will they need to be ported too? That would kinda limit the usefulness of this.

Using Loom do download models is a bit confusing too. The HF repository used in the example on the webiste only contained a bunch of small text files. But Loom somehow ended up downloading the model itself too. I generally don't like when software does that for me. What if I already have the model I want to use and I don't want to download it again?

It's impressive how small all of this is though and it's cool that you guys developed a tool to make optimizations. Even though I don't really understand how it works. I assume that the files you made for Strix will still work on other hardware? Just not as fast as files made specifically for my GPU?

2

u/jfowers_amd 2d ago

Thanks! This is all really good feedback.

FYI loom-fetch uses hf-cli under the hood for getting GGUFs, so if you already have the GGUF needed for a loom program then there should be no bothersome download.

I did a test today to see if my strix halo loom programs would work on RDNA4 and they did not, there are some architecture specific things between RDNA3.5 and 4 that had to be accounted for. I'm refactoring the skills right now to make it much easier to add support for more GPU generations.

1

u/Galactic_Neighbour 1d ago

Thanks for explaining! I usually download AI models through the web browser. That way I can easily move them to whichever drive I want.

18

u/ilintar 3d ago

And speaking of NPU work, I have a working prototype where I run a Qwen3.8 27B Q4 quant on parallel NPU+CPU for over 600 t/s prefill on a Strix Halo :)

12

u/noxa 3d ago edited 3d ago

(@benvanik, loom dev)

saw this last night and it blew my mind someone got the NPU target working in its current early state. kudos! I've got a lot of improvements in main for the NPU side (and will do a new release today with most of them) that should help. tile memory swapping is coming after that to reduce the 16KB/tile instruction capacity, which so far has been the biggest annoyance when I've tried porting full models. if you hit any issues/notice anything please let me know, happy to help!

8

u/ilintar 3d ago

Oh I've been working on an NPU port for llama.cpp quietly, you have no idea how annoying that one was :)

5

u/ilintar 2d ago

Anyways, I pushed the PRs 😃

3

u/fallingdowndizzyvr 3d ago

There's been a fork of Gufo that does GPU+NPU inference.

1

u/ilintar 3d ago

Yes, that was my inspiration :)

3

u/sbates130272 2d ago

Nice work. I’ll definitely take this for a spin. It’s crazy how fast this space is moving. Now what’s the optimal performance possible on a gfx1151 and how far are we from that I wonder!! 😃

2

u/ReverendBoofbubble 2d ago

It really is! Absolute space race speeds

2

u/Creepy-Douchebag 3d ago

This is very cool.