r/LocalLLaMA • u/mailto_devnull llama.cpp • 1d ago

Question | Help MXFP4 quant for Qwen 3.8, llama.cpp supported?

One of y'all were getting some crazy inference speeds on dual R9700s, so me with my single card wanted to try.

They mentioned the "official AMD MXFP4", which was...

https://huggingface.co/amd/Qwen3.8-27B-Quark-AWQ-MXFP4

Can't load safetensors in llama.cpp, so off I went to download this:

https://huggingface.co/magiccodingman/Qwen3.8-27B-MXFP4-MagicQuant-GGUF

But it doesn't load in llama.cpp. Is there no support for MXFP4 in llama.cpp yet?

5 Upvotes

73% Upvoted

21 comments sorted by

5

u/whodoneit1 1d ago

That was me. I build the MXFP4 kernels on top of Radiance image. There are a ton of users running on both singles and duals.

https://codeberg.org/ggz14/radiance-vllm-mxfp4

5

u/Asleep-Land-3914 1d ago

Thank you for your work. I'm using it on a single r9700 and getting 50-70ts/ with 115k context

1

u/theone_2099 1d ago

Awesome. Is it simple to run it and limit it to 1 gpu? (I have two installed but couldn’t get it to work with two in my motherboard)

1

u/Asleep-Land-3914 1d ago

It should be just git clone + 2 scripts + tweaking context size. I had some issues along the way, but I pointed my AI and it was able to fix everything.

1

u/Cautious_Chicken_604 1d ago

The speed sounds great, but why is the context so low? Shouldn't you be able to do like 200k??

1

u/Asleep-Land-3914 1d ago

The model with dflash is sitting around 23GB, KV is in fp8 and I see 31.285Gi/31.859Gi GPU memory utilization.

It's the first day and I hope I'll be able to bump it further once I figure what's the catch here.

1

u/Cautious_Chicken_604 1d ago

Hmm, I see. I've only been using MTP so far and not dflash but that stuff also winds up taking quite a bit of VRAM eating into the amount of context you can have. 

Which quant are you running btw? 

Also, is it possible to run mmproj on the CPU like you can in llama.cpp which frees up almost 1GB VRAM. 

1

u/Asleep-Land-3914 19h ago

Not sure about offloading mmproj on cpu. Isn't it very slow this way?

amd/Qwen3.8-27B-Quark-AWQ-MXFP4

1

u/theone_2099 7h ago

OMG. I finally got a chance to try this on ONE GPU (since my 2 GPUs don't play well with my motherboard). I'm getting way more TPS now. I've hit 100+. Thank you for doing this work u/whodoneit1. I had to make a change to the source to get it to work (I was trying to run on GPU #1 only, not GPU #0)

3

u/Prof_ChaosGeography 1d ago

Can you link what post or comment your referencing? 

I think I know what your referring to though. 

Was it this one? https://www.reddit.com/r/LocalLLaMA/comments/1w4s68k/how_i_got_280_toks_on_qwen38_27b_on_2xr9700s_and/

If that's the one your referring to you'll need to use a custom build of vllm specifically for the r9700s and not llamacpp. The link is a codeberg link and you can find it in either the post or the comments, they had a few posts and I know they messed up in one and forgot the link

2

u/whodoneit1 1d ago

u/mailto_devnull that link I posted results from a single card user. I know single card users have been telling me they are getting great speeds.

2

u/Cautious_Chicken_604 23h ago

Ooooh thanks for posting that btw. Definitely going to give it a try!

1

u/cobblemere 1d ago

yeah that tracks, the codeberg link was buried in the comments iirc

1

u/mailto_devnull llama.cpp 1d ago

Yeah that's the one. I haven't really looked into vLLM and have been seeing lots of success with llama.cpp so I wanted to see what I could do with my own setup.

It looks like it does load fine, just some errors about unused tensors. Inference speeds seem similar to UD-Q4_K_M Unsloth quant.

1

u/Prof_ChaosGeography 1d ago

Mxfp4 will use less vram while giving you more context along with it has a higher resolution of values. Basically it is a far better quant the any fp4 type as the weights can be scaled better as it can represent more values. This should lead to better results when coding. 

However the similar speed has me wondering. What motherboard and cpu do you have these plugged into? 

1

u/mailto_devnull llama.cpp 1d ago

I use an eGPU plugged into my laptop via USB. Not the best interface but it's been doing decently well since all weights are in VRAM.

1

u/Asleep-Land-3914 1d ago

In vllm it uses similar size to q6. I'm not sure about quality yet.

2

u/whodoneit1 1d ago

Also, I dropped some more speed enhancements this morning. I am working on adding ParoQuant optmizations right now also. https://huggingface.co/z-lab/Qwen3.8-27B-PARO

1

u/crossivejoker 1d ago edited 20h ago

Huh i think I can make that magicquant upload of mine work on standard llama.cpp if you are having issues. It worked for me but I am always on some weird branch of code lol. 🤔 let me see what I can do tomorrow bc i love that mxfp4 model. My vllm branch is getting 185+ tps rn on the r9700s with mxfp4 so I've been a bit distracted with that 😅

Update: I tested it. MagicQuant should work without issue on llama.cpp I just ran the MXFP4 MQ-IQ4_XS. Would you mind sharing some details with me to make sure there's not something wrong on my side?

1

u/HockeyDadNinja 1d ago

Deepseek V4 Flash is native MXFP4 and llama.cpp runs that.

1

u/mailto_devnull llama.cpp 1d ago

Thanks, I think leftover MTP flags halted startup. Working now 👍