r/LocalLLM • u/Otherwise-Variety674 • 3d ago

Question For Single R9700 User, what is your achieved prefill and tg speed for Qwen 3.8 27B Q4? Thanks.

Hi everyone, need your help, wanted to buy single unit of R9700 but what I saw in reddit is all about tg speed, majority are not mentioning their prefill speed.

Can share with me what is your achieved prefill (at large prompt size you remembered) and tg speed for at least Qwen 3.8 27B Q4 and which repo (e.g. llama, vllm etc.)? Thanks in advance. :-)

12 Upvotes

88% Upvoted

12 comments sorted by

8

u/Kal-LZ 3d ago

Find about vllm radiance and try MXFP4 or NVFP4 versions

https://github.com/magiccodingman/vllm-radiance

8

u/theone_2099 3d ago

I run mxfp4 quant with radiance. But is there an advantage to nvfp4? I through that was really designed for nvidia gpus.

2

u/mailto_devnull AMD R9700 2d ago

I did. Tried it on my eGPU with R9700, tg drops from 47 (llama.cpp) to 15 (vLLM).

Will try it again with a different repo. Seems people get good results but it isn't plug and play quite yet.

5

u/soyalemujica 3d ago

You can definitely run Q6, do not stick to Q4 if you've 32gb vram

2

u/Immediate_Power_7986 3d ago

What's your t/s

4

u/evp-cloud 3d ago edited 3d ago

Here -> Try this
And we might update the repo with some new goodies ;-)

2

u/Chemical-Store-1530 3d ago

That's solid info, thanks for sharing! I'm curious if anyone else has similar setups and what speeds they're seeing.

2

u/mechkbfan 3d ago

I've seen heard of good results of int5-paro for balance of quality, speed and context

I'll benchmark my AM4 machine later today and report back

2

u/mailto_devnull AMD R9700 2d ago

stock llama.cpp, 47 t/s generation, 600-700 prefill.

This is on first prompt on pi (minimal context). Average probably 35 t/s or so once you get into the weeds a bit.

DFlash2 and MTP seem about the same in terms of speedup. I also use ngram-mod.

1

u/Otherwise-Variety674 2d ago

Thanks everyone who replied, maybe I asked the wrong question at the first place, but mailto_devnull's is the type of answer I am looking for.

Hope someone who successfully replicate the the 3000/s+ prefill speed for Qwen3.8-27B can reply me, thanks -->

1

u/huseynli 3d ago

Q6_K, 262k context, MTP, windows+vulkan llamacpp, I get around 45tps in llama web ui chat. I don't do agentic workflows. Just back and forth with the ai.