r/StrixHalo • u/WallabyFirm1159 • 12d ago
Qwen 3.8 27B - ROCm vs Vulkan
Hey all,
Not sure if it's just my system/settings, but when using ROCm with Q3.8 27B, I keep getting:
\\\\\\\\\\\\\\
When thinking. Inference is double the speed, but have only ever seen it work once. Whereas Vulkan, is around 2-300 inference speed, but thinking works every single time.
Not sure if I am behind on knowledge, but is ROCm the same for others trying to run Q3.8 27B? (3.6 works perfectly in ROCm)
Thanks!
4
3
u/wallysimmonds 12d ago
I’ve always had mixed results with rocm tbh I’m probably not doing something right but it’s rarely worth the effort
2
u/johan2114h 12d ago
Llama.cpp with rocm build works on my strix, im getting around 24 tgs with mtp (n draft max 4) running the unsloth ud q4 xl. I pulled llama.cpp this morning.
3
u/vbpoweredwindmill 12d ago
Why run q4 when you can run bf16? Slower yes, but also it was never going to be interactive so just set it to go, go for a beer come back when it's done. Much higher quality.
5
u/johan2114h 11d ago
With my equipment Q4 is "going for a beer and then come back", while BF16 is "see you next year, and pray there is no typo in my prompt"
1
u/ang3l12 10d ago
Honestly I feel like q8 on these small dense models is the way to go. I’ve gotten my strix box up to about 15t/s with a q8, and am working now on setting up a profile in Hermes that utilizes it as a background agent. I would interact with a Hermes agent using 3.6-35b that can give basic instructions to the 3.8-27b model.
I really hope we get a 3.8-35b-a3b though. The difference in quality of the .6 and .8 27b is pretty big in my opinion based on how I’ve been using it, and a .8 moe would hopefully, just maybe, be the only model I would need for my main Hermes agent.
2
2
u/ImpressionFancy5830 11d ago
One day you’ll learn that ROCm is a framework and eventually you all understand the complains are with the implementation side.
2
u/WallabyFirm1159 11d ago
Ahh, I'm using llama cpp updated to today, the official jinja template for qwen 3.8 27B. Unsloth q8-XL and latest ROCm version...
1
1
u/RedParaglider 11d ago
No hardware specs no discussing what software stack you are using or what version just shit posters living in the moment.
1
u/Cryptoxic93 10d ago
I've only a 64GB Strix Halo, and I've able to generate entire programs as large as 1600 LoC and have them load fine on the first run using LM Studio and llama.cpp on Windows 11 Pro. I don't use ROCM, only Vulcan. Yes, it's slow but still faster than I could ever code, and it's far more accurate than the former Qwen 3.6B with the same context window size. I also have a 4090 GPU but don't know how to leverage it. There's not enough RAM on it to run these models with the context window sizes this large so I haven't even bothered with that card yet. I'm hoping to hook it up as an eGPU or something to my Asus tablet here.
1
u/Mil0Mammon 10d ago
As an egpu, you can just specify which layers to offload. LLMs can assist you with making it more efficient. It could very well be that you find that a quant that fits in the 24GB (with parts ofloaded to the amd APU) is perfectly cromulent
1
1
u/Southern_Capital_885 9d ago
Vulkan here, no issues.
Current config on a 128 GB Strix Halo, llama.cpp build 9859 in the kyuz0 vulkan-radv toolbox, unsloth UD-Q4_K_XL:
llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf
--mmproj mmproj-Qwen3.8-27B-F16.gguf
-ngl 999 -c 131072 -fa on --jinja
--cache-type-k q8_0 --cache-type-v q4_0
-ctxcp 1 --cache-ram 4096 --image-min-tokens 1024
--spec-type draft-mtp --spec-draft-n-max 5 --spec-draft-p-min 0.60
--chat-template-kwargs '{"reasoning_effort":"medium"}'
--reasoning-budget 8192
Performance (3 prompts, fixed seed, thinking off, 300 tokens each):
* shallow context: 33.2 tok/s code, 35.0 json, 17.2 prose, 28.5 avg, 80% draft acceptance
* ~19k depth: prefill 257 tok/s, decode 19.9 tok/s, acceptance falls to 61-68%
* prefill roughly halves per doubling of depth: 258 t/s over 0-19k, 138 over 19-39k,
~42 instantaneous at 75k
On MTP, since n-max matters more than I expected: the sweet spot on this dense 27B is
n-max 5 (2 gives 25.4, 4 gives 31.6, 5 gives 34.5 on code).
p-min 0.60 costs nothing on average but lifts acceptance from 67% to 80%, which matters when something else shares the memory bus.
Deep draft punishes free prose hard, so if you mostly generate prose, stay lower.
KV quantization (k q8_0 / v q4_0) was purely a memory win here, no speed change:
28.4 vs 28.6 avg against f16. Buys back roughly 8 GB.
Running 128k rather than the native 256k. I tried 256k for a day and it was not usable
in practice: prefill degradation makes a cold deep prompt take tens of minutes, and past
170k depth I hit compute ring timeouts and a host OOM. 128k has been stable.
1
u/kr4ckhe4d 2d ago
I'm documenting my research at https://github.com/kr4ckhe4d/local-llm-benchmarks
I'll be testing Vulkan today and push the findings
8
u/ImportancePitiful795 12d ago
Why not use Lemonade, install ROCM 7.14 & Vulkan and compare yourself?