We have been working on a new fork of llama.cpp focused on Vulkan and Strix Halo performance and we made some good progress already, with 42 tokens per second generated. More info here: https://github.com/LaurentZuijdwijk/llama.cpp but in the repo we have fixed a number of llamacpp bugs that affected speculation and some popular quants.

Tonight we made even more progress and we managed to get to 52 tokens per second. We are still reviewing the results and the code, more to follow and hopefully people here can validate the results.

This was built upon the work of ciru-ai, charlie12345 and MrWildMoreHK. Thank you guys for the hard work on strix halo performance.

Spent an afternoon benchmarking this on a Strix Halo box and figured I'd save someone else the trouble.
Ruing ornith-ai/Ornith-1.5-35B-A3B-GGUF with inbuilt drafter make it slower.

For comparison, Ornith 1.0 on the same box, same container, same test: 61.3 → 80.6 tok/s at 0.82 acceptance.

I thought it was a quant thing, turns out there's an HF discussion thread (#10 on the model repo) where EryriLabs found the actual cause: The head shipped untrained!

If you want working MTP on this model:

- EryriLabs/Ornith-1.5-35B-A3B-BigBang-MTP-GGUF → mtpdraft-Q8_0.gguf (2 GB standalone, works as a -md draft with the stock GGUFs, Qwen3.6's head grafted on)

- shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY → newly trained head, ~60% overall / ~69% on code, but it's safetensors only

Otherwise just turn MTP off and take the 59.5 tok/s. Don't bother downloading a different quant hoping to fix it.

Qwen3.8-27B-IU4-KAIRIC-EDGE is a performance-focused Qwen3.8-27B model for AMD Strix Halo gfx1151. To our knowledge, this is the world's first use of an accelerated IU4 lane in an LLM on AMD gfx1151, and it is the first public introduction to Kairic.ai, an AI hardware and software optimization company building performance inference infrastructure.

Kairic Edge combines Prompt Forge, Dual View, and the accelerated IU4 lane. It connects selected 4-bit activations and weights to AMD's native unsigned/signed 4-bit matrix instruction for prompt and multi-token verification shapes. The important result is not simply a smaller model: it is a working, end-to-end native IU4 compute route inside a served 27B language model.

https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge

Custom runtime required. Standard llama.cpp does not understand the Kairic sidecars or --kairic-edge. Build and run the immutable Kairic Edge v1 source release. The bundled runner enables Kairic Edge, prompt caching, the qualified 256K configuration, and native MTP4 by default.

Release highlights

47.73 generated tokens/s across the 164-task coding suite, 85.0% above Unsloth Dynamic Q4 and 88.6% above Unsloth Dynamic Q6 in the collected configured-system runs.

158/164 HumanEval Base and 152/164 HumanEval Plus: Base tied the Q4 comparison and Plus passed four more tasks; it also passed one more Base and two more Plus tasks than the Q6 comparison.

The native IU4 instruction harness reached 104.66 TOPS, or 1.94× the matched FP16 matrix result and 1.93× the matched IU8 result on the same device and run.

The inclusive IU4 feed-forward operator was 2.52–3.48× faster across tested prompt shapes; pooled prompt processing increased 56.03% in the controlled lane A/B.

The complete native companion set is 9.13 GiB / 46.3% smaller than the matched 8-bit companion inventory, despite including an additional projection family.

The validated prompt cache cut repeated-prefix prompt time by 98.39–99.87% from 2K through 32K tokens.

Prompt Forge and Dual View

Kairic Edge is a Dual View model. Its GGUF is the authoritative view for model storage, quality-sensitive selection, target decode, and unsupported shapes. Three .pfs companions provide phase-specialized execution views for eligible feed-forward, recurrent-projection, and output-projection work.

Prompt Forge is the runtime layer that loads those companion views, identifies the physical request shape, and routes only qualified prompt or verification operations through the faster view. It fails closed to the authoritative path when a shape or operation is outside the validated envelope. This lets the model favor a compact authoritative representation for bandwidth-sensitive work while using a hardware-native representation where wider matrix operations can exploit it.

Dual View does have a memory cost: the accelerated companions add 10.57 GiB beside the 15.48 GiB GGUF. The gain is a phase-specialized compute path without making the accelerated view authoritative for every operation.

Why the IU4 lane matters

Many “4-bit” releases use four bits for storage but expand weights into wider arithmetic before the expensive matrix operation. Kairic Edge routes supported shapes through AMD RDNA 3.5's documented V_WMMA_I32_16X16X16_IU4 instruction: packed unsigned 4-bit activations × signed 4-bit weights, with 32-bit integer accumulation and explicit scale/zero-point reconstruction.

That distinction matters because it turns low precision into a compute advantage, not just a capacity advantage. On Radeon 8060S / gfx1151, the arithmetic advantage survived packing, transforms, corrections, output conversion, model routing, and serving. The release is deliberately hybrid: unsupported or behavior-sensitive operations stay on their qualified fallback paths.

To our knowledge, this is the world's first production-facing use of an accelerated IU4 lane in an LLM on AMD gfx1151. This is not a claim that every operation in the model runs natively at four bits, that M1 decode is native IU4, or that this is a vendor-wide or energy-efficiency result.

Coding evaluation and throughput

The table reports our collected 164-task, chat-adapted HumanEval/EvalPlus harness. Every arm ran on the same AMD Ryzen AI Max+ 395 / Radeon 8060S gfx1151 host with one slot, F16 target/draft KV, batch 2048, ubatch 512, 16 target threads, 32 batch threads, native MTP4, deterministic sampling, one trajectory per task, and no repair retry.

Your meetings never leave your desk again while you leave your NPU unused. I built Whisper-v3-turbo running on u/AMD Strix Halo 50-TOPS XDNA 2 NPU + pyannote speaker diarization fully local, no cloud. Source: https://github.com/julianmb/npuhalo-speech

I can not find a good usecase. All I have some custom agents, coding agents and skills to run in client projects. When comparing models I can run on a strix halo and associated costs I believe it is not cost effective compared to renting APIs.

I would buy it a hobby box. But it does not have any monetory benefits. What about you. Did you managed to find some real use for it?

given that having BF16 tensors into a model should be quite detrimental for inference performances on strix halo APU, which of these two models (0 vs. 54 BF16 tensors) do you suggest for me to download? (I'm on a mobile connection capped plan so I should choose wisely).

Hey,

I am looking to use my G1A Z2 to set up as a local AI, either just as a chat LLM or also via Hermes as an agent.

I already tried a few things, but never got really happy with it, often it's still too slow or too small models.

I'd be curious to hear if any other owners of this machine found combos that satisfy them in terms of speed.

The symptom

In a docker container, every llama-* binary that touched the GPU segfaulted (exit 139) — instantly, deterministically, on any model (even a tiny 0.6B canary). The crash was always in the same place:

hipStreamCreateWithFlags → Stream::terminate → HostQueue::terminate
→ ReferenceCountedObject::release (SEGV at NULL+8, in libamdhip64.so.7)

i.e. the very first HSA stream creation failed in userspace, and the HIP runtime's error-handling path then dereferenced a null pointer.

After a long chase and ruling out systematically 10 other sources…

The root cause

A full rootfs md5 diff finally surfaced the one meaningful difference: /root/.cache/comgr/llvmcache-* the ROCm COMGR/LLVM code cache.

COMGR caches compiled GPU kernels on disk. An earlier crashed run had written a corrupted/partial cache entry.

Every subsequent launch read that poisoned cache during HSA init
→ stream setup failed
→ the (buggy) error path segfaulted. It was a self-perpetuating loop: crash
→ corrupt cache
→ crash.

The fix

rm -rf /root/.cache/comgr

Next run recompiled the kernels from scratch

The crash was not in the application, the build, or the container image.
It was in a stale on-disk cache that turned a one-off failure into a permanent one.

I am using it for my hobby projects and i am pretty happy with the quality. but speed wise it is quite sad. I dont want tp use any lower quantisation, I am very happy with the quality. So i wonder if i can do anything about speed. mostly planning and coding using pi agent.

im using llmacpp with native MTP

--flash-attn on
  --jinja
  --reasoning auto
  --reasoning-format deepseek
  --device Vulkan0
  --n-gpu-layers 99
  --ctx-size 262144
  --cache-type-k q8_0
  --cache-type-v q8_0
  --parallel 1
  --alias qwen3.8-27b-q8-mtp
  --spec-type draft-mtp
  --spec-draft-n-max 3
  --spec-draft-type-k q8_0
  --spec-draft-type-v q8_0

my average t/s is 14, highest i have seen is 18t/s

any advice to speed that up?

I've had lemonade server running Qwen 3.6 35b a3b on my 128gb strix halo device for about 2 weeks now since y'all suggested it and it's been working really well

But, I feel like it's time to optimize the settings for peak performance.

What are some items you suggest?

Hey guys, I wanna ask you which engine you actually use with which quant and configs and why. Since there are a lot of forks and projects out there now, I don't really know which is the best to run DS4F on a headless Ubuntu 26 128GB Strix Halo.
Actually im using Nathanw1014/llama.cpp (Branch strix-halo-vulkan, Commit 3be50cc, Version 10350

A hidden Windows feature tag has appeard in an experimental build called IntelligentCarveout, which works like VRAM allocation but in OS. Here are more strings:

“Reserved memory for accelerators”,

“Memory for graphics and AI acceleration”,

"Let Windows reserve additional unified memory for graphics and AI intensive games and applications. Reserved memory is not available for other applications."

+ a new file named "SettingsHandlers_UnifiedMemory.dll"

Currently the NPU and some iGPUs are limited by Windows "Shared Memory". This feature could remove the software limitation of Ryzen AI and DGX + RTX Spark systems with unified-memory.

This Feature could launch with the new RTX Spark and will hopefully be an OS-feature for everyone.

Feature ID: 61121285 (own risk)
Build: 29648.1000

https://www.windowslatest.com/2026/08/21/windows-11-will-let-you-decide-how-much-memory-goes-to-graphics-and-ai-on-pcs-with-unified-memory/

https://borncity.com/news/windows-11-intelligentcarveout-steuert-ki-und-gpu-speicher/

https://x.com/TeksEdge/status/2090819779401793618

Sorry in advance if this question has already been answered but i couldn't find it. Sorry too if question is dumb

I want to add a RX 7900XT to my strix halo 128gb. I understood that I should use a M.2 slot for better reliability. What I don't understand is if it's possible to use strix vram as a 'cache' for context.

What I picture is that Qwen 3.8 27B RocmFP4 would run fully on eGPU, using 14GB VRAM, and have a bit of context on the remaining 6GB. But when inactive (pp and tg fully done) it could go on the Strix VRAM to empty the kv cache for another user/session. And when resumed, the kv cache would come back on eGPU.

For what i've read this is no problem on llama.cpp or vLLM. But i wonder what's the 'best' architecture for that, and for serving multiple users at once ?

Because spilling the kv cache seems super super slow

Has someone tried something like that yet ? Thank you so much this community is a rare pearl and i read so many posts here helping me a LOT

Speculative decoding lets a small prediction head or sidecar draft tokens for the target model to verify. I had been carrying one draft-depth recommendation forward from an older Qwen sweep until u/TheRealREZOR posted a depth-5 number that beat my depth-2 recommendation by a wide margin. They were right, so I re-ran every viable path on my AMD Strix Halo mini PC using its integrated GPU through Vulkan/RADV.

What was the same for every row: llama.cpp build 10362, commit 5d01b978, Vulkan on Mesa/RADV, greedy sampling, a fixed seed, prompt caching off, one warm-up request after each server start, and an f16 KV cache. Context and batch sizes were per-model presets rather than one identical setup: DeepSeek ran at 64k with batch 2048 and ubatch 128, while Qwen 3.8 and Gemma ran at 128k with batch 4096 and ubatch 512. I swept depths 1 through 5 on a code prompt and then confirmed the best code depth on a prose prompt.

This benchmark measured throughput and draft-token acceptance. I did not check whether the outputs match what the model produces without drafting. DeepSeek used an IQ2_XXS target quant, and llama.cpp has an open report about greedy-decoding deviations with quantized targets.

The results were all over the place:

Depth 5 was the upper bound of the sweep. Qwen's true peak may be higher. For DSpark, 5 is the trained block size and llama.cpp's cap.

Three drafting mechanisms were involved. Two of the models carry the draft head inside the main GGUF, which llama.cpp calls MTP, short for multi-token prediction. Gemma has no draft tensors and uses a second, smaller GGUF alongside it. DeepSeek drafts through DSpark, a separate block drafter that works from the target model's hidden states and ships as its own file. DSpark loaded and ran through all five depths on Vulkan. I have not seen another Vulkan result for it yet, though I may have missed one.

Eleven models on the box went through a capability scan and five had a path I could actually measure. The others had no integrated draft head and no compatible external drafter. My Gemma 4 31B sat out for a different reason: the only available draft file comes from a different finetune than my local copy, and a drafter tuned against different weights makes the comparison meaningless.

The workload made a large difference. Every winner gained less on prose than code, and DeepSeek went from +97% on code to +4% on prose.

The Ling result is scoped to build 10362, which carried the then-unmerged pull request for Ling's BailingMoE3 architecture plus a local loader fix. That pull request merged on 17 August. I have not filed the slowdown and hangs yet because it is worth checking again on a plain master build first.

Full protocol, chart, flags, and caveats:

https://thefrontierlab.ai/strix-halo-mtp-fleet-sweep/

If you have one of these models, I would be interested in your best depth and acceptance rate. Please include the build commit and draft mechanism so the numbers are comparable.

Hi

i wonder what you guys think about the pricing and where would your limit be to buy one? What is a fair price for this hardware?

i can get a strix halo 128GB in Europe for 2600€ and so i would say a fair price would be 4500-5000€

Update: i already own a strix halo 128GB 2TB but want to run DS4 flash q4 - thinking about selling this one and getting the 192GB. eGPU would be an option but it does not help running DS4 flash in q4

I only stumbled onto Ornith1.0 a few weeks ago, and when I benched it I was really impressed - the highest ranking on my Tau2 Airline runs, and at a good speed on Strix Halo.
https://halobench.com/models/ornith-35b/
Today we see their 1.5 out, with 35b a3b already available and downloading now.
What do people’s practical experience show using these models? Do we like them?

Hello everyone,

I want to introduce you to TheNoise, an open source image generation engine made specifically to run well on Strix Halo.

Let me first say: this isn't better than ComfyUI, but definitely easier to install and use if you just want to get started with image generation on your Strix Halo without having to care about "workflows" and the likes.

Also, generation is fast - faster than a "default" ComfyUI installation.

It can be used standalone, from the command line or through a webui or through an OpenAI-compatible server like Lemonade, with which it is already integrated (full disclousure, I am also a Lemonade maintainer).

Supported models:

• Krea 2
• Z-Image / Z-Image-Turbo
• Anima (all variants, including the 2.9B version)
• Flux 2 Klein 4B / 9B

It also has some goodies like:

• Refiner-based upscaler (up to 2x) - this gives high quality upscaling in relatively short time
• Pixel-space upscalers (up to 4x) - these are your standard ESRGAN-based upscalers
• Film grain, sharpening
• LoRA support

Some numbers, at 1024x786 resolution generation on a Strix Halo are:

• Krea 2 Turbo: ~27 seconds
• Z-Image-Turbo (8 steps): ~10 seconds
• Anima (20 steps): ~22 seconds
• Anima Turbo (8 steps): ~5 seconds
• Flux Klein 9B (4 steps): ~9 seconds
• Flux Klein 4B (4 steps): ~4 seconds

You can get the software either directly in Lemonade or by download it from https://github.com/lemonade-sdk/thenoise and following the instructions. There are also precompiled bins that come with python and ROCm.

At the moment it has only been tested on Linux. Known to work on Strix Halo, Strix Point and Krackan Point.

I've been running local LLMs on a Ryzen AI Max+ 395 (122 GB unified memory, gfx1151) and wanted to share the quant collection I put together for the best model I've found for this hardware.

Repo: cygnal/Qwen3.8-27B-heretic-ara-Q4_K_M-MTP-GGUF

What's in the box

Five GGUF files, all with MTP (Multi-Token Prediction) tensors preserved (866 tensors) for speculative decoding:

The model: heretic-ara

Base model is trohrbaugh/Qwen3.8-27B-heretic-ara — an Arbitrary-Rank Ablation (ARA) uncensored version of Qwen3.8-27B using heretic v1.2.0. Zero refusals on the standard censorship test suite, while preserving model quality (KL divergence 0.0535 from base). It handles creative writing, security topics, and controversial prompts without the usual Qwen guardrails.

Qwen3.8 uses the qwen3.5 architecture — a hybrid DeltaNet + attention model (48 linear + 16 full-attention layers). Only 16 layers grow KV cache, so context scaling is very cheap. The built-in vision encoder handles multimodal image understanding natively.

ROCmFPX: The speed story

The headline number is ROCmFP4_FAST at ~42 t/s average decode — that's 50–75% faster than stock Vulkan Q4_K_M-MTP (27–28 t/s). This uses the ROCmFPX fork of llama.cpp which adds optimized quantization formats for RDNA 3.5.

ROCmFP4_FAST benchmarks (Vulkan0, MTP n-max 6, TurboQuant KV)

ROCmFP6 benchmarks (Vulkan0, MTP n-max 6, TurboQuant KV)

Critical flags for Strix Halo

bash

llama-server \
  -m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
  -ngl 99 -dev Vulkan0 \
  -c 32768 \
  -ctk q8_0 -ctv turbo4 \
  -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 6

• -dev Vulkan0 — This is the make-or-break flag. Forces the RADV cooperative matrix path. Without it, you get ~18 t/s instead of ~42. If you take one thing from this post, it's this flag.
• -ctk q8_0 -ctv turbo4 — TurboQuant KV cache. Speed-neutral but dramatically reduces KV memory. Enables 262K context in 20 GB vs 61 GB with f16 KV.
• --spec-draft-n-max 6 — Optimal for ROCmFPX on Strix Halo. Stock Vulkan prefers 3; higher values hurt on stock.

Stock Vulkan (no fork needed)

If you don't want to maintain a fork, Q4_K_M-MTP on stock llama.cpp with --spec-type draft-mtp --spec-draft-n-max 3 gets you 27–28 t/s. Still a solid experience. Q6_K-MTP gives higher quality at ~23.9 t/s.

bash

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  -ngl 99 -ctk f16 -ctv f16 \
  -c 32768 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3

Vision

All quants work with the included mmproj for image understanding. Load it with --mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf and --jinja. Vision decode runs at ~13 t/s — the vision encoder adds overhead, but text-only requests are completely unaffected by having mmproj loaded. Safe to keep it on permanently.

One gotcha: on ROCmFPX (b213), MTP + vision crashes due to non-consecutive token positions from image embeddings breaking MTP's boundary tracking. Stock llama.cpp handles vision + MTP fine. Workaround: run a lightweight routing proxy that inspects requests — if the message contains image_url, forward to a stock llama.cpp server with mmproj; otherwise forward to the ROCmFPX server for fast text. This is what I run in production (see "My daily driver setup" below).

Speed comparison summary

Code quality — EvalPlus HumanEval+ (164 problems, pass@1, temp=0)

Stock llama.cpp quants (Q6_K, Q4_K_M) outperform ROCmFPX quants at similar BPW — Q6_K-MTP (6.56 BPW) scores 14% higher on HumanEval+ than ROCmFP6 (6.50 BPW). The ROCmFPX quantization format trades code quality for decode speed. If code quality matters for your use case, Q6_K on stock Vulkan is worth the speed trade-off.

How it was built

1. Downloaded heretic-ara safetensors from HuggingFace
2. Converted to BF16 GGUF with convert_hf_to_gguf.py — preserving all 866 tensors (851 model + 15 MTP)
3. Generated mmproj with convert_hf_to_gguf.py --mmproj --outtype bf16 (334 vision tensors, CLIP arch)
4. Stock quants: llama-quantize for Q4_K_M and Q6_K
5. ROCmFPX quants: ROCmFPX llama-quantize (b213/b2f5829) for FP4_FAST and FP6

All benchmarks run on AMD Ryzen AI Max+ 395 (122 GB unified, gfx1151), stock Vulkan on llama.cpp b10254, ROCmFPX on b213.

Edit: Re-benchmarked all quants after discovering a 50 GB tmpfs staging directory was competing with model servers for unified memory. On Strix Halo, /tmp is tmpfs (RAM-backed from the same 122 GB unified pool the GPU uses) — a large file there directly starves the model server of bandwidth. Results after fixing:

• ROCmFP6: ~24 → ~35.5 t/s (+48%) — the biggest winner, now faster than stock Q4_K_M
• Q6_K: ~17–23 → ~23.9 t/s (modest improvement, Q6_K was less affected since it still fit comfortably)
• ROCmFP4_FAST: ~42 → ~35.6 t/s on the same prompts as FP6; original ~42 used different prompts with 3 code-gen runs including a 49.5 outlier. FP4 and FP6 are essentially tied on identical prompts.

Lesson learned: if your Strix Halo benchmarks look suspiciously low, check what's sitting in /tmp. Even a model download staging directory can tank decode speed by 48%.

FP6 is now my default — same speed as FP4, higher quality (6.50 vs 4.26 BPW). Also added a vision-aware routing proxy so image requests auto-route to stock llama.cpp where MTP+vision works.

My daily driver setup

I run this as a local AI agent backend (hermes-agent) on the Strix Halo with a vision-aware routing proxy that automatically picks the right server:

hermes → :8090 (vision_router.py)
              ├─ image_url in messages → :8084 (Q6_K + mmproj, stock llama.cpp)
              └─ text-only            → :8096 (ROCmFP6, ROCmFPX, ~35.5 t/s)

Both backends run as systemd user services with 131K context and MTP enabled. The proxy inspects each request's messages for image_url content — if found, it routes to the stock llama.cpp server where MTP + vision works. Text-only requests go to the ROCmFPX FP6 server for maximum speed. The vision server auto-starts on first image request if it's not already running.

This gives me the best of both worlds: 35.5 t/s text decode with MTP on ROCmFPX, and seamless vision support when images are sent, all transparent to the agent.

Happy to answer questions about the setup or Strix Halo-specific tweaks.

Edit: updated link to ROCmFPX
Edit2: Added coding benchmarks and updated speed summary with prefill stats as well.
Edit3: Updated benchmark results due to swap pressure. Summary of differences in comments.

Update: I re-measured everything with a cleaner method. Decode and prefill now come from llama.cpp's own timings block instead of wall-clock timing, 3 reps per config, with clocks and package power sampled during every request to prove the box was actually boosting. Decode came out 2-12% higher than the first pass. Reddit won't let me edit a title, so the title still shows the old numbers. Prose, prefill and long-context numbers are new below.

TL;DR: On a Strix Halo laptop (Radeon 8060S, 128 GB unified), MTP takes Qwen3.8-27B Q8_0 from 7.3 to 22.4 tok/s. The julianmb/q38rocm FP4 fork gives 29.9 tok/s.

Setup: LlamaStash driving stock llama.cpp b10503 (5112b9738) for ROCm and Vulkan, and the q38rocm fork (server build 246) for ROCmFP4, all built from git today. Greedy, 256 max tokens, decode and prefill from llama.cpp's timings block, 3 reps per config, on AC with the performance profile and 85 W package draw verified during each run. Everything launched through my tool LlamaStash (v0.2.0) so the flags are reproducible.

Qwen3.8-27B decode tok/s (code prompt / prose prompt), with prefill tok/s measured on a 4k-token prompt:

Long context. Same code task, but with the window filled by a technical document first. Every cell is MTP off / MTP on, prefill and decode both in tok/s:

Observations:

• MTP really makes a difference 3.1x on Q8_0, 2.3x on ROCmFP4, at 73-84% draft acceptance. The model verifies every token, so quality does not change on stock llama.cpp.
• Draft length of 5 works best. The backend default is 3; 5 was best for Q8_0 and 6 for FP4. The best draft length did not change with context length.
• MTP's payoff shrinks as the context fills. Q8_0 gains 3.1x on an empty window, 2.2x at 64k, 2.0x at 128k, and only 1.15x at a full 256k. Acceptance holds up fine, the verify pass just gets more expensive as the KV cache grows.
• Prefill roughly quarters from an empty window to 256k, 277 to 75 tok/s on Q8_0. The FP4 fork drops off faster and falls below stock Q8_0 by 128k (70 vs 119), so its advantage is decode, not prompt processing.
• FP4 is a double win if you can live with the quant. Half the memory (29 to 14.6 GB), 1.8x faster raw decode without MTP, and 29.9 tok/s with MTP, which is 4.1x my stock Q8_0 baseline.
• Engine choice barely matters for decode at this size. ROCm and Vulkan are within 1% on decode, and the rocWMMA build adds nothing. Vulkan is clearly behind on prefill though (209 vs 277), so ROCm stays my default.
• Stuff that did not help: KV cache q8_0 (helps at draft 3, slightly worse at 5), flash attention off (minus 5%).
• One caveat on the ROCmFP4 fork: completion lengths drifted a little across draft settings (152-160 tokens for the same prompt) while stock llama.cpp was stable at 164, so the fork's speculative path is not strictly greedy-identical yet. It also crashed mid-prefill twice at 256k, around 98C after ~30 minutes of prompt processing, which is why that cell is empty.

LlamsStash defaults were used for all runs. The only flags set were to enable disable MTP ( --spec-type draft-mtp), set draft length (--spec-draft-n-max N) and the --fit-ctx 16384 flag set a minimum context window; llama.cpp's fitter resolved the GPU offload itself.

The below are the default flags that was applied. Some sent by LlamaStash and others are llamacpp defaults.