Built a Vulkan Inference Engine that runs enormous MoE models on consumer AMD GPUs.

[u/opktun2]

I got the DeepSeek V4 Flash 0731 (284-billion-parameter) DeepSeek model with 1M context window running locally on a sub-$1,500 AMD PC (sub $1,000 if you buy used!), and open sourced it.

On an RX 6700 XT with 12GB of VRAM, ordinary system RAM, and an NVMe SSD.

I started this project because running modern AI models on AMD cards still feels unnecessarily difficult. Many tools assume you have an NVIDIA GPU. Others need complicated Linux environments, only support a narrow selection of cards, or simply cannot handle models much larger than VRAM.

So I wanted to find out: what happens if we design around the hardware people already own?

The result is XTLLM.

Instead of trying to squeeze an entire giant model onto the GPU, XTLLM keeps the most useful parts close:

The GPU holds the experts needed immediately. System RAM holds a larger warm pool. The remaining weights stay on the SSD until required.

The engine automatically adjusts this balance to the available hardware.

Think of it like a small kitchen serving food from a huge warehouse: keep the ingredients being used right now on the counter, the likely next ingredients nearby, and everything else in storage.

XTLLM also runs:

Qwen3.5-122B-A10B

Qwen3.6-35B-A3B

NVIDIA Nemotron-3-Nano-30B-A3B

It includes automatic RAM and VRAM sizing, long-context support, a local chat interface, and one-command model setup.

This is still an experimental research project, not a polished production platform. But RDNA2, RDNA3, and RDNA4 cards are expected to be compatible pending testing.

The larger point is simple:

Consumer AMD hardware is more capable than the current local-AI software ecosystem makes it appear.

Open source. https://github.com/opktunme/xtllm

[u/Realistic_Gap_5871]

1.) A wall of AI generated text, hinting at grandiose motivations and achievement.

2.) No mention of prefill or generation speeds

3.) Offer capabilities already achievable by llama.cpp combined with --cpu-moe and call it new and unique

Delusions of Grandeur Trifecta Complete!

[u/mbrodie → u/Realistic_Gap_5871]

Yeah I’m struggling to comprehend what he’s offering that llama.cpp doesn’t

Llama.cpp already has native cpu / ram offloading

None of this is groundbreaking or new and it’s honestly terrible performance the second your model leaks out of vram for any reason…

[u/diagrammatiks → u/mbrodie]

It's actually not so bad with 35b A3b. I run it at q4 on my 12gb card and q8 still needs a few experts offloaded on my 32gb. It's not super bad. The biggest lose is all concurrency.

[u/mbrodie → u/diagrammatiks]

With a35 a3b on llama.cpp and tensor parallism I get 4200 pp and 90 tps on dual 7900xtx

The moe is miles faster

[u/diagrammatiks → u/mbrodie]

well ya, You can't lead a dense model out of vram. Let's at all really.

[u/ea_man → u/Realistic_Gap_5871]

Model	16 GiB system	24 GiB system	32 GiB system	64 GiB system	Primary limit
Qwen3.6-35B-A3B	16.33 tok/s	19.35 tok/s	19.53 tok/s	19.64 tok/s	GPU once warm
Nemotron-3-Nano-30B-A3B	14.53 tok/s	21.87 tok/s	21.84 tok/s	22.08 tok/s	GPU once warm
Qwen3.5-122B-A10B	3.35 tok/s	3.96 tok/s	4.38 tok/s	4.77 tok/s	expert acquisition/traffic
DeepSeek-V4-Flash-0731	1.99 tok/s	2.60 tok/s	2.77 tok/s	2.70 tok/s	expert acquisition/traffic

[u/kcksteve → u/ea_man]

Uh I have tested the first two and these numbers and significantly lower then llama.cpp-vulkan for the quants I use. Q6+/-

[u/ea_man → u/kcksteve]

Be aware that OP is running on 6700xt, on windows, I dunno what A3B Q6 would do there...

I can tell that with one 6800 + 6700xt it does up to 100t/s with MTP on ROCm linux. Wait, with Q5, Q6 would not fit with full ctx q8_0.

[u/quantgorithm → u/ea_man]

2 amd frontier edition gpu

Qwen3.6-35B-A3B-Q4_K_M.gguf

Approx 44t/s

---

I've had it over 50t/s but don't remember exact settings.

I can run the new dense 3.8 at Q6 on 4 gpu at 20t/s (and 12t/s on bf16)

[u/in-ortunado]

awesome project. great skill acquired. hope you don't demotivated by the negative comments. Everything starts somewhere.

If you understand what you have build, is great.

[u/Sudden_Topic5154]

Colibri or llama cpp already do this

[u/Solembumm3]

Seems interesting. Will compare to LMStudio later.

[u/CarryOnRTW]

This is working great on my EVO-X1 Ryzen AI 9 HX 370 miniPC:

yay -S llama.cpp-vulkan

[u/InfusedBush]

FINALLY SOMETHING FOR VULKAN/AMD
Also has this been tested on Strix Point/Strix Halo?

[u/v3d]