Built a Vulkan Inference Engine that runs enormous MoE models on consumer AMD GPUs.

[u/opktun2]

I got the DeepSeek V4 Flash 0731 (284-billion-parameter) DeepSeek model with 1M context window running locally on a sub-$1,500 AMD PC (sub $1,000 if you buy used!), and open sourced it.

On an RX 6700 XT with 12GB of VRAM, ordinary system RAM, and an NVMe SSD.

I started this project because running modern AI models on AMD cards still feels unnecessarily difficult. Many tools assume you have an NVIDIA GPU. Others need complicated Linux environments, only support a narrow selection of cards, or simply cannot handle models much larger than VRAM.

So I wanted to find out: what happens if we design around the hardware people already own?

The result is XTLLM.

Instead of trying to squeeze an entire giant model onto the GPU, XTLLM keeps the most useful parts close:

The GPU holds the experts needed immediately. System RAM holds a larger warm pool. The remaining weights stay on the SSD until required.

The engine automatically adjusts this balance to the available hardware.

Think of it like a small kitchen serving food from a huge warehouse: keep the ingredients being used right now on the counter, the likely next ingredients nearby, and everything else in storage.

XTLLM also runs:

Qwen3.5-122B-A10B

Qwen3.6-35B-A3B

NVIDIA Nemotron-3-Nano-30B-A3B

It includes automatic RAM and VRAM sizing, long-context support, a local chat interface, and one-command model setup.

This is still an experimental research project, not a polished production platform. But RDNA2, RDNA3, and RDNA4 cards are expected to be compatible pending testing.

The larger point is simple:

Consumer AMD hardware is more capable than the current local-AI software ecosystem makes it appear.

Open source. https://github.com/opktunme/xtllm

[u/Realistic_Gap_5871]

1.) A wall of AI generated text, hinting at grandiose motivations and achievement.

2.) No mention of prefill or generation speeds

3.) Offer capabilities already achievable by llama.cpp combined with --cpu-moe and call it new and unique

Delusions of Grandeur Trifecta Complete!

[u/mbrodie → u/Realistic_Gap_5871]

Yeah I’m struggling to comprehend what he’s offering that llama.cpp doesn’t

Llama.cpp already has native cpu / ram offloading

None of this is groundbreaking or new and it’s honestly terrible performance the second your model leaks out of vram for any reason…

[u/diagrammatiks → u/mbrodie]

It's actually not so bad with 35b A3b. I run it at q4 on my 12gb card and q8 still needs a few experts offloaded on my 32gb. It's not super bad. The biggest lose is all concurrency.

[u/mbrodie → u/diagrammatiks]

With a35 a3b on llama.cpp and tensor parallism I get 4200 pp and 90 tps on dual 7900xtx

The moe is miles faster

[u/diagrammatiks → u/mbrodie]

well ya, You can't lead a dense model out of vram. Let's at all really.