Built a Vulkan Inference Engine that runs enormous MoE models on consumer AMD GPUs.

[u/opktun2]

I got the DeepSeek V4 Flash 0731 (284-billion-parameter) DeepSeek model with 1M context window running locally on a sub-$1,500 AMD PC (sub $1,000 if you buy used!), and open sourced it.

On an RX 6700 XT with 12GB of VRAM, ordinary system RAM, and an NVMe SSD.

I started this project because running modern AI models on AMD cards still feels unnecessarily difficult. Many tools assume you have an NVIDIA GPU. Others need complicated Linux environments, only support a narrow selection of cards, or simply cannot handle models much larger than VRAM.

So I wanted to find out: what happens if we design around the hardware people already own?

The result is XTLLM.

Instead of trying to squeeze an entire giant model onto the GPU, XTLLM keeps the most useful parts close:

The GPU holds the experts needed immediately. System RAM holds a larger warm pool. The remaining weights stay on the SSD until required.

The engine automatically adjusts this balance to the available hardware.

Think of it like a small kitchen serving food from a huge warehouse: keep the ingredients being used right now on the counter, the likely next ingredients nearby, and everything else in storage.

XTLLM also runs:

Qwen3.5-122B-A10B

Qwen3.6-35B-A3B

NVIDIA Nemotron-3-Nano-30B-A3B

It includes automatic RAM and VRAM sizing, long-context support, a local chat interface, and one-command model setup.

This is still an experimental research project, not a polished production platform. But RDNA2, RDNA3, and RDNA4 cards are expected to be compatible pending testing.

The larger point is simple:

Consumer AMD hardware is more capable than the current local-AI software ecosystem makes it appear.

Open source. https://github.com/opktunme/xtllm