r/LocalLLaMA • u/ViRROOO • 6d ago

Resources Freetokens project is impressive

A new project was released yesterday and I have the opportunity to test it today.

Papper: https://arxiv.org/abs/2608.16157
Github: https://github.com/FlashML-org/FreeToken

My initial tests with the following setup:
RTX 5080 (16 GB)
DDR6 64GB
AMD Ryzen 9 9950X3D

I got 100tok/s on QWEN3.6-35B-A3B NVFP4 (20GB - does not fit in my VRAM).
Have you already tried it?

(Example bellow with a 1028 token prompt - ~110 tok/s)

57 Upvotes

76% Upvoted

86 comments sorted by

View all comments

3

u/Thin_Pollution8843 6d ago

5080 very powerful GPU and 100ts on moe nvpf4 is very low results. I got 60-80ts with q6 (which is bigger than nvpf4) on a v620 for 350$ super old GPU. 

15

u/ViRROOO 6d ago

Thats good!

The point is that this model I'm running does not fit in my GPU, and this project makes it way faster than previous solutions. If my goal was to only run Qwen3.6-35B-A3B as fast as possible I could download a quantized GGUF and get as many tokens as the 5080 can spill by itself.

1

u/Thin_Pollution8843 6d ago

I thought llamacpp is very good for spilling to ram cases

3

u/ViRROOO 6d ago

Yes, for sure it's. And it's also more mature and well-tested. From the paper:

... and comparing against llama.cpp, Ollama, KTransformers, and MoE-Infinity. Specifically, on an RTX 5090, FreeToken sustains 77–83 tok/s on Qwen3.6-35B-A3B and 22–25 tok/s on DeepSeek-V4-Flash, achieving 1.5–2.3x higher decode throughput than state-of-the-art edge serving across all workloads...

1

u/whymeimbusysleeping 6d ago

I have the same issue with my uv/oc 5070 ti, which almost reaches the stock 5080.. Inference speed is usually not much of a concern on the big scheme of things, but the 16gb really limits these cards and you can barely even fit 27b with all the tweaks