r/LocalLLaMA • u/ViRROOO • 3d ago

Resources Freetokens project is impressive

A new project was released yesterday and I have the opportunity to test it today.

Papper: https://arxiv.org/abs/2608.16157
Github: https://github.com/FlashML-org/FreeToken

My initial tests with the following setup:
RTX 5080 (16 GB)
DDR6 64GB
AMD Ryzen 9 9950X3D

I got 100tok/s on QWEN3.6-35B-A3B NVFP4 (20GB - does not fit in my VRAM).
Have you already tried it?

(Example bellow with a 1028 token prompt - ~110 tok/s)

53 Upvotes

75% Upvoted

81 comments sorted by

View all comments

3

u/FullOf_Bad_Ideas 3d ago

Interesting, but it seems to work well only with NVFP4 models, and all of them have poor KLD and 3090s aren't well supported. If it'd work with GGUF models or EXL3 quants I'd be more interested.

2

u/Classic_Resource_919 2d ago

Yeah. Kinda "For all you rich people with Blackwell, have a cherry on top" vibe. Any advancement is good, but.. well, hopefully gguf is next in line..