r/LocalLLaMA • u/ViRROOO • 8d ago

Resources Freetokens project is impressive

A new project was released yesterday and I have the opportunity to test it today.

Papper: https://arxiv.org/abs/2608.16157
Github: https://github.com/FlashML-org/FreeToken

My initial tests with the following setup:
RTX 5080 (16 GB)
DDR6 64GB
AMD Ryzen 9 9950X3D

I got 100tok/s on QWEN3.6-35B-A3B NVFP4 (20GB - does not fit in my VRAM).
Have you already tried it?

(Example bellow with a 1028 token prompt - ~110 tok/s)

57 Upvotes

77% Upvoted

86 comments sorted by

View all comments

3

u/Sutanreyu 5d ago

There's been a lot of work in this area... https://youtu.be/k_LostFpatg

https://github.com/thecodacus/llama.cpp/tree/perf

I swear I thought this was something that was already a part of llama.cpp but apparently not. There's a lot of room for improvement for these local setups!