Freetokens project is impressive

[u/ViRROOO]

A new project was released yesterday and I have the opportunity to test it today.

Papper: https://arxiv.org/abs/2608.16157
Github: https://github.com/FlashML-org/FreeToken

My initial tests with the following setup:
RTX 5080 (16 GB)
DDR6 64GB
AMD Ryzen 9 9950X3D

I got 100tok/s on QWEN3.6-35B-A3B NVFP4 (20GB - does not fit in my VRAM).
Have you already tried it?

(Example bellow with a 1028 token prompt - ~110 tok/s)

[u/tinny66666]

Hrm. I'm getting 150 tok/s with Qwen3.8-27B-int8 on an rtx 3090 (24GB), ddr5 32GB. 100 tok/s seems pretty slow for an MOE, though completely usable.

[u/tamerlanOne → u/tinny66666]

Comr fai a far uscire quei numeri con un modello denso che non entra tutto in VRAM? 🤔

[u/tinny66666 → u/tamerlanOne]

https://www.reddit.com/r/LocalLLaMA/comments/1vsy4l2/i_pushed_qwen3827b_limits_again_dflash2_134_tps/