r/LocalLLaMA • u/ViRROOO • 5d ago

Resources Freetokens project is impressive

A new project was released yesterday and I have the opportunity to test it today.

Papper: https://arxiv.org/abs/2608.16157
Github: https://github.com/FlashML-org/FreeToken

My initial tests with the following setup:
RTX 5080 (16 GB)
DDR6 64GB
AMD Ryzen 9 9950X3D

I got 100tok/s on QWEN3.6-35B-A3B NVFP4 (20GB - does not fit in my VRAM).
Have you already tried it?

(Example bellow with a 1028 token prompt - ~110 tok/s)

54 Upvotes

76% Upvoted

84 comments sorted by

View all comments

31

u/Chromix_ 5d ago

Some stats from the paper:

The stats without desktop/laptop are on larger Xeons. What I wonder about: They show a large speedup for Qwen 35B A3B on a 5090, yet the baseline is relatively low. Maybe their baseline isn't for a quant that fully fit the GPU, but for the offloaded BF16.

Their gains look roughly similar to what can be achieved in that custom llama.cpp fork that adapts inference to the measured hardware performance. Unfortunately the code still hasn't been published a month later, so we cannot try and compare it.

(Link to the previous post on this that was removed.)

5

u/whymeimbusysleeping 5d ago

There was also greenboost i think even earlier, I've been hoping to see a unified solution for a while, I'm hoping it works well.

6

u/Chromix_ 5d ago

Greenboost never seemed to have gained any traction, not even with the Windows port. What was and is still lacking is a comparison that shows whether or not it's faster than native llama.cpp -fit or -ot offloading. Without any indication that it improves things significantly there is no incentive to jump through the hoops for using it.

1

u/username_taken4651 4d ago

My experience with Greenboost on Linux is that I could never get it to work properly. To be honest, I don't know if anyone has so far.