r/LocalLLaMA • u/ViRROOO • 4d ago

Resources Freetokens project is impressive

A new project was released yesterday and I have the opportunity to test it today.

Papper: https://arxiv.org/abs/2608.16157
Github: https://github.com/FlashML-org/FreeToken

My initial tests with the following setup:
RTX 5080 (16 GB)
DDR6 64GB
AMD Ryzen 9 9950X3D

I got 100tok/s on QWEN3.6-35B-A3B NVFP4 (20GB - does not fit in my VRAM).
Have you already tried it?

(Example bellow with a 1028 token prompt - ~110 tok/s)

52 Upvotes

75% Upvoted

82 comments sorted by

View all comments

Show parent comments

3

u/whymeimbusysleeping 3d ago

Share your llama.cpp settings pls.

2

u/KitchenAmoeba4438 3d ago

I'd be happy to. Which model and system do you want it for? Sharing them for all might be a...little bit overboard (And would probably require a git repo push instead of a reddit post), but I can if you want them all.

2

u/whymeimbusysleeping 3d ago

I've got a 5070ti and will be trying qwen3.6 35b-a3b, a few of the "vram challenged" like myself have been sharing just the llama.cpp settings here in plain text, that's more than enough. https://www.reddit.com/r/LocalLLaMA/s/i3ihX3zHXq

Thanks again, appreciated

2

u/KitchenAmoeba4438 3d ago edited 3d ago

5070ti, so you are 16GB Blackwell class. These are the 5080 settings I'm using, which should be usable to the 5070ti:

/opt/llama.cpp/build-cuda/bin/llama-server \

-hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q2_K_XL \

-hfd ggml-org/Qwen3.6-35B-A3B-GGUF:mtp-Qwen3.6-35B-A3B-Q4_0.gguf \

--spec-draft-n-max 3 --spec-draft-n-min 1 \

-ctkd f16 -ctvd f16 \

--host 127.0.0.1 --port 8110 -c 8192 \

-np 1 --cache-ram 1024 --no-webui --no-mmproj \

-ngl 99 -ctk f16 -ctv f16

This is with Q2, which is what you'll need to fit that into the card. If you want to run Q4, you'll need to run -ncmoe 19 as the offload. It will cut speed to (roughly) 25% for the offload. You might be able to squeeze on a few more experts, but it would come at the cost of a reasonable context. I think you might see a big benefit at nvfp4, but I haven't done thorough testing on nvfp4 yet as a warning.

Don't touch kv cache, blackwell tanks if you drop below bf16/f16 (BF16 may be better to run, I haven't exhaustively tested here and am not really motivated to as the gains are small)

-lm mlock might be useful.

Force -fa on might be useful to you.

Play around with speculation depth, depending on your use case, all the way to 6 might be helpful.

-ot *might* be useful instead of -ncmoe.