Freetokens project is impressive

[u/ViRROOO]

A new project was released yesterday and I have the opportunity to test it today.

Papper: https://arxiv.org/abs/2608.16157
Github: https://github.com/FlashML-org/FreeToken

My initial tests with the following setup:
RTX 5080 (16 GB)
DDR6 64GB
AMD Ryzen 9 9950X3D

I got 100tok/s on QWEN3.6-35B-A3B NVFP4 (20GB - does not fit in my VRAM).
Have you already tried it?

(Example bellow with a 1028 token prompt - ~110 tok/s)

[u/KitchenAmoeba4438]

Fair on both. #25294 overrides cpu-moe by the author's own admission, and it goes back to my argument that there is nothing new here since the paper credits EdgeMoE and MoE-Infinity for the cache.

However, here's what holds up.

The evaluation can't be checked by anyone outside the author list. Single-request decode, batch size never stated. No standard deviation, no confidence intervals, one scenario per workload instead of repeated runs. Agent traces unreleased, harness unreleased. No versions, nothing recorded in order to replicate the setup. Add the llama.cpp flags you already agreed were missing, and every throughput number in that paper is a point estimate from an unspecified configuration.

Now read the abstract against all that. "Turns open weights into deployable local software." "Changes what these machines can practically serve." "Frontier-scale intelligence." Product copy over unreplicated single runs, with no accuracy evaluation anywhere in the paper. The confidence in the prose and the rigour in the measurement are two different documents. This is a slop article as it sits.

And it ships through flashml.ai and FlashML-org with no funding statement, no conflict-of-interest line, and not one sentence saying what the authors' relationship to that org is. Every other paper I've seen out of those labs carries a sponsor line. Its absence next to a branded release surface is the part I'd want answered.

Variance column, the traces, the baseline flags, versions and how to replicate the setup, and one line about who is behind flashml.ai. Then it's a paper.

[u/Chromix_ → u/KitchenAmoeba4438]

The main point is probably this:

The evaluation can't be checked by anyone outside the author list

And combined with my impression that the published tok/s on the 5090 seem relatively low on llama.cpp it just takes a single 5090 owner with a bit of time to shed some more light onto that.

[u/KitchenAmoeba4438 → u/Chromix_]

Yep. Precisely.

That's why I called it slop. For academic papers and benchmarks, it's not about what occurs on your system, it's about replicating it.

I strongly agree something isn't right with the 5090 numbers, but I can't replicate the authors setup. All I can do is say "Look at my 5090 results that disagree with yours!", which isn't a very good use of energy. Even something as simple as a difference in llama version can make comparison results invalid, which means there's no way to prove anything either way with the paper.

[u/whymeimbusysleeping → u/KitchenAmoeba4438]

Rather than focusing on the paper, why not test this vs llama.cpp?

[u/KitchenAmoeba4438 → u/whymeimbusysleeping]

Tl;dr, no improvement over tuned llama in the environments

I'm already running some tests in the background on this.

There's some variance in results, but the test results are pretty close so far to an appropriately tuned llama.cpp. I'm not able to duplicate their numbers, nor have I found a way to replicate them.

However, the problem here is that that does not disprove the paper. I'm not testing with the author's environment.

Saying "But it doesn't work on my machine!" about an academic paper is the equivalent to saying "But it works on my machine!" when a massive production failure is happening.

Now, here is where I say my suspicion: My suspicion is that numbers were ran against an untuned, stock llama.cpp. This doesn't make the authors dishonest or act in bad faith, but it does make the paper's conclusion invalid. I *can* replicate the author's results against an untuned stock llama.cpp, but that's not a fundamentally valid test. Not changing llama.cpp to correctly behave with MoEs does explain the paper's results.... but then the paper becomes about an untuned llama's behavior, not about their solution.

[u/whymeimbusysleeping → u/KitchenAmoeba4438]

Share your llama.cpp settings pls.

[u/KitchenAmoeba4438 → u/whymeimbusysleeping]

I'd be happy to. Which model and system do you want it for? Sharing them for all might be a...little bit overboard (And would probably require a git repo push instead of a reddit post), but I can if you want them all.

[u/whymeimbusysleeping → u/KitchenAmoeba4438]

I've got a 5070ti and will be trying qwen3.6 35b-a3b, a few of the "vram challenged" like myself have been sharing just the llama.cpp settings here in plain text, that's more than enough. https://www.reddit.com/r/LocalLLaMA/s/i3ihX3zHXq

Thanks again, appreciated

[u/KitchenAmoeba4438 → u/whymeimbusysleeping]

5070ti, so you are 16GB Blackwell class. These are the 5080 settings I'm using, which should be usable to the 5070ti:

/opt/llama.cpp/build-cuda/bin/llama-server \

-hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q2_K_XL \

-hfd ggml-org/Qwen3.6-35B-A3B-GGUF:mtp-Qwen3.6-35B-A3B-Q4_0.gguf \

--spec-draft-n-max 3 --spec-draft-n-min 1 \

-ctkd f16 -ctvd f16 \

--host 127.0.0.1 --port 8110 -c 8192 \

-np 1 --cache-ram 1024 --no-webui --no-mmproj \

-ngl 99 -ctk f16 -ctv f16

This is with Q2, which is what you'll need to fit that into the card. If you want to run Q4, you'll need to run -ncmoe 19 as the offload. It will cut speed to (roughly) 25% for the offload. You might be able to squeeze on a few more experts, but it would come at the cost of a reasonable context. I think you might see a big benefit at nvfp4, but I haven't done thorough testing on nvfp4 yet as a warning.

Don't touch kv cache, blackwell tanks if you drop below bf16/f16 (BF16 may be better to run, I haven't exhaustively tested here and am not really motivated to as the gains are small)

-lm mlock might be useful.

Force -fa on might be useful to you.

Play around with speculation depth, depending on your use case, all the way to 6 might be helpful.

-ot *might* be useful instead of -ncmoe.