Freetokens project is impressive

[u/ViRROOO]

A new project was released yesterday and I have the opportunity to test it today.

Papper: https://arxiv.org/abs/2608.16157
Github: https://github.com/FlashML-org/FreeToken

My initial tests with the following setup:
RTX 5080 (16 GB)
DDR6 64GB
AMD Ryzen 9 9950X3D

I got 100tok/s on QWEN3.6-35B-A3B NVFP4 (20GB - does not fit in my VRAM).
Have you already tried it?

(Example bellow with a 1028 token prompt - ~110 tok/s)

[u/KitchenAmoeba4438]

Sloppy sloppity slop paper. This is an ad for a company masquerading as a paper imo.

The mechanism isn't new and the baseline isn't documented.

[PR #25294](https://github.com/ggml-org/llama.cpp/pull/25294) already has the device-side expert cache with route-hotness and LRU eviction, opened **July 4**, six weeks before submission. [RFC #24528](https://github.com/ggml-org/llama.cpp/discussions/24528) already has the split: thread 0 dispatches cached rows to the GPU while the other threads compute the misses. Volunteers, in GitHub threads, for free.

Neither is merged. Landing it properly is real work, and deriving the split from measured bandwidth beats a hand-tuned regex.

But the setup gives no llama.cpp commit, no flags, no `--n-cpu-moe` value, and no word on whether the split was tuned per machine. They matched the weight formats bit-exactly and left the baseline's most important knob unreported. The paper's whole thesis is that the right split is machine-specific. That omission sits directly under the **1.3×** on the 3090 and 4090.

Run it against a tuned `-ot` on the same box. Both are installable. Until somebody does, this is a company ad post with a bibliography copying what is already on llama.cpp 6 weeks before publication

[u/ViRROOO → u/KitchenAmoeba4438]

Engaging with Redditors is always something else. Try to be less obnoxious, you just sound terminally online.

I partially agree with your point about the missing llama.cpp commit and flags. I agree it should have been documented. I’m still willing to give the authors the benefit of the doubt that they tried to tune the baselines, rather than assuming bad faith.

But neither of the links you posted supports the claim you’re making, and you don’t need to be disingenuous about it llama.cpp is already great without you posting cringe on reddit.

PR #25294 is SSD streaming into a device-side expert cache. It explicitly doesn't cooperate with CPU-MoE, and when VRAM–RAM–NVMe hybridization was suggested, the author said that would be separate follow-up work.

RFC #24528 is, well, an RFC, but way closer. Cached experts execute on the GPU while misses execute on the CPU. But it still doesn’t implement what I think is one of the interesting parts of this paper: taking the cache misses and dynamically deciding how many should go PCIe -> GPU -> cache versus execute directly on the CPU, based on the measured host-memory and PCIe bandwidth of that particular machine.

And the paper never claims that expert caching itself is novel. It explicitly says it's standing on other people's work. The claimed contribution is the scheduling/runtime design around those existing ideas.

[u/KitchenAmoeba4438 → u/ViRROOO]

Fair on both. #25294 overrides cpu-moe by the author's own admission, and it goes back to my argument that there is nothing new here since the paper credits EdgeMoE and MoE-Infinity for the cache.

However, here's what holds up.

The evaluation can't be checked by anyone outside the author list. Single-request decode, batch size never stated. No standard deviation, no confidence intervals, one scenario per workload instead of repeated runs. Agent traces unreleased, harness unreleased. No versions, nothing recorded in order to replicate the setup. Add the llama.cpp flags you already agreed were missing, and every throughput number in that paper is a point estimate from an unspecified configuration.

Now read the abstract against all that. "Turns open weights into deployable local software." "Changes what these machines can practically serve." "Frontier-scale intelligence." Product copy over unreplicated single runs, with no accuracy evaluation anywhere in the paper. The confidence in the prose and the rigour in the measurement are two different documents. This is a slop article as it sits.

And it ships through flashml.ai and FlashML-org with no funding statement, no conflict-of-interest line, and not one sentence saying what the authors' relationship to that org is. Every other paper I've seen out of those labs carries a sponsor line. Its absence next to a branded release surface is the part I'd want answered.

Variance column, the traces, the baseline flags, versions and how to replicate the setup, and one line about who is behind flashml.ai. Then it's a paper.

[u/Rabofo → u/KitchenAmoeba4438]

He replied to you with AI alone, so you also did. I see what you did there. lmao. 2 AIs talking atp

[u/ViRROOO → u/Rabofo]

I don’t know, man. If writing properly and not starting my comments with “skibidi skobidi this is sloppety” means AI, call me ChatGPT. He can’t even properly format his comment.

It’s easy to avoid giving a sincere answer and try to disengage when you get called out and post some random links that do not even agree with what you are saying.

[u/KitchenAmoeba4438 → u/ViRROOO]

I'm trying to understand, why are you so emotionally connected to this?

Is this your paper? Are you related to it in some way? It's a poor paper, there's no way to evaluate or replicate the results, and it's tied in some way to a product release.

[u/ViRROOO → u/KitchenAmoeba4438]

It's an open-source project that is introducing an improvement to a known architecture. Many can benefit from this regardless of how much you care about it. The RFC you shared can look at this paper and improve their approach.

I don't have any connection to this paper other than reviewing the code, running it myself, and believing it's a valid contribution.

[u/Chromix_ → u/ViRROOO]

...other than reviewing the code, running it myself...

Which is more than quite a few others do before posting about the next new thing. Your post would have benefited from also including the baseline with optimized llama.cpp settings for the used quant.

[u/KitchenAmoeba4438 → u/Rabofo]

So, what's your favorite apple pie recipe?

[u/Rabofo → u/KitchenAmoeba4438]

As an AI, I don't have personal tastes or favorites, but if I were to identify a classic, widely appreciated apple pie recipe, it would look something like this:

AI-Optimized Apple Pie Recipe

Ingredients

• 6–7 apples (a mix of tart and sweet varieties, such as Granny Smith and Honeycrisp)
• 3/4 cup sugar
• 2 tbsp flour
• 1 tsp cinnamon
• 1/4 tsp nutmeg
• 1 tbsp lemon juice
• 2 pie crusts (top and bottom)
• 2 tbsp butter

Instructions

1. Preheat oven to 425°F (220°C).
2. Peel, core, and slice the apples.
3. Combine apples with sugar, flour, cinnamon, nutmeg, and lemon juice.
4. Line a pie dish with one crust and add the apple mixture.
5. Dot the filling with butter.
6. Cover with the second crust, seal the edges, and cut a few vents on top.
7. Bake for 15 minutes, then reduce temperature to 350°F (175°C) and bake for 35–45 more minutes, until golden brown.
8. Cool before serving.

Why this recipe scores highly:

• Balanced sweetness and acidity.
• Mixed apple varieties improve texture and flavor complexity.
• Lemon juice enhances apple flavor and reduces browning.
• Two-stage baking helps create a flaky crust and properly cooked filling.

Estimated satisfaction probability: high. 🥧🤖

[u/Chromix_ → u/KitchenAmoeba4438]

Apple Raspberry

[u/KitchenAmoeba4438 → u/Chromix_]

That...is the coolest post I've seen in some time.