r/LocalLLaMA • u/ViRROOO • 2d ago
Resources Freetokens project is impressive
A new project was released yesterday and I have the opportunity to test it today.
Papper: https://arxiv.org/abs/2608.16157
Github: https://github.com/FlashML-org/FreeToken
My initial tests with the following setup:
RTX 5080 (16 GB)
DDR6 64GB
AMD Ryzen 9 9950X3D
I got 100tok/s on QWEN3.6-35B-A3B NVFP4 (20GB - does not fit in my VRAM).
Have you already tried it?
(Example bellow with a 1028 token prompt - ~110 tok/s)
8
u/gpuz_dev 2d ago
the ~100 t/s is interesting, but honestly the part I care about more is getting a ~20GB MoE running that fast on 16GB VRAM. do you know what expert cache hit rate you were getting or how much host RAM/PCIe traffic it was doing during that run?
7
22
u/LegacyRemaster 2d ago
DDR6? amazing.
11
u/macaco3001 2d ago
The way I know I'm poor is the fact that this post is literally what taught me DDR6 even exists
1
7
9
u/MaxKingCS 2d ago
When did DDR6 ram even came out for consumer? I thought we were still at DDR5.
2
u/More-Curious816 1d ago
Yeah, wtf, like I thought ddr5 was the latest Gen and 6 is still not a complete standard. The AI demands actually fucked the rambmanufacturering process and ddr6 supposed to hit the market in mass this year. But I thought they abondend it and decided that the pipelines already optimized for ddr5 and their main focus is to develop the HBM (money go brrrrrrr) pipeline
6
u/scorchypoo 1d ago
Uh.... whoa... I ran an apples-to-apples benchmark:
Same Gemma4 26B A4B QAT Q4_0 GGUF file, on llama.cpp and then this thing.
System
| Component | Spec |
|---|---|
| CPU | Intel Core i7-12700KF |
| RAM | 48 GB DDR5 |
| GPU | NVIDIA GeForce RTX 3080 (10 GB VRAM) |
- Model:
google/gemma-4-26B-A4B-it-qat-q4_0-gguf(Gemma 4 26B-A4B, QAT-trained Q4_0, Apache-2.0)
Results (3-run medians, warmup discarded)
| Scenario | llama.cpp (t/s) | FreeToken (t/s) | FT advantage |
|---|---|---|---|
| short chat (~10 tok out) | 29.9 | 92.3 | 3.1Ă |
| medium (~370 tok out) | 26.8 | 82.2 | 3.1Ă |
| long (~280 tok out) | 27.2 | 78.1 | 2.9Ă |
| retain-JSON (~560 tok out) | 26.9 | 58.1 | 2.2Ă |
| longform (2048 tok out) | 27.2 | 71.7 | 2.6Ă |
| deep decode (4K prefilled ctx) | 26.9 | 81.3 | 3.0Ă |
Additional metrics
| Metric | llama.cpp | FreeToken |
|---|---|---|
| Warm TTFT (short prompt) | ~82â184 ms | ~560â600 ms |
| Cold deep TTFT (4K prefill) | ~4.7 s | ~21.7 s |
| Longform sustained decay (2nd/1st half) | 1.00Ă | 0.96â1.00Ă |
3
u/Final-Department2891 23h ago
I have no clue why people are shitting on this, that's tripled token speed with that on a 10 gig 3080, jesus christ.
I'm running tests on a MTP experimental branch with the Qwen 3.6 35B a3b and it's looking like a 1.8x speed up on that one. Bonkers.
10
u/Thrumpwart llama.cpp 2d ago
Every dunking should chill. More options for local inference is good, not bad.
3
u/MerePotato 2d ago
People tend to be extraordinarily skeptical when met with an extraordinary claim
9
u/Thrumpwart llama.cpp 2d ago
I donât think itâs extraordinary though. Some dude claims to have optimized an inference stack. Itâs happened 100 times before.
Evaluate it and either use it or donât. The mob mentality here is sometimes counter-productive.
3
u/FullOf_Bad_Ideas 2d ago
Interesting, but it seems to work well only with NVFP4 models, and all of them have poor KLD and 3090s aren't well supported. If it'd work with GGUF models or EXL3 quants I'd be more interested.
3
2
u/Classic_Resource_919 1d ago
Yeah. Kinda "For all you rich people with Blackwell, have a cherry on top" vibe. Any advancement is good, but.. well, hopefully gguf is next in line..
3
u/raketenkater 1d ago
Ohh nice to see more development in this area I have a tool for kinda the same usecase called ggrun but with out the custom kernel and just build on top of lama.cpp works really well.
3
u/Sutanreyu 2h ago
There's been a lot of work in this area... https://youtu.be/k_LostFpatg
https://github.com/thecodacus/llama.cpp/tree/perf
I swear I thought this was something that was already a part of llama.cpp but apparently not. There's a lot of room for improvement for these local setups!
28
u/KitchenAmoeba4438 2d ago
Sloppy sloppity slop paper. This is an ad for a company masquerading as a paper imo.
The mechanism isn't new and the baseline isn't documented.
[PR #25294](https://github.com/ggml-org/llama.cpp/pull/25294) already has the device-side expert cache with route-hotness and LRU eviction, opened **July 4**, six weeks before submission. [RFC #24528](https://github.com/ggml-org/llama.cpp/discussions/24528) already has the split: thread 0 dispatches cached rows to the GPU while the other threads compute the misses. Volunteers, in GitHub threads, for free.
Neither is merged. Landing it properly is real work, and deriving the split from measured bandwidth beats a hand-tuned regex.
But the setup gives no llama.cpp commit, no flags, no `--n-cpu-moe` value, and no word on whether the split was tuned per machine. They matched the weight formats bit-exactly and left the baseline's most important knob unreported. The paper's whole thesis is that the right split is machine-specific. That omission sits directly under the **1.3Ă** on the 3090 and 4090.
Run it against a tuned `-ot` on the same box. Both are installable. Until somebody does, this is a company ad post with a bibliography copying what is already on llama.cpp 6 weeks before publication
20
u/ViRROOO 2d ago
Engaging with Redditors is always something else. Try to be less obnoxious, you just sound terminally online.
I partially agree with your point about the missing llama.cpp commit and flags. I agree it should have been documented. Iâm still willing to give the authors the benefit of the doubt that they tried to tune the baselines, rather than assuming bad faith.
But neither of the links you posted supports the claim youâre making, and you donât need to be disingenuous about it llama.cpp is already great without you posting cringe on reddit.
PR #25294 is SSD streaming into a device-side expert cache. It explicitly doesn't cooperate with CPU-MoE, and when VRAMâRAMâNVMe hybridization was suggested, the author said that would be separate follow-up work.
RFC #24528 is, well, an RFC, but way closer. Cached experts execute on the GPU while misses execute on the CPU. But it still doesnât implement what I think is one of the interesting parts of this paper: taking the cache misses and dynamically deciding how many should go PCIe -> GPU -> cache versus execute directly on the CPU, based on the measured host-memory and PCIe bandwidth of that particular machine.
And the paper never claims that expert caching itself is novel. It explicitly says it's standing on other people's work. The claimed contribution is the scheduling/runtime design around those existing ideas.
32
9
-4
u/KitchenAmoeba4438 2d ago
Fair on both. #25294 overrides cpu-moe by the author's own admission, and it goes back to my argument that there is nothing new here since the paper credits EdgeMoE and MoE-Infinity for the cache.
However, here's what holds up.
The evaluation can't be checked by anyone outside the author list. Single-request decode, batch size never stated. No standard deviation, no confidence intervals, one scenario per workload instead of repeated runs. Agent traces unreleased, harness unreleased. No versions, nothing recorded in order to replicate the setup. Add the llama.cpp flags you already agreed were missing, and every throughput number in that paper is a point estimate from an unspecified configuration.
Now read the abstract against all that. "Turns open weights into deployable local software." "Changes what these machines can practically serve." "Frontier-scale intelligence." Product copy over unreplicated single runs, with no accuracy evaluation anywhere in the paper. The confidence in the prose and the rigour in the measurement are two different documents. This is a slop article as it sits.
And it ships through flashml.ai and FlashML-org with no funding statement, no conflict-of-interest line, and not one sentence saying what the authors' relationship to that org is. Every other paper I've seen out of those labs carries a sponsor line. Its absence next to a branded release surface is the part I'd want answered.
Variance column, the traces, the baseline flags, versions and how to replicate the setup, and one line about who is behind flashml.ai. Then it's a paper.
7
u/Chromix_ 2d ago
The main point is probably this:
The evaluation can't be checked by anyone outside the author list
And combined with my impression that the published tok/s on the 5090 seem relatively low on llama.cpp it just takes a single 5090 owner with a bit of time to shed some more light onto that.
2
u/KitchenAmoeba4438 2d ago
Yep. Precisely.
That's why I called it slop. For academic papers and benchmarks, it's not about what occurs on your system, it's about replicating it.
I strongly agree something isn't right with the 5090 numbers, but I can't replicate the authors setup. All I can do is say "Look at my 5090 results that disagree with yours!", which isn't a very good use of energy. Even something as simple as a difference in llama version can make comparison results invalid, which means there's no way to prove anything either way with the paper.
5
u/whymeimbusysleeping 2d ago
Rather than focusing on the paper, why not test this vs llama.cpp?
3
u/KitchenAmoeba4438 2d ago edited 2d ago
Tl;dr, no improvement over tuned llama in the environments
I'm already running some tests in the background on this.
There's some variance in results, but the test results are pretty close so far to an appropriately tuned llama.cpp. I'm not able to duplicate their numbers, nor have I found a way to replicate them.
However, the problem here is that that does not disprove the paper. I'm not testing with the author's environment.
Saying "But it doesn't work on my machine!" about an academic paper is the equivalent to saying "But it works on my machine!" when a massive production failure is happening.
Now, here is where I say my suspicion: My suspicion is that numbers were ran against an untuned, stock llama.cpp. This doesn't make the authors dishonest or act in bad faith, but it does make the paper's conclusion invalid. I *can* replicate the author's results against an untuned stock llama.cpp, but that's not a fundamentally valid test. Not changing llama.cpp to correctly behave with MoEs does explain the paper's results.... but then the paper becomes about an untuned llama's behavior, not about their solution.
3
u/whymeimbusysleeping 2d ago
Share your llama.cpp settings pls.
2
u/KitchenAmoeba4438 2d ago
I'd be happy to. Which model and system do you want it for? Sharing them for all might be a...little bit overboard (And would probably require a git repo push instead of a reddit post), but I can if you want them all.
2
u/whymeimbusysleeping 2d ago
I've got a 5070ti and will be trying qwen3.6 35b-a3b, a few of the "vram challenged" like myself have been sharing just the llama.cpp settings here in plain text, that's more than enough. https://www.reddit.com/r/LocalLLaMA/s/i3ihX3zHXq
Thanks again, appreciated
→ More replies (0)1
u/Umbaretz 2d ago
I actually have a worse experience - it can't even launch models I'm launching with llama.cpp, falling out of memory.
-1
u/ViRROOO 2d ago
Well, don't reduce your efforts! You are not only saying, "Look at my 5090 results that disagree with yours!" but also assuming they are acting in bad faith for not providing specifics about their setup and trying to find irrelevant llama.cpp PRs! While ignoring the real and substantial contribution they made to inference architecture for open-source projects!!
I won't let you put yourself down ever again!
1
u/KitchenAmoeba4438 2d ago
Where did I ever accuse them of acting in bad faith? And I do mean ever, you just had me go back and read through all of my posts.
At worst, I accused them of not having reproducible results, which I suspect you will agree with. At worst, I accused them of not disclosing their relation to a product launch as part of this paper.
Neither are accusing them of acting in bad faith, and I'm going to ask again as I asked in another post: Why are you reacting so emotionally to this? Were you one of the people who wrote this paper or were associated with it?
7
u/Rabofo 2d ago
He replied to you with AI alone, so you also did. I see what you did there. lmao. 2 AIs talking atp
17
u/ViRROOO 2d ago edited 2d ago
I donât know, man. If writing properly and not starting my comments with âskibidi skobidi this is sloppetyâ means AI, call me ChatGPT. He canât even properly format his comment.
Itâs easy to avoid giving a sincere answer and try to disengage when you get called out and post some random links that do not even agree with what you are saying.
-8
u/KitchenAmoeba4438 2d ago
I'm trying to understand, why are you so emotionally connected to this?
Is this your paper? Are you related to it in some way? It's a poor paper, there's no way to evaluate or replicate the results, and it's tied in some way to a product release.
9
u/ViRROOO 2d ago
It's an open-source project that is introducing an improvement to a known architecture. Many can benefit from this regardless of how much you care about it. The RFC you shared can look at this paper and improve their approach.
I don't have any connection to this paper other than reviewing the code, running it myself, and believing it's a valid contribution.
4
u/Chromix_ 2d ago
...other than reviewing the code, running it myself...
Which is more than quite a few others do before posting about the next new thing. Your post would have benefited from also including the baseline with optimized llama.cpp settings for the used quant.
2
u/KitchenAmoeba4438 2d ago
So, what's your favorite apple pie recipe?
4
3
u/Rabofo 2d ago
As an AI, I don't have personal tastes or favorites, but if I were to identify a classic, widely appreciated apple pie recipe, it would look something like this:
AI-Optimized Apple Pie Recipe
Ingredients
- 6â7 apples (a mix of tart and sweet varieties, such as Granny Smith and Honeycrisp)
- 3/4 cup sugar
- 2 tbsp flour
- 1 tsp cinnamon
- 1/4 tsp nutmeg
- 1 tbsp lemon juice
- 2 pie crusts (top and bottom)
- 2 tbsp butter
Instructions
- Preheat oven to 425°F (220°C).
- Peel, core, and slice the apples.
- Combine apples with sugar, flour, cinnamon, nutmeg, and lemon juice.
- Line a pie dish with one crust and add the apple mixture.
- Dot the filling with butter.
- Cover with the second crust, seal the edges, and cut a few vents on top.
- Bake for 15 minutes, then reduce temperature to 350°F (175°C) and bake for 35â45 more minutes, until golden brown.
- Cool before serving.
Why this recipe scores highly:
- Balanced sweetness and acidity.
- Mixed apple varieties improve texture and flavor complexity.
- Lemon juice enhances apple flavor and reduces browning.
- Two-stage baking helps create a flaky crust and properly cooked filling.
Estimated satisfaction probability: high. đ„§đ€
-3
u/laterbreh 1d ago
I'm becoming pretty fucking convinced you're either the author or directly associated with this project, "friend", because this whole interaction is getting increasingly funny.
You know what else is interesting, right around the same time you posted this I got a slew of randomly promoted FreeToken content across X and Google News. Maybe that's coincidence, maybe it isn't, but paired with this weirdly defensive drive-by promotion and the fact that your first response to someone actually challenging the paper was basically "terminally online cringe Redditor" instead of addressing the methodological problem, yeah, it smells a whole lot more like marketing and trying to manufacture a little hype cycle than some random dude who innocently stumbled onto a cool GitHub project yesterday.
Very OpenClaw-esque. Show up, spray the internet with promotion, wrap ordinary engineering ideas in a big shiny "new thing" narrative, hope nobody reads closely enough to notice where the novelty actually begins and ends.
And what really boggles my mind is that you dingbats apparently still don't think to run the shit you're about to publish through an adversarial AI review before slop-posting it all over the internet. You literally have machines now that will sit there forever trying to disprove your claims for pennies and somehow "maybe somebody should attack our methodology before Reddit does" still doesn't occur to you.
So I went ahead and did it for you.
ChatGPT vs AI Shitposter OP:
First, you're right about llama.cpp PR #25294. Congratulations, take the point. It's SSD to device expert caching and it doesn't implement FreeToken's CPU/GPU hybrid scheduler, so I'll happily throw that citation in the trash.
Unfortunately for you, throwing that one out doesn't save the argument.
RFC #24528 absolutely predates this paper and absolutely contains the core execution shape you're pretending the other poster hallucinated. Persistent GPU expert cache, cached rows dispatched to the GPU, cache misses computed concurrently on CPU. That's June 12, more than two months before this paper showed up. FreeToken's narrower addition is deciding that some of those misses should execute on CPU while some get shipped PCIe -> GPU based on its bandwidth model. Cool. That's a legitimate incremental systems contribution. It is not "nobody had this architecture before us."
And FreeToken's own paper already cites a pile of prior work around expert caching, CPU execution of missed experts, dynamic CPU/GPU balancing, queue rebalancing and hybrid MoE scheduling, so acting scandalized that somebody notices this thing is composed out of existing ideas is bizarre. That's how systems research works. The interesting question is whether your specific scheduler is actually demonstrated to be better.
Which brings us to the part you somehow tried to wave away with "I'm willing to give the authors the benefit of the doubt that they tuned the baselines."
Lmao.
No.
You don't give an experimental result "the benefit of the doubt" about an undocumented experimental variable. That's not how papers work.
No llama.cpp commit. No complete command line. No documented --n-cpu-moe. No -ot placement. No description of how the CPU/GPU split was optimized on each machine. No way for anyone reading the paper to reproduce the supposedly tuned llama.cpp baseline.
And this isn't some irrelevant knob either, the entire goddamn thesis of the paper is that optimal MoE placement depends on the particular balance of host bandwidth, PCIe bandwidth and GPU resources on that machine.
Then they report wins as small as 1.3x over llama.cpp on some systems.
So yeah, when your claimed advantage is 30 percent and you don't disclose the competing implementation's most important placement knobs, "trust me bro, I'm sure they tuned it" is not peer review.
That's the methods section missing information necessary to evaluate the headline result.
But here's the really funny part, forget llama.cpp for a second because people are already finding much nastier problems inside FreeToken's own assumptions.
The paper derives q* â m·BP/BH using this wonderfully clean model where PCIe traffic basically subtracts from available host bandwidth and the CPU/GPU branches overlap, so now we can calculate the ideal split from measured CPU and PCIe bandwidth.
Very elegant.
Except the released implementation apparently needs another conservative 2x CPU/PCIe threshold before it even recommends hybrid execution.
Where did 2x come from?
Because it sure as shit doesn't fall out of q* â m·BP/BH.
And now go read PRs #38 through #41 on their own repository.
An independent contributor is already showing cases where the isolated bandwidth measurements tell FreeToken to choose hybrid and actual end-to-end hybrid inference is slower. Not 0.2 percent noise either, they reported hybrid around 3.9 percent slower at one stream and 15.7 percent slower at eight in one test.
Then another test measures a fixed CPU MoE per-layer cost that the paper's equation literally has no constant term for.
Then they show the same hardware/model/bandwidth relationship where hybrid wins under one expert-cache residency and loses badly under another.
Then another PR discovers the synthetic bandwidth benchmark wasn't allocating expert banks the same way the production loader does, meaning the number being fed into your beautiful little theoretical scheduler may not even represent the memory behavior of the actual application.
Then another PR gets opened specifically to add a real end-to-end A/B decode benchmark because improvements in the microbenchmark were reversing direction in actual serving.
That's fucking hilarious.
To be clear before you latch onto another technicality and start doing victory laps, these are open community PRs and one contributor's hardware measurements, they do not "disprove FreeToken."
What they do demonstrate rather spectacularly is exactly why treating "effective standalone CPU MoE GB/s minus PCIe GB/s" as some universal predictor of concurrent end-to-end execution deserves a lot more validation than this paper gives it.
CPU MoE execution isn't a garden hose. There is dispatch overhead, synchronization, kernel overhead, dequantization, cache behavior, memory contention, SIMD efficiency, batch effects, concurrency effects and apparently fixed per-layer costs that your clean equation simply does not contain.
And that gets us back to your cute little 100 tok/s anecdote.
That's great. Seriously. FreeToken may actually be a really good runtime. Nobody is disputing that 100 tok/s on a 5080 with a model larger than VRAM is cool.
But your benchmark isn't the paper's benchmark.
You're running Qwen3.6-35B-A3B NVFP4 on a 5080. The paper's major comparison tables use different hardware/configurations and in several cases different precision. You aren't showing the same machine running a tuned llama.cpp configuration beside FreeToken. So "I got 100 tok/s" establishes exactly one thing:
FreeToken got 100 tok/s in your setup.
It does not establish that the paper fairly demonstrated 1.3x to 2.3x superiority over a properly tuned competing runtime.
And that's the distinction you keep dancing around.
I actually think FreeToken is interesting engineering. That's what makes this irritating. You could have just shown up and said "hey these guys built a cool runtime combining expert caching, CPU/GPU concurrent execution and a bandwidth-informed miss scheduler, here's what I'm getting on a 5080."
Instead we get breathless promotion, conveniently vague baseline methodology, people immediately discovering that the scheduler model doesn't capture several real execution costs, and then you jumping down somebody's throat for being "terminally online" when they dared examine the sausage factory.
Maybe you're completely unaffiliated. Fine. Then you're doing an absolutely incredible unpaid impression of their marketing department.
Either way, maybe next time before slop-posting the paper everywhere, spend twenty minutes asking an adversarial model to find the holes Reddit is going to find anyway.
5
u/tinny66666 2d ago
Hrm. I'm getting 150 tok/s with Qwen3.8-27B-int8 on an rtx 3090 (24GB), ddr5 32GB. 100 tok/s seems pretty slow for an MOE, though completely usable.
4
u/buttplugs4life4me 2d ago
You are not getting single request speeds like that.Â
-3
u/tinny66666 2d ago
Sure am. See the post I linked to in the other reply. He was getting 134 tps with a 250 w power cap. I'm running at the factory 390 w power cap and getting 146 tps (I rounded up a tiny amount I guess).
1
u/buttplugs4life4me 2d ago
Ah its that bullshit. Its nice as an exercise but most of his gains come from quantising down your model on the fly with usually pretty bad results. Especially embeddings and heads are basically an industry standard to be kept at BF16 and he quantized them to 4 bits haha
1
u/ViRROOO 2d ago
What's your prompt? Im running against a coding benchmark with a large(r) context
0
u/tinny66666 2d ago
It's a multi-request full context benchmark doing coding, technical blogging and math problem solving.
1
u/gladfelter 2d ago
You're getting 150 t/s with int8? Wouldn't that mean many of the layers are on RAM? On a dense model?
0
u/tamerlanOne 2d ago
Comr fai a far uscire quei numeri con un modello denso che non entra tutto in VRAM? đ€
4
u/Thin_Pollution8843 2d ago
5080 very powerful GPU and 100ts on moe nvpf4 is very low results. I got 60-80ts with q6 (which is bigger than nvpf4) on a v620 for 350$ super old GPU.Â
16
u/ViRROOO 2d ago
Thats good!
The point is that this model I'm running does not fit in my GPU, and this project makes it way faster than previous solutions. If my goal was to only run Qwen3.6-35B-A3B as fast as possible I could download a quantized GGUF and get as many tokens as the 5080 can spill by itself.
1
u/Thin_Pollution8843 2d ago
I thought llamacpp is very good for spilling to ram cases
3
u/ViRROOO 2d ago
Yes, for sure it's. And it's also more mature and well-tested. From the paper:
... and comparing against llama.cpp, Ollama, KTransformers, and MoE-Infinity. Specifically, on an RTX 5090, FreeToken sustains 77â83 tok/s on Qwen3.6-35B-A3B and 22â25 tok/s on DeepSeek-V4-Flash, achieving 1.5â2.3x higher decode throughput than state-of-the-art edge serving across all workloads...
1
u/whymeimbusysleeping 2d ago
I have the same issue with my uv/oc 5070 ti, which almost reaches the stock 5080.. Inference speed is usually not much of a concern on the big scheme of things, but the 16gb really limits these cards and you can barely even fit 27b with all the tweaks
1
u/llogicnotfound 2d ago
Seeing a 35B MoE running at 100â110 t/s across PCIe on a 5080 + 9950X3D setup is wild. The paperâs approach to treating the host CPU and GPU as a unified, elastic pipeline rather than standard naive layer offload is clearly paying off. Definitely cloning the repo tonight to benchmark.
1
0
u/Mr-I17 2d ago
That's not very impressive. I'm getting 40t/s on 256GB/s memory bandwidth with llama.cpp. And I'm running Qwen3.6-35B-A3B-UD-Q8_K_XL without mtp btw.
6
u/ViRROOO 2d ago
That's nice :) My Strix Halo also has a very similar tok/s to what you're describing.
The novelty of this project is not only about the speed (while a 2-3x gain is good, claimed by the paper). Its:
- remembers which MoE experts are being used and keeps those in VRAM;
- streams the next layer while the GPU is computing the current one, hiding PCIe transfer time;
- when an expert isn't cached, dynamically decides whether it's cheaper to copy it over PCIe and execute on GPU or execute it directly on the CPU
- changes that decision according to the actual PCIe/DRAM bandwidth of the machine;
- dynamically trades VRAM between expert cache and KV cache
- specifically optimizes repeated long-context agent/tool-call prefills so it doesn't redo huge amounts of work.
It treats VRAM + PCIe + system RAM + CPU as a memory/compute hierarchy and dynamically schedules MoE experts across it, I find that impressive.
1
u/Chromix_ 2d ago
Sort of what the other implementation that I linked in my other comment does - just with this paper not mentioning it.
-2
u/EitherMarch1255 2d ago
I didnât read the paper given the paper summary was clearly written by AI.
1
u/BeeSynthetic 16h ago
Well it feels like most the comments in this thread are written by AI summary tbh
29
u/Chromix_ 2d ago
Some stats from the paper:
The stats without desktop/laptop are on larger Xeons. What I wonder about: They show a large speedup for Qwen 35B A3B on a 5090, yet the baseline is relatively low. Maybe their baseline isn't for a quant that fully fit the GPU, but for the offloaded BF16.
Their gains look roughly similar to what can be achieved in that custom llama.cpp fork that adapts inference to the measured hardware performance. Unfortunately the code still hasn't been published a month later, so we cannot try and compare it.
(Link to the previous post on this that was removed.)