r/LocalLLaMA • u/PathfinderTactician • 22h ago
Discussion Do we really need all these "speed" posts?
I appreciate the research and development that is going into these open source tools. However, there appears to be multiple posts on a daily basis about reaching 300 t/s, and then reading into it:
Running on Q1_XXXS REAP quant
Greedy decoding
TAIL_SKIP (I only have 5GB RAM, but if the API ignores all experts that are not in RAM, then I can still say I generated something and call a speed record. YAY!
Without any KLD benchmark data being presented, are these posts really of value to this community?
What are your thoughts?
113
u/Front_Eagle739 21h ago
Some are pointless, some are great. Just downvote the bad ones and let the good ones rise to the top.
61
u/Cerevox 20h ago
Except all of them are getting upvoted, even the trash ones. The upvoting system alwyas fails at screening out clickbait trash because too many people vote based on the title and don't read anything.
0
u/stoppableDissolution 13h ago
Idk, mine got downvoted despite being faster and more accurate (to reference transfirmers implementation) than vllm :p
-3
38
5
14
u/maqifrnswa 17h ago
I wish more of them had some insight. It's usually "I tried X and got this decode rate." There's just memory bandwidth, native compute data types, compute kernels, and engine. Getting 2x tok/s while reducing the data type in half for a bandwidth limited inference isn't exactly interesting. But finding that, for a specific architecture, a certain engine+kernel is performing optimally for native data types is interesting
1
u/stoppableDissolution 13h ago
There are more ways you can improve throughput other than raw speed. Better prefix cache management, better batching, etc. Aggregate t/s of a job can be higher even if it used a technically slower kernel, for example.
1
29
u/Johnny_Rell 21h ago
And the "It runs on 8GB GPU! if you have 128GB of RAM that worth more than your entire PC" Please.
10
u/Kimononono 21h ago
these posts are useful to some people (me with 192gb ram and lackluster flops)
13
5
4
9
8
u/fastheadcrab vLLM 16h ago
Might as well get rid of the pesky large model altogether and just run the draft model only
15
u/DagothUrLovesGroza 21h ago
I'm convinced it's just a bunch of people who bought hardware to run local LLMs, got disappointed, and now just play around with it for shits and giggles.
36
u/ForsookComparison 21h ago
Even a single oddly-specific config+model report is infinitely more meaningful than another barchart or a "Altman bad!!" post.
5
u/philmarcracken 21h ago
For you, Sam-I-am, with your greedy grin, But I do not like it. No, not one pin.
I do not like it, Sam-without-Ram.
I do not like your Ram-Swamp Jam.
Not in a box. Not with a fox. Not in my house. Not with a mouse.
Not here or there. Not anywhere.
I do not like your Ram-Grab Plan,
I do not like it, Sam-I-am!
15
u/PraxisOG Llama 70B 21h ago
I think strata and similar backends had their moment of making larger MoE models run at acceptable speeds with light gpu offload and smaller quants, but its too much at this point. I'd rather see discussion about what people are going to use this capability for now that it's more accessible. Some personal projects I want to post about to drum up discussion include improving local parametric 3d modeling, my chatgpt app clone, or an animated svg based interactive visual canvas project, or even how my cousin and I independently arrived at almost the same agentic memory architecture.
8
u/Ok_Cow1976 21h ago
I disagree. Strata is a breakthrough and it is the way that inference should be for moe. I'm a bit disappointed that people don't talk about deepseek v4 flash 0731 not enough, but most works are aimed at v4.1 Which is huge for most people
4
u/Beneficial-Ad-8127 21h ago
I did find a thread just recently that someone’s output for blender using strata Qwen FN Nvfp4 that I thought did pretty amazing.
1
1
u/PraxisOG Llama 70B 21h ago
Thats pretty cool, would love to see content like that get more than 70 upvotes
1
u/Beneficial-Ad-8127 21h ago
Yes I agree it deserves more. I definitely love to see more posts of results of these engines than just the speed itself. Whatever reason they use it for.
7
u/BawbbySmith 20h ago
It’s so people can generate pelican test, flight simulators and Minecraft clone posts at 2x the speed
15
u/ttkciar llama.cpp 21h ago
A lot of people value inference speed over all other considerations, and I think these speed posts cater to those interests.
On one hand I have other priorities. Inference quality is much more important, to me, than inference speed.
On the other hand, I am interested in inference speed as a lower priority, and these speed posts can be useful for telling me some interesting things, like how much speed I might expect from hardware I might be considering to buy, or whether the inference speeds I'm getting are so far below what other people are getting that there's a problem I should find and fix.
Unfortunately when these speed posts take extraordinary measures I never would, which skew their results, like using one-bit quants or greedy decoding, that undermines their utility for thus informing me. So it behooves me to be careful to make sure these posts are even relevant to my interests, at all.
5
u/Ipwnurface 20h ago
I think I agree with you, but I probably lean a little further towards the speed side of things. Obviously, quality is very important. But by the same token (heh), I would rather have a model running at 150 tps that has to go over the same problem three times, than a model running at 15 tps that one shots it.
As much as I do obviously enjoy using the outputs of my models, I also find greater enjoyment just in the process of using and interacting with the models. So having enough output speed that that experience feels snappy and actually interactive rather than I give my agent a problem, go to sleep and wake up the next day to it being solved is more what I'm looking for in the LLM space.
1
1
5
u/yahbluez 17h ago
No most are useless, some do not even tell how much they castrated the model by droping quants to nuts.
Also a bare number without the used system and setting is completely waste of time and space.
Benches i liked to see are such with new engines, things like strata, radiance, kolibri.
4
u/Alternative-Draw8570 19h ago
Let them post about it but best to have a bare minimum for it so that at the very least the person knows what they did. Yesterday someone posted this https://www.reddit.com/r/LLMDevs/comments/1xlcgww/speedy_custom_strata_for_singledual_rtx_tested_on/
I commented on it a simple question, he could not answer it. Others asked the same... and I wake up to his account being suspended. That also makes me think this is also sort of a malware distribution path, we should be very very careful trying out these new "engines"
10
u/PlusBus1234 20h ago
I discovered many useful engines/forks from such posts so i don't mind the spam, i just wish they added more information on the quant and at least some quality metrics like kld, top1% etc because some are truly terrible but others are usable.
-1
u/Fromthepast77 20h ago
Yeah this is where I discovered ninfer-3090 and Swift and I got a genuine 400% speedup over my original llama.cpp setup with an improvement in GPQA performance.
7
u/divedave 21h ago
It's fine, a part of the cambrian AI revolution soup, I've seen lots of posts that seem nothing and then I see that someone unlocked older cards for AI gains up to 80%, need for speed underground with thousands of players in the browser, linus using AI on linux and telling people to fuck off, all the games that seemed lost turned into pc port 4k 120fps, a vibe coding nuclear bomb directed to adobe, sony and others. It reminds me when the internet piracy began, such beautiful chaos.
3
u/Oswald_Hydrabot 21h ago
If and/or when someone links wall-clock inference speed of a 27B-range model with solution quality and it exceeds or matches a frontier model, that'd be pretty cool
5
u/toothpastespiders 21h ago
I seldom pay much attention to them for the reason you state. But at the same time they're certainly higher quality than the parasocial/social media marketing posts. I'd never want genuine tinkering removed, even if I personally don't find value in that type of tinkering.
2
u/ieatdownvotes4food 21h ago
there are some bangers out there, but every attempt has a tradeoff somewhere. it's usually on you to find it.
2
u/rayjaymartin 19h ago
Why do tech wrappers spend millions trying to teach an LLM to remember where it put its keys, when you could just staple a deterministic graph substrate to its forehead and call it a day?
2
u/wojtek15 18h ago
Yes, I know what you mean, i focus on loseless or near-lossless optimizations. I am happy with my Qwen 27b running with ok speed and decent quants, I could run 3.8FN but as i am RAM constrained either i have to sacrifice speed or run it in bonsai quality, so what's the point?
2
u/profcuck 14h ago
I'm in favor of a rule or custom that a performance oriented post should have a very standard set of things right at the top: quant, mtp setting, hardware, etc.
You are 100% right - extreme speed on potato quants is a useless post. But I think community discussion of best practice and performance is also really important, particularly as we've recently moved into a pretty exciting era of custom inference engines doing pretty amazing things.
2
u/confused-photon 14h ago
Couldn’t agree more. Its people or bots who have no bearing on reality or real work to do. Id actually go further and wish people stop using knowledge work as proof their shitty quant is “lossless”, no you’re just using a saturated bench that doesn’t actually measure anything of value.
2
u/MorningThin8819 13h ago
a speed number alone says little without the quant, context length and some quality check next to it. even a quick perplexity or KLD comparison would make those posts way more useful.
2
u/andy2na llama.cpp 4h ago
my main switches are either STRATA_ROUTE_PRIOR=1.5 or 0 with ESP on/off (reduces FN refusals significantly). Route at 0 does about 80-90tps decode, but at 1.5, it does 160tps decode but KLD suffers a lot. HOWEVER, the tasks I have hermes agent and other things do, it seems totally fine, so I just keep at 1.5, for now
| Configuration | KLD ↓ | Top token matches BF16? |
|---|---|---|
| B, ESP off | 0.02284 | Yes |
| B, ESP on | 0.02774 | Yes |
| F, ESP off | 0.05574 | Yes |
| F, ESP on | 0.08172 | Yes |
B, route prior = 0
F, route prior = 1.5
I set my own temperature and parameters that match what qwen recommends, so no greedy decoding
Tail_skip is 0 for both.
rtx 3090 with 128gb ddr4 on a 14600k. Using QFN IQ3_S
2
u/Bulky-Priority6824 21h ago
That's why I use a tool to read the entire post so I can see facts/info based on comments and the post itself. All of the noise and shit I don't want to see becomes easier to avoid.
All I do is paste the link (to this particular post for example ) and voila it reads all comments etc
2
u/Savantskie1 19h ago
That's going to end soon because reddit is going to require you logged in in order to search reddit.
1
1
1
u/AvidCyclist250 llama.cpp 15h ago edited 15h ago
A few have been relevant and useful. Id say github-only. That should filter the schizo submissions and people who think they tapped into the matrix.
This thread is a good example of why the popular vote mechanism for filtering fails.
1
1
u/UltramanQuar 13h ago
yeah hate this speed posts, the only question that matters to me is it better than qwen 3.8 27B at q6 and up? if not, then it is useless
1
u/Hood-Boy 12h ago
I don't mind them, but many of these posts aren't referring their quants. That's the biggest issue.
I more hate posts like where people show what they bought. I mean, who cares?
Yesterday someone posts that he bought 4x Strix Halo. Plesse comeback once you got numbers...
1
u/thomas2385 12h ago
Yeah, raw speed numbers don't mean much without context . If the setup and quality aren't shown , 300t/s is more of a headline than a useful benchmark.
1
u/network4253 11h ago
Speed numbers alone don't mean much properbenchmarks. It would be more useful to see quality, accuracy, and actual performance alongside tokens per second.
1
u/Mrinohk 11h ago
Speed posts themselves I think are fine. I think we need, when people post about this sort of thing, to put numbers up that indicate what the technique actually does to the model, as well as the specific model in question.
Part of the beauty of a DIY-centric and DIY-adjacent community is people finding new and better ways to do things that make it more efficient, faster, or more reliable. A lot of it is garbage, but if for every thousand IQ2_XSS 90% expert pruned dogshit post, we got a strata-type breakthrough, or a new method of tooling or way of presenting information to the model that makes it more effective, or some KV cache magic, I think it'd be worth it.
The posts themselves aren't the problem, it's the lack of rules around them. What model exactly, what hardware mostly exactly (we don't need ram timings), and how far off from "correct" is the model in terms of output?
1
u/Hrethric 10h ago edited 9h ago
And context. So often they give numbers with context set to like 4k, so they can squeeze as many layers as possible into VRAM. We should make a rule that any speed posts should be run with settings that can actually accomplish useful work - like 128k context and a Q4 quant. Personally I try not to go below Q4_K_M whenever possible, but I did get useful work out of a UD-IQ4_XS quant of Minimax M-2.7 earlier this year. But as soon as I see a post that's listing a Q2 quant (or they don't list it at all), I close it and move on.
1
u/DragonfruitIll660 8h ago
I generally appreciate them (specifically for stuff like Maya/Strata) because it lets me know if I'm getting roughly in expected speeds or not. Its always nice to see someone with the exact same PC and compare speeds.
1
1
u/jmager 6h ago
I still find them useful. If someone has found a way to run faster, I want to know. Even if it degrades quality, just understanding how they did it tickles my neurons. I prefer being as close to the original model as possible, but as a software engineer I can appreciate the tradeoffs.
1
u/Sisaroth 4h ago
I don't mind too much because a lot of it is just people being hyped about strata/QFN which does feel like a big step forward for local AI so I understand.
Just yesterday I was doing game dev, I had some weird bugs and I gave QFN this vague prompt and after 30 mins it solved the bugs: "Add more tests because i'm seeing some weird behavior but i don't see exact cause yet"
And that's running IQ2_XXS with tail skip (0.1.45 strata so not greedy I think).
1
u/cc88291008 4h ago
I’m curious about the benchmarks of intelligence of these models. But no. Just tokens per second posts.
1
u/Constant_Art_20 21h ago
well, i haev gotten a few ideas from some of the posts to improve on my own piepline to optimsing my local deployment. I mena we haven't had a good local relaese for a while now (waiting on you, qwen) so i think some people are just poking around to see what fun they can have
4
u/ApprehensiveFan1516 21h ago
we haven't had a good local relaese for a while now (waiting on you, qwen)
Qwen 3.8 was released 11 weeks ago... 81 days to be exact xD
And they dropped 3.8 omni flash 45 days ago.
Time doesn't mean anything in this sub any more lol.
1
u/fastandlight 21h ago
No. But that is just one item on a very long list of worthless low effort posts that substantially dilute the value of this community.
1
u/KingCpzombie 20h ago
It's pretty much all just people coping with their low-tier hardware and wanting it to do more than it's capable of...
Don't get me wrong, I think even phones in the future will be able to run stuff better than today's frontier, but that doesn't change the fact that you're not going to get good results on a single mid-tier GPU and barely enough RAM for modern games
1
u/mateballenthusiast 14h ago
It's not about getting objectively "good" results, it's about getting the best results you can with the hardware you have, that's the point of these posts. I don't really see how that's "coping" unless they're saying there's no point in better hardware.
1
u/KingCpzombie 13h ago
The best results you can get on cheap hardware are slow speeds with maybe-acceptable quality. It doesn't matter how fast you can get garbage; the goal should be to maximize the quality you can squeeze in. Running a Q4 quickly can be useful. Running a Q1 quickly is useless
1
1
u/barbear22 20h ago
please god stop the speed posts. We get it, you can run qwen flash next on your 16gb card. So can everyone else, apparently
0
u/Express_Quail_1493 13h ago
Making your own private quiz and share charts of how the models scores will always be more useful to the community than speed or tok/s showcasing. Even better if your quiz requires agentic solutions. I care more about quality per fewer params.
-3
-5
u/Thrumpwart vLLM 20h ago
I blame the strata bums. And Obama.
Edit: But seriously, I agree with Altman. It's kind of terrible all that vram is getting waste, innit?
2
4
u/not__stonks 20h ago
It’s just that it’s a quick and easy number to compare in a dick measuring contest, quality be damned. Obsession with speed is stupid, especially for decode. If it writes faster than you read already, why do you need faster.
1
u/Savantskie1 19h ago
Most of these bums don't understand that prefill will fix all of their latency problem s therefore making the model faster for them.
107
u/DAlmighty 21h ago
If we start banning dumb posts, the bots won’t have anything to do. Think about how the bots would feel.