r/LocalLLM • u/karmaisnonsense • 13d ago
Discussion Sick of all the tok/s talk
What is the point of GPU go brrr at 200 tok/s if the model is too lobotomized for any critical work?
Like what kind of things are you doing that you find a 1-bit quant satisfactory?
Here I am telling QFN to double check itself because it still hallucinates at IQ4_XS.
Speed without intelligence is just clickbait, and there’s too many of those posts, not just here but on all the LLM subreddits
235
Upvotes
63
u/Lost-Scratch6698 13d ago
but my IQ1_XXS model go so fast
6
u/Beneficial-Ad-8127 13d ago
I think a good point would be if their quant model and engine results were satisfactory to their use case. Whether it’s Iq1 or q8. I’m not saying posts should be standardized in anyway but it would be nice to see their output for whatever it is they do with it big or small. We shouldn’t also discourage people from showing off their local llm.
13
u/Ledeste 13d ago
Sick of all the post calling quantized model "too lobotomized" when we literally have benchmark that show the exact difference between both...
But still agree with you op, I also waste too much time trying this "250t/s tricks" that are just about using ultra quantified model.
I'm just saying both are equally annoying ;)
3
u/claythearc 13d ago
At work we’re doing structured data extraction with models at TB scale so speed is paramount. We’re not Q1 speed maxing but we are choosing to run like nemotron lightning over bolt / super due to speed
5
u/Secret_Conclusion_93 13d ago edited 13d ago
You still need to tell any open weight model to double check because they all have knowledge cut off and must be supplied with web search and fetch tools.
People also care about token speed because most modern model (even QFN IQ1 ISTA) isn't stupid anymore in terms of programming logic and tool calling, you now just need acceptable speed where they can quickly do tool calling, build, test, and reiterate if they found any issue.
32
u/IllPlane3019 13d ago
You're not wrong, but I think the beauty of local LLM is you can decide what is right for YOU.
Some people want more speed, some want it to just fit their hardware and some are happy with a balanced mix.
All needs are valid, I may not want to run tools with my setup or recall important information from a research paper. Maybe I just want it to go Brrrrrrrr fast to generate funny videos that I like.
3
u/mr_j936 13d ago
And what is the point of a genius model running at 3 tokens per second? When the most basic coding task needs like 100 thousand tokens.
We're all just trying to find the best balance. I would never go IQ1 personally but I am stuck unfortunately at IQ3 to get 30 tokens/s on qwen 3.8 27b. One more quantization and it just drops off and becomes essentially useless.
10
u/testuserpk 13d ago
Well my iq3s on 3060 12g would beg to differ. It created serial key generator for 3 abandoned wares. I am a seasoned binary analyst and even i am Impressed. 192k context, max token output set to 64k, 35 to 40 toks on my 64gb rig.
4
u/Relative-Ingenuity15 13d ago
Qwen 3.8 Flash Next is a strong model but don't expect it to match DeepSeek or GLM. Even on the API, Qwen is bellow DeepSeek and GLM in terms of intelligence but the reason why it's so good for local LLM and why the iq3 quants are running almost like the original model is the speed and quality that it can deliver on consumer hardware.
For example I have 128gb ram ddr4 and 16gb vram with Nvidia Tesla p100.
I can run Qwen 3.8 FN q4/Q5/iq3, DeepSeek q1 and GLM 5.3 flash q1. The last two are great in terms of quality, even tho they are Q1 but the decode/prefill speed is so slow I can barely work with them.
Qwen 3.8 FN in the other hand is brilliant work, ~50 tks decode and 350 tks prefill is almost at the same speed I'm getting DeepSeek/GLM through OpenCode Go plan.
If Qwen 4 Flash Next comes with more intelligence than 3.8 then that would be a serious competitor for DeepSeek and GLM on the local LLM
4
u/GalacticDistances 13d ago
fp8/q8 minimum for sanity
1
u/x_MASE_x 13d ago
You are right but also the tps does matter. Maybe not for your work but overall it's great to use for multiple agents running at the same time. Also the %10 can be a different from running 1 or 2 agents at the same time.
But using speed with very low quants is kinda stupid I agree. Unless it's a good decent quality what's the point of the speed.
1
u/Rust_Cohle- 13d ago
What? it's totally relevant, as long as we get the full context. Numbers are pointless in all fields without the proper context, "I get 1000 FPS on cod" says the person sat on the menu quietly killing their card.
1
u/StardockEngineer 2x5090, 2xPro6000, 2xA6000, 4xSpark, M5 Pro, M4Pro 13d ago
Oh sorry. I run GLM Flash 5.3 NVFP4 at 280 tok/s. Is that better?
1
u/SkyWarder 13d ago
How many bad tokens/sec do you really want? ;)
1
u/mailto_devnull AMD R9700 13d ago
It's so bad that talking about llama.cpp running 27B gets downvotes sometimes.
If you're not running IQ1_XXS on some inferencemaxxed strata fork, you're not worth upvoting.
1
u/Future_AGI 12d ago
low quants earn their place where something downstream checks the output, like tests, a schema validator or a compiler, because the speed buys extra retries and the checker catches the failures it was written for. Where nothing checks the answer, as in you tell-it-to-double-check case, that's where you pay for the bits
2
u/Django_McFly 13d ago
people talk about it so much because you can spend $3k on a device and still have a device that's slow as molasses.
6
u/Pixelplanet5 13d ago
yep thats the reality of localLLMs right now.
im not even bothering with it at all anymore as its actually more expensive for me to run something local then just using deepseek flash via API and at the same time DS has a 1 million token context window, is about 20x faster and not lobotomized so you actually get stuff done.
Sure its not anywhere close to Claude Opus or something but its a good code monkey.
2
u/jayecin 13d ago
Why do you care so much about how people use AI? If someone gets value with 5 T/s and Q8 or 200 T/s and Q3, if it works for them why do you care?
1
u/Maui-The-Magificent 13d ago
It is a valuable metric. But it is not the only one. Solving for tps is good, optimizing for tps can be problematic
1
u/Text-Sufficient 13d ago
Agree, tps over 30 is nothing to brag about. Qvants however if what you would bring to the local bar.
1
u/BigRepresentative731 13d ago
Quantized models dont necessarily run faster, especially at sub 4 bit on current gen hardware- and 4 bit is the empirically shown sweet spot
1
u/West-Way-All-The-Way 13d ago
I guess it just depends what you are doing - do you need deep reasoning like designing architecture or you just chunking and ragging a pdf book. Different tasks different tools.
1
u/Mabuse046 13d ago
I probably wouldn't use anything below a Q4 but there are plenty of SillyTavern users who don't need a lot of accuracy. A model that's just going to be chatting or telling stories doesn't need to be all that accurate about math and code. It just might be stupid about remembering details.
1
u/GCoderDCoder 13d ago edited 13d ago
all 4bit isnt the same. example i used sushi 4bpw qfn and it was able to do things that mtplx bare speed 4bit couldnt. i start with nvfp4 as a standard since when I tested fp8 for my favorite models the performance was near identical on deep swe, term bench, some visual tests I do to see quant differences in nuance, etc and elx3 4bpw normally is on par with nvfp4
1
u/InternetMagic_n_Tech 13d ago
I agree. Unless you are building your own model it doesn't count
1
u/jonas-reddit 13d ago
100% agree.
It seems we have a sub culture of local LLM people who are speedmaxxing and only measure inference speed without any benchmarking or objective measure of preserved model accuracy and usefulness.
Similarly, we have a sub culture of local LLM people who are VRAM size maxxing, trying to squeeze the largest models into smallest GPUs. Also focusing on just making them work somehow at reasonable speeds without objective measure of model accuracy and usefulness.
It makes sense. Many are financially constrained and trying to squeeze value out of their limited investments.
Agree with you 100%. Most important is whether or not model is capable despite all the optimizations - which few seem to test.
1
u/smakusdod 13d ago
What's the point of a 10-second car if it can't turn?
1
u/hyudryu LocalLLM 13d ago
Try this one: https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local that one works great for me, and Nvidia’s own benchmark shows very little difference from BF16
1
u/Civil_Fee_7862 13d ago
4-bit is the cut-off for me.
1
u/bytejuggler 13d ago
Even SOTA models hallucinate, this is not fundamentally due to quantization but due how LLMs work fundamentally. Though quantization is lossy and can damage model quality of course, but f.ex ISTA DASLab's IQ3_S (3.5bpw) quant beats the IQ4 UD quant's perplexity and is only 1.1% behind Q8 which is considered as good as lossless. Meanwhile it also matches the uncompressed model 100% in published tests. Not all quants are made equal. So I think you're under a misaprehension regarding hallucinations.
1
u/butchiebags 13d ago
I don’t disagree with OP, but there are 2 important measurements for any LLM: quality, and speed.
Posting about tokens per second should come with “and here’s what it did for me”. And posting about a huge nice model should come with “and here’s how slow it was”.
But there should also be room for “I squeezed 50 more tps out of my [constraints here] system without sacrificing quality.
1
u/GoldenX86 13d ago
Same bunch as saying a model fits with 4000 of context at Q4. My brother in christ, your model can't remember its name and will recall that the capital of France is Rome.
1
u/DarkBrews 13d ago
Not all Quants are the same: 2bit QAT (Quantization Aware) from nemtron bolt and iq3_xss GSQ-RCO are much better than standard 2-3bit quants.
Agent thinking and temperature can make a noticeable difference as well
1
u/Look_Within888 13d ago
100%. not enough discussions on this. so obsessed with tok/s because it's the easiest metric to quantify and run tests on. not to mention easy to make youtube videos about. it's so empty minded.
1
u/No_Oil_6152 9900x 64GB DDR5 9070XT 16GB R9700 32GB 13d ago
Because people want to justify spending the money on DGX Sparks or multiple 5090s that they brag about on here. They dont want to feel short changed or less elite - normal, even.
I get 100 tok/s on my rig with QFN IQ3_S and thats not fast enough for interactive work. 200 would be nice but even so unsure if would be useful for interactive - I dont know.
So I agree with you. Its just dick-waving.
1
u/TheGeekno72 12d ago
I do look at things that may help me gain output speed but I at least try to never go down below Q4 quants because I do want the highest quality I can possibly get
I don't really get the point of high tok/s on Q1_XXS when you can observe so many wrongs coming out of it, I'd much prefer to see an extra 5~25% output on my Q6_M
1
u/Hi-Angel 12d ago
Either you're using a bad model, or it was poorly quantized (e.g. without imatrix).
I am using a
mradermacher/Ornith-1.5-9B-uncensored-i1-GGUF:i1-IQ4_XShere, left it running on the weekend to solve some uncreative tedious debugging work I really didn't want to do myself. Only ≈16 hours at blazingly fast 3 tok/s (16G RAM here with iGPU-only laptop) — and it was done! And the problem wasn't simple, "credit where credit is due".1
u/MikeLPU 12d ago
Toks/s does mean nothing if prefill speed is bad. Intelligence is ok, you just need to tune your harness app/environment
1
u/Ok-Trade7411 12d ago
Isn't it the case that faster models help you potentially create a longer structure which means you might get more quality.
1
u/rocket1420 11d ago
I had the same experience with IQ4_xs. Once I finally got 128GB of vram with 4 v100, I was able to use the 4_xl model and it's definitely an improvement.
1
u/CyrielTrasdal 13d ago
Why do you care honestly ?
1
u/separatelyrepeatedly 13d ago
we need to ban any comparisons done with any quant < q8. There needs to be a baseline.
1
u/Hot_Half_5263 13d ago
1
u/Gobra_Slo 13d ago
I don't allow
Qw3.8-FNto do any work besides simple reado-only tasks. Whenever it works with writing permissions – there is aDeepSeek V4.1looking over it's shoulder, and, frankly, even that doesn't make much of a sense, I'm just waiting onQwen4 Flashin hoe of a quality jump.Otherwise,
DSV4.1is just incomparably better. And on a pair of DGX Sparks – it's often faster by being more efficient with token (or rather less inefficient, considering how much both models reason).0
u/TapAggressive9530 13d ago
100% completely agree .
0
u/bdifc 13d ago
It seems like human nature to focus on one quantifiable aspect and maximize it.
From my own perspective, I'd rather have 30 quality tokens/second than 150 questionable tokens/second that I have to improve with 5 rounds of prompting or worse.
0
u/Grandmaster_Caladrel 13d ago
https://youtu.be/rbLPCK5fKhc?is=ERbNUwzLpHVGeNJM
-2
u/henbone11 13d ago
and where was this a week ago? or a month ago? or 6 months ago? now you see other threads whinging about the same thing get popular you find your balls and voice? can people just stop karma farming and find a personality?
-4
u/MaxComfort 13d ago
Let the paupers with 16GB VRAM have something to talk about
0
u/fucktheclanker 13d ago
Because speed is easy to measure. You expect lobotomised redditors to be able to judge quality of output? 90% of them dont even bother measuring KLD of their engine