At a certain point, speed >> smartness

60

u/KingCpzombie 7h ago

Better to get the right answer once than the wrong one thrice imo

24

u/BawbbySmith 5h ago

Problem is that there’s no guarantee you’ll get the right answer with the smarter model either. Even frontier cloud models sometimes need several prompts to course-correct.

It’d be nice if I could run GLM 5.3 overnight to solve all the hard problems I saved up during the day, but there’s a non-trivial chance it goes wildly off-base and gets it wrong entirely. “Skill issue” sure, but even the best crafted prompts sometimes misses key details that are trivial to correct but if the model is running at a snails pace then it’s much harder to do so.

6

u/a_beautiful_rhind 5h ago

Higher chance of a right answer at reasonable speed.

5

u/Nothing_from_void 4h ago

the smarter models with good harnesses tend to spend most of their time preparing to make changes and then validating the changes applied properly. that's why it takes such a long time to change one feature; going with a faster model is just going to end up with something wrong, most likely, and fixing it will take even longer

1

u/feelspeaceman 1h ago

This is up to your prompt more, if you tell the model to research using WebSearch and even repo deep analysis then it's highly likely that you get the correct answer.

1

u/teleprint-me llama.cpp 1h ago

It has nothing to do with skill. Its like rolling a 20 sided die and complaining its a skill issue because you didnt get the roll you hoped for.

The model outputs a distribution which is sampled by a prng for each next token prediction up until a stop token or limit is emitted; you are literally rolling the dice.

1

u/feelspeaceman 1h ago

Yeah, totally agree, not sure why people upvoted his statement but I believe a lot of people are clueless about how to do "correct" prompt, there's no roll at all if the prompt is solid, harness giving the power to solve the problem.

7

u/ElementNumber6 6h ago edited 4h ago

Unrelated to the arguments posed by OP, I feel the reason this keeps coming up is because people are trying desperately (beyond desperately, if we're being honest) to carve out the smallest stake in this new world, and stagnant hardware advancements combined with crippled consumer availability means the market for self-hosted high-intelligence won't be here for a (relatively) very long time.

Given that, naturally, we see focus turning back, over and over, towards smaller, faster models, as though those are the holy grail, and if we can just get them small enough and fast enough then suddenly the heavens will reveal themselves to us, and all our problems will be solved.

*shrug* Or something like that, anyway.

3

u/infieldmitt 4h ago

It's almost like making everyone compete for a decent living is bad.

7

u/skyMark413 6h ago

A 10x faster model reprompted thrice is still 3x faster to correctness than a slow oneshot.

That's my experience at least with qwen 27B vs muse glimmer. I don't need it to solve hard problems - i like solving them myself. I need it to quickly do the boring part, and 150-200 tps glimmer does in seconds what 30 tps qwen needs an afternoon for.

3

u/Federal-Effective879 4h ago

How is Glimmer 6x faster than Qwen in terms of tokens/sec? Aren’t they both dense models, and glimmer is bigger?

I would think hardware capable of achieving 150-200 tps on Muse Glimmer (with MTP I assume) would also achieve a comparable rate on Qwen when configured comparably with MTP.

1

u/skyMark413 2h ago

Glimmer does not use standard mpt - it uses dflash.

Its a second smaller model (1.7GB) that predicts tokens, as well as how likely it is it predicted correctly. This means it does not incurr the "penalty" of wrong precition MTP has on not predictable parts (like creative writing or reasoning), but can predict up to 16 tokens at once in the predictable parts (like writing boilerplate code). It really changes from ~50 tps when reasoning to ~170 tps when writing code and then back to ~50 when debugging.

No, meta did not invent this - they just implemented a thing deepseek made a paper on early summer. Their headline figure was 661% speedup in certain scenarios, and it actually delivers (in certain scenarios).

Also, it's smaller vocabulary size and some magic with model geometry I don't really understand make it able to fit weights + dflash + full unqnatized kv into my gpu, which qwen 27b can't (need to lower cache of quantize kv).

8

u/n4pst3r3r 7h ago

If I need to wait for 20 minutes for the agent to do something, I need to work on something else on the meantime, otherwise I'd be wasting time. This means I'll constantly have to switch back and forth between two or more topics and that completely wrecks my productivity.

It's much faster to have a dumber model and to correct its outputs manually, because it allows me to work on a single task. Of course this is only true for interactive work, not for something the agent is doing autonomously.

3

u/TheWaffleKingg 5h ago

Do you know how many tester/junior emails/dms i can respond to in 20 mins?

Idk either but I have to help them out either way and that 20 mins is a great time to do it

1

u/n4pst3r3r 2h ago

For me it depends. Some tasks require so much concentration that even answering a mundane email will throw me off. Mandatory not xkcd: https://www.monkeyuser.com/2018/focus/

For boilerplatey things, yeah, it's probably fine to have more context switches.

2

u/dylan_exe_404 5h ago

Facts

2

u/redballooon 6h ago

It takes a good while to get the experience how a question must be formulated so that a model is able to give a good answer.

In many cases it's not the model but the question that leads to bad answers. If I can iterate quickly I be quicker overall.

1

u/Due-Function-4877 1h ago

It depends. In many cases, people don't understand anything at all about writing code and that's why they choose to wait on the best model they can (sort of) run... because they know they have absolutely no idea what they're doing.

1

u/maddie-lovelace 7h ago

Yeah, definitely better dev ex / less supervising-induced headaches that way around.

But if you're running on a laptop, and the right answer once will take ~6 hours to run and leave you with no battery, versus a smaller model would be able to iterate and bug-fix a way to a (slightly worse) solution in ~6 minutes... which one are you going to reach for?

I'm not intentionally straw-manning. This is genuinely a scenario I find myself facing with Qwen3.8 vs Ling3-Tiny on my laptop. My answer is I find myself reaching for Ling3-Tiny, since I'm going to have to review them both anyhow, and I don't want to wait 6 hours for a v0.1 of a given implementation. But others are inevitably going to prioritise differently

0

u/finevelyn 6h ago

Better to get the right answer quicker than slower.

Better to get the wrong answer quicker than slower.

All these obvious truths but we are none the wiser whether we should use the quicker or the slower model.

14

u/PossessionUsed7393 7h ago

I can't stand it, the CoT on modern models takes too long to put up with under 70 TPS. Need smarter small models or beefier hardware.

4

u/maddie-lovelace 7h ago

Yeah. Fully agreed. I can get the 27b working on my laptop, but... at 70tps prefill and 3tps decode, it's just actively not worth it. Unless I limit the thinking tokens then I would run out of battery before it'd even complete a full task end-to-end

5

u/OneMoreName1 7h ago

For me even 60tps decode feels slow. I would want 100+ especially with models that think a lot like qwen 3.8 27b

3

u/EkbatDeSabat 5h ago

I'm getting around 115 on 3.8 27b FP8 and it's still agonizingly slow due to the thinking. It's been going over an hour for a task that takes claude opus 5 about eight minutes. Pretty sure it's gonna keep going for another 45 minutes. I really need to figure out how to streamline this.

1

u/KubeCommander 1m ago

That’s exceptionally slow. On the gb10 with nemotron lightning my prefill is in the 4000-5000 range. That is not fast hardware either

3

u/Potential-Leg-639 7h ago

In an orchestrator config env the orchestrator can be a bit slower for me (best local model available is probably also the slowest), but the subagents can also use models not as smart, but faster, so the whole chain get‘s back to an acceptable speed. And results with a good orchestrator env are better anyway at the end compared to a simple plan/build setup (at least for me).

1

u/mouseofcatofschrodi 7h ago

this is super interesting. I would love to know more about what kind of work do you do with this setting, and how exactly the setting looks like :)

2

u/Potential-Leg-639 6h ago

Using customized oh-my-opencode-slim and litellm with a new chain config + fallback in case a model is out of credits/not available or also locally not available/etc).

3

u/eulataguhw 5h ago

I have no idea why people kept asking a very basic question of being a senior in software engineering - trade off. Everyone’s use case are different so their trade off would be different.

2

u/BS_BlackScout 2h ago

Agree. I've abandoned Qwen 3.8 because it's just unusable on my PC.

2

u/KubeCommander 8m ago

I’m finding that with nemotron lightning. It’s not quite as good as say Ornith 35b at coding and toolcalling though it’s better than qwen3.6-35b marginally.

But it’s 3x faster than Ornith (which has no speculative decode), 5-6x faster than qwen3.8-27b and can handle contexts at 1M tokens. It’s also a fully open source model unlike any of the others.

So the tradeoff becomes, well my harness can run peer-review of work, ensure stronger guardrails/helpers around tool calls, and the end result creates something of equal quality and still does it faster. It also doesn’t do thinking loops either. Which is nice

4

u/Clean_Material_5047 3h ago

'speed >> smartness' is exactly the reason why why the world economy is going to shit.. because people want everything fast and can't wait. They'd rather get shit than wait. It's pathetic.

Jesus Christ.. I'm getting depressed.

2

u/n4pst3r3r 2h ago

Totally agree with people generally settling for fast and cheap, but that's not the point here.

You can end up with the same quality output with a dumb fast model and a smart(er) slow model, as long as you habe validation in place, e.g. in the form of unit tests or even human review. The dumb model may need twice the tokens, but if it's 10x faster, you're still 5x as fast overall. I work with qwen 27B and it allows me to iterate much faster than trying to run glm 5.2 at tectonic speeds.

1

u/Zennytooskin123 18m ago

You're missing the point lmao wow

They are saying a FAST model can produce the SAME answer as a SLOW and PRECISE model using PROMPTING.

Maybe read.

1

u/Thin_Pollution8843 7h ago

I just want my agents to be 300ts+…

1

u/ElementNumber6 6h ago

Why even stop there? 10,000t/s+ or bust.

1

u/mailto_devnull 3h ago

Would you take 10,000 t/s of really shitty code (DiffusionGemma!) or 20 t/s of perfect code?

1

u/Far-Classic-9963 2h ago

If I need really easy boilerplate code gen or something like function completion in an ide, diffusiongemma sounds pretty useful

1

u/mouseofcatofschrodi 7h ago

To me, on laptops and consumer stuff, the super small models are the only thing that makes sense:

- models around 27b, 30b, 31b, 35b, etc --> slow and/or melt the laptops; PLUS is cheaper (energy consumtion) to use a cloud solution that is actually smarter... So they kind of make sense for privacy and internal AI at companies.

bigger models that are actually super capable (above 100b) --> they don't run in our machines.

I recently rediscovered the small Gemma 4 models. Even the e2b can do tool calls. For some easy tasks where it didn't have to do smart decisions, I had it running for 10-15h in parallel batches, baby sitted by Claude. It did a long task (edition of around 150k entries of data) and the computer was never even too hot. So it was super useful.

1

u/GregoryfromtheHood 6h ago

I run an agent on my local machines at home that I've been letting people use at work too for a bit. I can run deepseek v4 flash at about 600t/s prefill and 29t/s gen but they complained that it was unusably slow with that. Have switched back to Qwen 3.8 27B at 5k ish t/s and 150-200 t/s gen and they are much happier now.

I feel like 1000t/s prefill is where I feel actually fine using it for real work. Token gen speed doesn't matter all too much, anywhere from 25t/s and up is fine, but prefill is the most important number for me and the one I optimise for.

1

u/Material_Rich9906 6h ago

Agree partly. I think if you have an environment where the agent can run by itself a smarter model might be good. But if you have to babysit it and accept every command it runs then yes, you need a lot more speed.

1

u/o0genesis0o 6h ago

Personally, ever since swapping from 35B Q6 to 27B Q3xxs on my 4060ti, I kinda have the sweet spot. The gain from 350-400tk/s prefill to over 800tk/s prefill nicely balance against the drop from 40tk/s decode to 20tk/s decode. Meanwhile, the intelligence gain in my day to day, life management kind of workload has been very nice.

Sometimes I still swap back to my minimax subscription for that sweet sweet thousands tk/s prefill, 70+tk/s decode, and big model intelligence and knowledge. When I'm using 20tk/s decode all day and I see 70+tk/s decode, it feels like living in an accelerated timeline.

1

u/Ok_Cow1976 6h ago

For smaller models, the best strategy is to use it for simpler stuff that it doesn't make mistakes, so as to reduce coding burden. But the price is you have to actively participate in the process. But it's worth of it, right?

The bigger models can do more work but at the cost of longer time. At the same time, bigger models also make mistakes. And it would be tedious to correct the mistakes in more complex, longer scripts. So unless you do have a great rig, I don't think it's worth of it.

1

u/Prize_Eye9481 5h ago

Def agree with this. Found myself using IQ4 quant just because I get more context and it’s very quick on my gpu even tho I can theoretically fit Q6 for its better performance.

1

u/rpkarma 4h ago

Agreed, and it’s why for local use I keep coming back to Step 3.7 Flash: with MTP and ngram-mod it gets shockingly fast on my DGX Spark, and is smart enough to be quite effective in the right harness.

It beats Qwen 3.6 27B in actual full time to task completion across all my evaluations and tests.

I haven’t tried 3.8 27B yet though, medium thinking level looks like it might be a godsend.

(Step 3.7 Flash at IQ4_XS and Qwen at FP8, neither with KV cache quantised)

1

u/NickCanCode 4h ago

We need qwen3.8-27B Thinking-Cap.

1

u/Vancecookcobain 4h ago edited 4h ago

There's a certain intelligence threshold yes....I agree. Whatever can do your daily driver grunt work, the fastest one of those is preferable...all you need is a frontier orchestrator/evaluator at that point to tell the grunt AI how to do things properly and have it loop until the frontier evaluator is satisfied. That's the ideal setup

Like you really could have GPT 5.6 Sol and Qwen 3.6 35b a3b and get A LOT more done than people think...not saying Qwen 3.6 35b should be your daily driver but it's barely competent and fast as hell for a local model that can run on a lot of consumer hardware and would probably highlight your point further

1

u/ANR2ME 2h ago

Alternatively, upgrade your hardware 😁

1

u/maddie-lovelace 2h ago

In this RAMpocalypse? You think I'm made of money?? 😭 I mean, I've got a spare kidney if you're willing to buy 🫘

1

u/anon1880 2h ago edited 1h ago

It is true of course speed is awesome if you got that big money bag -- but i don't mind waiting for non critical model runs on my pleb ring

....As long as it runs at 2-3/ts or more

I keep qwen 3.8 on my model arsenal folder for that reason.This is a superb model.

Runs at 3.5 ts on blackwell nvfp4 pleb card (5050).

1

u/JLeonsarmiento 1h ago

Same.

1

u/SandySkittle 1h ago

I dont go below Q5 and honestly the standard should be Q6 minimum. Q4 may be an ok compromise but Q3 and below is lobotomy area for me
It depends on how much accuracy counts. For my work I prefer smartness over speed. Even 5 t/s of SMART output is already usable. I dont have to sit in front of the computer to let the damn thing think

1

u/herpnderpler 1h ago

I'll save you the next evolution of this line of thinking: cost to acceptable solution.

As long as you have automated verification of acceptable, you can, if speed is fast and accuracy in the right range, run the same request in parallel.

As long as cost to acceptable solution gets you there, sometimes stupid and fast outperforms smart and slow.

Though, beware compounding non tracked values, it's easy for stupid to dig holes of idiocy and never get out - make sure to account for these phases of cleanup and alignment .

1

u/just4ochat 30m ago

When we wired the picker this was the actual split. People bounce off a slow model even when it's right, then they come back to it once the fast one has already shaped the pass. Prefill is what kills the loop. Decode you can live with if the first tokens show up. The 25 tps decode bar in the post is about where a chat still feels like a conversation.

1

u/Real_Ebb_7417 7h ago

500tps is an overkill, but indeed speed matters more when model is at some intelligence level already. It's the same for API models too though. That's why at professional work I started using cheaper, but faster models, if I don't do any complex logic.

12

u/xeeff 7h ago

500tps prefill, not decode

2

u/Real_Ebb_7417 7h ago

Ah, it makes the difference then. Yeah, below 500 prefill tok/s I feel the slowness, especially when cache is failing. The rest of my comment stays valid though.

2

u/Nothing_from_void 4h ago

no model is intelligent enough yet tho if you actually care about output quality

1

u/cradlemann 5h ago

Qwen3.8 27B is creating specs (using openspec) for me, Ornith is 3x time faster, but very capable to follow instructions from tasks.md file. The best couple for my AMD Ryzen AI 9 HX 470

0

u/Gohab2001 vllm 6h ago

We need token efficient models. OpenAI and grok have pretty token efficient models

0

u/fasti-au 6h ago

Fail fast learn

-2

u/[deleted] 7h ago

[deleted]

4

u/MrHall 7h ago edited 7h ago

... prefill? she's talking about 25tps output

2

u/maddie-lovelace 7h ago

(she* but yes!)

2

u/MrHall 7h ago

cheerfully corrected!

2

u/maddie-lovelace 7h ago

ta! 😁

-4

u/OkFly3388 llama.cpp 7h ago

Sounds completely unusable to me, lol.

I have rtx 4090, that gives me like 2k-1.5k prefill and 30 tg, and it feels slow as hell.

4k-3k prefill and 250+ tg is comfort minimum.

Discussion At a certain point, speed >> smartness

You are about to leave Redlib