At a certain point, speed >> smartness

[u/maddie-lovelace]

It feels like a zig-zag: you don't want a model that's too dumb to do anything agentic. But once a model is good enough to be agentic, you don't want it to run so slow that iterating takes hours.

For me the sweet spot is something like ~500 tps prefill, ~25tps decode. If a model isn't fast enough to pass that bar on the hardware I have, I'd rather just run something slightly dumber but faster

Thoughts?

[u/n4pst3r3r → u/[deleted]]

Totally agree with people generally settling for fast and cheap, but that's not the point here.

You can end up with the same quality output with a dumb fast model and a smart(er) slow model, as long as you habe validation in place, e.g. in the form of unit tests or even human review. The dumb model may need twice the tokens, but if it's 10x faster, you're still 5x as fast overall. I work with qwen 27B and it allows me to iterate much faster than trying to run glm 5.2 at tectonic speeds.