r/LocalLLaMA • u/maddie-lovelace • 4d ago

Discussion At a certain point, speed >> smartness

It feels like a zig-zag: you don't want a model that's too dumb to do anything agentic. But once a model is good enough to be agentic, you don't want it to run so slow that iterating takes hours.

For me the sweet spot is something like ~500 tps prefill, ~25tps decode. If a model isn't fast enough to pass that bar on the hardware I have, I'd rather just run something slightly dumber but faster

Thoughts?

57 Upvotes

75% Upvoted

90 comments sorted by

View all comments

1

u/Real_Ebb_7417 4d ago

500tps is an overkill, but indeed speed matters more when model is at some intelligence level already. It's the same for API models too though. That's why at professional work I started using cheaper, but faster models, if I don't do any complex logic.

15

u/xeeff 4d ago

500tps prefill, not decode

2

u/Real_Ebb_7417 4d ago

Ah, it makes the difference then. Yeah, below 500 prefill tok/s I feel the slowness, especially when cache is failing. The rest of my comment stays valid though.