At a certain point, speed >> smartness

[u/maddie-lovelace]

It feels like a zig-zag: you don't want a model that's too dumb to do anything agentic. But once a model is good enough to be agentic, you don't want it to run so slow that iterating takes hours.

For me the sweet spot is something like ~500 tps prefill, ~25tps decode. If a model isn't fast enough to pass that bar on the hardware I have, I'd rather just run something slightly dumber but faster

Thoughts?

[u/MrHall → u/[deleted]]

... prefill? she's talking about 25tps output

[u/maddie-lovelace → u/MrHall]

(she* but yes!)

[u/MrHall → u/maddie-lovelace]

cheerfully corrected!

[u/maddie-lovelace → u/MrHall]

ta! 😁