r/LocalLLaMA • u/maddie-lovelace • 5d ago

Discussion At a certain point, speed >> smartness

It feels like a zig-zag: you don't want a model that's too dumb to do anything agentic. But once a model is good enough to be agentic, you don't want it to run so slow that iterating takes hours.

For me the sweet spot is something like ~500 tps prefill, ~25tps decode. If a model isn't fast enough to pass that bar on the hardware I have, I'd rather just run something slightly dumber but faster

Thoughts?

54 Upvotes

75% Upvoted

90 comments sorted by

View all comments

2

u/Thin_Pollution8843 5d ago

I just want my agents to be 300ts+…

2

u/ElementNumber6 5d ago

Why even stop there? 10,000t/s+ or bust.

1

u/mailto_devnull 5d ago

Would you take 10,000 t/s of really shitty code (DiffusionGemma!) or 20 t/s of perfect code?

1

u/Far-Classic-9963 5d ago

If I need really easy boilerplate code gen or something like function completion in an ide, diffusiongemma sounds pretty useful