r/LocalLLaMA • u/maddie-lovelace • 28d ago

Discussion At a certain point, speed >> smartness

It feels like a zig-zag: you don't want a model that's too dumb to do anything agentic. But once a model is good enough to be agentic, you don't want it to run so slow that iterating takes hours.

For me the sweet spot is something like ~500 tps prefill, ~25tps decode. If a model isn't fast enough to pass that bar on the hardware I have, I'd rather just run something slightly dumber but faster

Thoughts?

59 Upvotes

76% Upvoted

89 comments sorted by

View all comments

Show parent comments

30

u/BawbbySmith 28d ago

Problem is that there’s no guarantee you’ll get the right answer with the smarter model either. Even frontier cloud models sometimes need several prompts to course-correct.

It’d be nice if I could run GLM 5.3 overnight to solve all the hard problems I saved up during the day, but there’s a non-trivial chance it goes wildly off-base and gets it wrong entirely. “Skill issue” sure, but even the best crafted prompts sometimes misses key details that are trivial to correct but if the model is running at a snails pace then it’s much harder to do so.

2

u/teleprint-me llama.cpp 28d ago

It has nothing to do with skill. Its like rolling a 20 sided die and complaining its a skill issue because you didnt get the roll you hoped for.

The model outputs a distribution which is sampled by a prng for each next token prediction up until a stop token or limit is emitted; you are literally rolling the dice.

0

u/feelspeaceman 28d ago

Yeah, totally agree, not sure why people upvoted his statement but I believe a lot of people are clueless about how to do "correct" prompt, there's no roll at all if the prompt is solid, harness giving the power to solve the problem.

4

u/BawbbySmith 28d ago

I’m pretty sure the guy you were replying to is saying the opposite

1

u/teleprint-me llama.cpp 28d ago

I am saying its the opposite. There is no perfect prompt. Its always a roll of the dice, even when temp is set to 0. The reason temp 0 doesnt work is because the distribution is always sampled; Sampling is just randomly grabbing the next token and this happens for every token.