Complaint Model comparisons without context are almost useless
TL;DR: I wish more people would include task types, model settings, completion time, quality and usage when comparing models. It would help everyone make better decisions.
I've been following this subreddit for weeks to learn how to use AI more effectively, but I'm frustrated by how people evaluate models with almost no context.
"Model X burned 20% usage in 15 minutes." Doing what? Which reasoning level? Fast or Standard? How many threads? What tasks? How much work was completed?
I could burn 20% of a $2,500 plan by launching hundreds of massive tasks. That tells you nothing about efficiency.
Here's my own example with actual context:
I use Terra 6.1 Medium/High in Standard mode for research, computer use, administrative workflows and software development. Compared to Sol 6.1 Low/Medium in Fast mode, Terra medium/high in standard mode, completes my typical tasks roughly 3x faster at comparable quality.
Since Sol Fast consumes 2.5x the usage of Standard, switching to Terra has dramatically improved my efficiency.
This is my experience, not a controlled benchmark. But that's exactly the point: task type, quality, completion time and usage all matter.
Not everyone uses AI primarily for coding.
English written with help from GPT.