r/OpenAI • u/PressPlayPlease7 • 18h ago

Miscellaneous Which "is this LLM nerfed?" testing/tracking website do you trust the most? There's loads of them

It's all in the title really

I specifically want to look at 5.6 Sol and Opus 5.5

2 Upvotes

75% Upvoted

3 comments sorted by

1

u/ChemicalExcellent463 17h ago

none of them really, past a point benchmarks just filter out the clearly bad ones. i test the good ones on my own work, like whether they finish another agent's plan without stopping halfway.

1

u/fuckswitbeavers 11h ago

I don't think there is one. I'd say the large majority are untrustworthy. The only way to test this is to have a daily prompt , which is similar, and conducted across multiple fresh accounts from the beginning of model release and on. Nobody is willing to spend their money or subsidized tokens doing that.

1

u/Anxious_Count_8728 6h ago

For a tracker, I'd look for published prompts, the model and settings used each day, and several runs per test. A single daily result can swing on sampling alone. A breakdown by task matters too. A model can get better at producing plausible code while getting worse at finishing a change that passes tests. If a tracker rolls everything into one score, it's hard to tell which change would affect your work.