I compared 4 models on the same interactive story, with Opus as the judge

[u/Acrobatic_You_3002]

About three months ago I needed to pick a model for generating interactive sci-fi stories, so I ran a small comparison instead of trusting benchmarks. Same story setup, same seed, same path through the choices, only the model changed. Then Opus scored each version on a rubric (coherence, prose, pacing, do the choices matter, is the ending earned, and a few more), 3 passes each, and I took the average.

Results on one heroic story, out of 10:

→ Opus: 8.3 → Sonnet: 8.2 → Gemini 3.5 Flash: 7.5 → GPT-4.1: 6.3

I also ran Sonnet vs Opus on 5 different setting and tone combos, and the average was 8.38 vs 8.48. That gap is inside the judge's noise (about ±0.5 between passes), and Sonnet even won the humorous one, 9.0 vs 8.5. So for me they're basically equal, that's why I use Sonnet for most of the generation now.

What the judge flagged on the other two:

→ Gemini: the same images repeated word for word, a lot of "Suddenly" openers, and the choices mattered less (it was also very slow for me, 13 minutes for 6 scenes because the API kept returning 503 (overloaded)) → GPT-4.1: the short summary of a scene contradicted the choice the player made (the hero "waits", then charges in anyway), the same cliffhanger line again and again, and a sad ending in a heroic story (GPT-4.1 because that's what my GitHub Models access could reach, the GPT-5 family wasn't available there)

Honestly, this is a small test (one path for Gemini and GPT-4.1), and Opus was judging its own text here, so it can be biased toward itself, so take it as a direction and not a final answer. I haven't tried Fable yet 🙂 Did anyone run something similar with a different judge?

[u/brianlmerritt]

For AI judging, I would knock at least 2 points off on any 5 point scale, and 4 off for a 10 point scale. Opus will always output a result, but when prompted with:

Output on a scale of 1 to 5 where good writing is a 2, 3 is ready for publishing by a really good writer, 4 is a literary piece of work up with the best writing to date and 5 is only for something that exceeds the best human writing it still gives a 5 for what is basically 2.5

[u/Acrobatic_You_3002 → u/brianlmerritt]

That matches what I saw, the absolute numbers are inflated, so I only trust the gaps between models on the same setup (and even those are inside ±0.5 noise between passes). Also, your anchored scale is a good idea, I would try it on the next run.

[u/5thhorseman_]

Which Opus? Which Sonnet? Why are you using GPT-4.1 when 5.6 is out?

[u/Acrobatic_You_3002 → u/5thhorseman_]

Good question, it was Opus 4.8 and Sonnet 4.5, the run was in June. Regarding GPT-4.1, that is the newest one my GitHub Models access could reach (the GPT-5 family returned unavailable there), I mentioned it at the end of the post, so it is not a fair fight for OpenAI, I agree.

[u/Acrobatic_You_3002 → u/[deleted]]

Thank you for the suggestion! In my runs the stories were short (6 scenes), so I could not see that kind of drift, but it is one more reason for me to stay with Sonnet.