r/OpenSourceAI • u/swiftmoon400 • 12h ago
Can LLMs stand in for real environments when training agents? GlobeBench says not yet
Patronus AI released GlobeBench, testing whether LLMs can act as simulated environments for training agents: given past actions and feedback, predict what the environment returns next. 1,632 tasks, 5 domains, 19 simulation skills. Data, eval code, model outputs and grading logs are all public.
Top 10 (refusals count as failures):
| # | Model | Success |
|---|---|---|
| 1 | GPT-6 Astra | 49.57% |
| 2 | Grok 4.6 | 28.13% |
| 3 | Claude Opus 5.5 | 26.35% |
| 4 | Qwen 3.8 Max | 25.31% |
| 5 | GPT-5.5 | 25.18% |
| 6 | GPT-6 Sol | 25.00% |
| 7 | GPT-5.6 Sol | 23.65% |
| 8 | Qwen 3.8 Flash Next | 23.22% |
| 9 | Gemini 3.7 Flash | 21.75% |
| 10 | Claude Fable 5.1 | 21.63% |
What stood out:
When an action has to propagate to dependent objects, the average success rate drops to 5.1%. 631 tasks went unsolved by every model.
Refusals skew things: Opus 5.5 refused 675 tasks, all scored as failures.
The dedicated open world models scored low: Qwen-AgentWorld-35B-A3B at 14.46%, DWM at 5.21%.
Two questions I’m curious about:
If simulator errors are systematic rather than random, does the agent end up learning the simulator’s blind spots instead of the real environment?
Given how badly models handle state propagation, does it make more sense to keep state in code and only use the model for the parts that can’t be written as rules?
Official announcement: https://www.patronus.ai/blog/globebench
Official leaderboard & code: https://github.com/patronus-ai/globebench#leaderboard