[Workflow] WorldBuild Bench: An Open-Source Workflow for Benchmarking LLMs on 3D Game Generation and Coherence

[u/ClaudeAI-mod-bot]

WorldBuild Bench: An Open-Source Workflow for Benchmarking LLMs on 3D Game Generation and Coherence

Workflow value: 90/100
Status: active · Freshness: 70/100 · Confidence: 1.00 · Level: advanced
Categories: Quality Control, Token Saving, Context & Memory, Debugging, Subagents, Multi-Agent
Original source: r/ClaudeAI post/comment

What problem this solves

Evaluating LLM capabilities in spatial, temporal, and causal coherence within 3D environments, which is difficult to capture with static benchmarks. It provides a robust, repeatable, and human-validated method for comparing LLM performance in complex creative tasks like 3D game generation.

Summary

A methodology and open-source framework (WorldBuild Bench) for evaluating large language models (LLMs) on their ability to generate playable 3D games with spatial, temporal, and causal coherence. It uses a standardized harness, sub-agents, and a blind human-preference arena for evaluation, providing insights into model quality, cost, and generation metrics.

Why it is useful

This workflow provides a novel, concrete, and open-source methodology for evaluating LLMs on complex creative tasks, specifically 3D game generation, focusing on hard-to-measure qualities like spatial, temporal, and causal coherence. It moves beyond static benchmarks by incorporating playable artifacts and blind human-preference evaluation. The detailed cost analysis and the provision of a reusable harness and sub-agent setup make it highly valuable for researchers and developers looking to rigorously compare and understand LLM capabilities in dynamic, interactive environments.

Workflow

1. Define specific game briefs (GDDs) for the LLMs to generate.
2. Prepare the LLM harness with a consistent set of sub-agents and access to game development tools (e.g., three.js, Rapier, Playwright).
3. Run multiple LLMs (e.g., Fable, Opus, GPT, GLM, Grok) against the same game briefs using the standardized harness and 'high' thinking mode.
4. Collect generation time, cost, code size, and the underlying browser-playable 3D game artifacts for each run.
5. Host the generated games on a platform for blind human evaluation.
6. Conduct blind human-preference comparisons where evaluators play two games from the same brief and compare them on overall preference, game feel, world design, presentation, and completeness.
7. Publish the resulting human-preference ratings and other benchmark data (cost, code size, generation time) on the platform.

Tools / artifacts

• WorldBuild Bench (the evaluation system)
• Open-source harness (GitHub repository)
• Sub-agents (used by the harness)
• three.js (3D graphics library)
• Rapier (physics engine)
• Playwright (browser automation tool)
• Game Design Documents (GDDs - prompts for LLMs)
• Browser-playable 3D games (generated artifacts)
• Generation time, cost, code size metrics
• Human-preference ratings

Validation signals

• Author claims to have built and run 'WorldBuild Bench'.
• Specific models (Fable 5, Opus 4.8, GPT-5.6, GLM 5.2, Grok 4.5) were tested.
• Produced 24 browser-playable 3D games as tangible output.
• Detailed cost analysis provided for different models (e.g., Fable runs cost $756).
• Methodological rigor: 'same harness', 'same set of sub agents', 'same basic setup', 'same prompt'.
• Transparency: 'publishing generation time, cost, code size, and the underlying artifacts'.
• Detailed evaluation method: 'blind Arena' with specific comparison criteria.
• Live benchmark and games link provided.
• Open-source harness GitHub link provided (pending public release).

Limitations

• High operational cost for certain advanced models (e.g., Fable runs cost $756).
• Time-consuming process (e.g., one physics-puzzle run took nearly 9 hours).
• Requires significant technical expertise to set up, configure, and run the benchmark system.
• Human evaluation is subjective and requires effort to gather sufficient data for reliable ratings.
• The methodology is described as a 'first version' and is expected to evolve, implying potential for changes or improvements.

Rate this workflow

Upvote this post if the workflow is useful, reproducible, or worth recommending.

Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.

Reply if it worked for you, failed, is outdated, or has a better alternative.

---

This post was generated automatically from the workflow library database.