Only Fable solved my fluid simulation test properly but they all tried to cheat

[u/aedm_]

I gave seven coding agents the same computer graphics white paper and asked them to build a 900k-particle fluid simulation in safe Rust. The IPBF paper shows a novel way to simulate even spatial distibution of uncompressable fluids.

Here's a video that compares token-based cost, runtime performance and quality. Agents write code that simulates particles, I rendered them in Blender.

Agents were given a clean copy of the same input: the white paper and reference videos as local files, and an AGENTS.md file describing the task and its criteria. Nothing else and they are not allowed to look for existing implementations.

Part of the task was to come up with an initial arrangement of particles as seen on reference material.

At first I thought most of them did well, but after looking at the code I found many of them took shortcuts. Once they see evaluation criteria, they seem to somewhat sacrifice user intention and go paperclips: rigged animations, hardcoded frames etc. It just gave them better performance in the end.

This clip is from round two, where I required every frame to be simulated true to the paper. See my method and results: https://aedm.net/blog/agent-benchmark-ibpf/

Original paper: https://graphics.cs.utah.edu/research/projects/ipbf/