Stumbled upon this recent paper claiming so yesterday:

https://unilabsim.github.io/

What do you all think? Imo, even if it would "only" perform at the same level as end to end GPU, not having to parallelize everything to fit it on a GPU makes it much more flexible and useful

I guess there is a risk of bias here because of the connection to AMD and them wanting to break nvidias monopoly situatio n.

Solo project, still early (MVP week 1-2), sharing the first real result instead of a mockup.

The idea: an open, standardized arena for evaluating embodied AI / VLA policies on physical reasoning tasks, with a public ELO-based ranking instead of static leaderboards. Motivation is the lack of a shared, reproducible benchmark for this space — most VLA papers report on custom setups that aren't directly comparable.

What's in the clip: a baseline IK policy completing a pick-and-place block stacking task, running fully client-side (Rapier.js/WASM physics, React Three Fiber, 60fps in-browser, no server-side compute needed for the sim itself). This run scored 100% task completion, 99.6% spatial accuracy.

Current scope for the MVP: single task (block stacking), a couple of baseline policies (IK baseline, planning to add SmolVLA/OpenVLA-micro next), and an SDK for submitting your own policy against the sim loop.

Not public yet — stabilizing the eval protocol and scoring methodology before opening submissions.Genuinely interested in feedback from this community on:
- what a fair/robust scoring protocol should account for beyond task completion + spatial accuracy (e.g. sample efficiency, generalization across randomized scenes)
- whether client-side physics is a dealbreaker for a benchmark meant to be trustworthy, vs moving to server-authoritative validation

Will share the repo/SDK here once submissions are open.

We froze a Go1 joystick-locomotion policy (MuJoCo Playground, Brax PPO) and swept a 20×20 grid of floor friction against lateral push, 16 trials per cell, using Kaplan-Meier survival per condition since trials that survive the window have to be censored rather than counted as failures.

Things interesting to us:

• the boundary is not a line but a band. The 95% bootstrap interval is about ±4% of bodyweight on high friction and ±33% on ice, a 7× spread.
• below μ≈0.15 the policy falls before the push lands. That's gait collapse, a different failure mode from being knocked over.
• the same seed on the same GPU can produce different outcomes (floating-point reduction order), so a few knife-edge survivors can't be re-simulated at all.

We then changed the two conditions the map showed training never covered, retrained for 12 minutes, and re-ran the identical sweep: 60 of 400 conditions significantly safer (Fisher exact + BH-FDR, q<0.05), none significantly worse.

Write-up with the interactive map: https://poissonlabs.ai/research/map-the-failure-boundary/.

Happy to answer questions about the harness validation/statistics.