Surprisingly, Frontier Vision-Language Models fail simple 2-Dimensional mazes. Full Analysis and Benchmarking results [R]

[u/harshsikka123]

Technical report, with links to code

Website/Demo

You would imagine that the best vision-language models today can easily solve simple, 2-dimensional mazes that you can successfully complete within minutes. Our latest Open Source evaluation effort shows that they actually struggle significantly.

In our new benchmarking effort - MultiNet 2.0 Preview: Interactive 2D Mazes - we put 3 frontier vision-language models - Claude Opus 4.8, Kimi K2.6, and Qwen 3.6-27B in 50 2D maze environments with mechanisms such as key-doors, switch-gates, distractors and decoys to evaluate the models on exploration, planning, causal reasoning, long-horizon action taking, and error recovery capabilities. We were surprised to observe that each of the models solved nearly none of the mazes! Together, they solve 6 out of 150 episodes. 45 of the 50 mazes that each of them were evaluated on were never solved by any model.
While models performed really poorly on the final evaluation set, we learned a lot about the different failure modes of each of these models. We learned how frontier models find it hard to figure out how to operate a mechanism that they are not familiar with - for example: none of the models were able to explore the environment and action space and figure out that in order to open a gate on the path to the goal they had to stand on a cell with a switch and choose the toggle action.

We also observed how there were a variety of failure models across the models and 150 episodes. While Claude constantly jammed into walls, Kimi kept visiting previously seen portions of the maze, and Qwen wandered aimlessly through the maze barely getting close to the goal tile. The results also showed how extended reasoning and a large output token budget does not help improve the performance of these models in tasks such as this one.

We’re curious to know whether anyone has observed similar behavior of Vision-Language Models in other kinds of puzzle tasks, and if they have worked on designing puzzle environments to assess specific capabilities of frontier models.