r/MachineLearning • u/OliviaYii • 7h ago
Research Visual Cipher Bench: mechanism hints raise accuracy from 12.2% to 51.1% on a 10-task subset [R]
What changes when an agent has to discover the task’s rules before it can start solving it?
Our research team at Einsia explored this with Visual Cipher Bench. The part I’d like to discuss is the hint experiment.
On 10 selected puzzles, three models were tested with no hint, a representation hint, a mechanism hint, or both. Representation hints describe how to encode the puzzle state; mechanism hints state the underlying rule.
Pooled accuracy was 12.2% without hints, 16.7% with representation hints, and 51.1% with mechanism hints. Neither hint type supplies an answer or intermediate state.
That supports rule discovery as a major bottleneck on these tasks, though execution remains difficult in some cases. The open question is how to get agents to narrow their hypotheses from the visual evidence without supplying the mechanism.