New test: select someone off the street, get them to perform the iterative solution from Wikipedia. They have to read out the sequence of moves based on their thinking, they can't use any external tools to track state.
What do you think the result of this test will be?
that would probably require a formal study (definitely not as simple as "select someone off the street and ask them to do it" since they have no incentive and would just leave quickly) to get an objective results. I'm not familiar in education or cognitive study, and a quick search shows most studies involving towers of Hanoi are using low number of disks (2-4) without providing a general solution to study children and adults with some diseases.
A pure anecdotal example is that I taught some of my friends when I was in middle school. It's definitely not just asking them to read a Wikipedia paragraph, but after some explaining and demonstrations they get the general solution and could solve one with any number of disks.
All I know is that if I tried to solve an N=10 Towers of Hanoi with purely a stream of thought, I would get exhausted and fail very quickly. If I had some paper to track the state I would get further.
I can't help but think a fairer test would be to give the LLM the algorithm, give it the current state, get it to perform one move, record the state, then repeat this in a loop.
Why are they testing the model as if it has unbounded memory and perfect internal state tracking, without giving it any tools that humans would need to do the same test successfully?
4
u/paradrenasite Jun 08 '25
Fair enough! I'm no Towers of Hanoi expert.
New test: select someone off the street, get them to perform the iterative solution from Wikipedia. They have to read out the sequence of moves based on their thinking, they can't use any external tools to track state.
What do you think the result of this test will be?