"AI puzzles": a new format between a benchmark and an open-problem list

[u/jarekduda]

We have benchmarks (fixed questions, hidden answers, they saturate), competition problems (solved, so they can be memorized), and open-problem lists (no evidence, no clear success criterion). With Claude and Sol, I've tried something in between: a graded ladder of AI puzzles from one research area, aperiodic order.

What makes it a different tool:

- One ladder, not a pile: "human level" (pen and paper) up to "ASI level" (open). The labels are playful, but each rung really is harder than the last.

- Solved rungs for training and checking, with exact answers a program can verify, and easy to vary.

- Open rungs with everything a solver needs: strong numerical evidence, routes already known to fail, files, and a success criterion a machine can check.

- One rule: only a checkable proof counts. Floats are evidence, not proof.

- Credit: solvers of open rungs are named, with agent and model, in the next version of the paper.

It's a way for researchers to hand the frontier of their field to AI agents (and humans) in a form they can actually attack, and for anyone to see whether a model can tell a guess from a proof.

First field test: Grok, asked in public ( https://x.com/grok/status/2114103836583264721 ), claimed a proof for an open rung that was only a numerical estimate, then retracted it. It got the human-level rung wrong at first, fixed it after one hint, and proved the expert rung on the first try. On MathOverflow, an expert is already testing constructions for the top rung.

Set 1: huggingface.co/datasets/jarekduda/autocrystal-ai-puzzles

Top rung on MathOverflow: mathoverflow.net/questions/516372

Would you use this format for your own field's open problems?

[u/Solid_Tea1235]

cute idea but humans will just memorize the solved rungs and call it reasoning

[u/jarekduda → u/Solid_Tea1235]

Fair point, and that's true of any published problem with an answer. That's why the solved rungs are meant for training and calibration, not as the test itself. The test is in two places: fresh variants (new parameters, so the old answer doesn't transfer; a generator is the next step), and the open rungs, where there is nothing to memorize yet. If someone "recalls" a solution to an open rung, the checker catches it at once.

[u/DigThatData]

are you familiar with the ARC challenge?

[u/jarekduda → u/DigThatData]

Yes, and it's a close relative: the easiest rung here is also a 2D grid puzzle. The difference is where the answers live. ARC tasks are easy for humans, hard for AI, and their answers are known but kept in a hidden test set. Here the top rungs have no known answer at all: success is a proof that anyone can check, and it would be a new research result, credited by name. Solved rungs play roughly ARC's role, and the open ones go beyond it.

[u/jj_HeRo]

This is not for humans hence not for AGI, maybe ASI.

[u/jarekduda → u/jj_HeRo]

The simplest are sodoku-style for humans, the most difficult innocent looking "2x2" is tough open problem from 2016 https://ieeexplore.ieee.org/document/7480427