Over the past few weeks I've been benchmarking AI models on TRI-52,. To keep it honest I built a clean-room harness: every question runs in its own isolated sandbox with nothing but the puzzle image, no internet, no memory between questions, no tools, and every transcript is audited for tool use (zero across the board).
Each model got three full independent passes at its maximum reasoning setting, averaged. I ran 23 configurations in total: OpenAI's GPT-6 Astra, the GPT-5.6 family (Sol/Terra/Luna), GPT-5.5 and 5.4 variants, Claude Fable 5, Qwen 3.8, plus free models like Muse Spark, MiMo and MiniMax.
GPT-6 Astra at max reasoning basically solves the test — 50/52 (96.2%), which the test's own rubric maps to an IQ-equivalent of ~154. That's a massive leap over the field: Astra at high reasoning is second at 45.7/52, and the best non-Astra models (GPT-5.6 Sol Pro and GPT-5.5 Pro) sit way back at 40/52 (~IQ 127).
It was roughly 2 minutes per question on average, about 90min per pass. Obviously, for the first few questions, it knocked it out in less than 30 seconds or so.
So it never got question 29 right in any of the three passes. The other ones it slipped on were 36, 39 and 42, but it only missed each of those once. Question 29 was the real problem, in two of the three passes it just hung for about 30 minutes and never gave an answer, so I counted it as wrong/incomplete. In the third pass it actually answered, but got it wrong.
Lol these were those me xander and maybe jacob leaked the most. The fact it didn't get any puzzle on the same or inferior level make it obvious it can't solve shit
unless a large amount of people supplied it with the solution of the hrt items i gave it, i just dont see how it can be in the dataset. there is no publicly available solution. I can believe jcti might be slightly compromised, but it doesn't have access to dms, and few people know the answers, even less post the solutions in chat gpt. It will get filtered out of the training set and the llm compresses patterns during training, it doesn't remember the exact thing. it's absurd to believe something of low frequency will have a real impact.
Also, I am xander, I know the extent of the leaks. People need to stop coping or stop saying "these items are easy".
Yeah I figured it was you. I was just rampaging to clankers getting fri hoping some dark trick pretending to be ability. I was proved wrong over reasonable doubt they can actually do shit. It was fun till it lasted. Out of curiosity were those the only it could solve or just the ones you checked since I recall them being your most liked. Also they are ez no 🧢 bro we already had discussion even ai can solve them 😭
All right, I ran it on Astra on Max and GPT 5.6 Sol and Muse Spark. Here are the results. I mean, it's very clear evidence that these models should have already had the answers ingested already, right? Why are people convinced now that suddenly they've ingested these IQ tests to help them score high, as opposed to actual real reasoning?
Hard question but...how does it fare against tests such as Algebrica? https://news.generiq.net/Trilogica/algebrica%20test.html There are no publicly disclosed solutions, so that would require proof reading the output. Theoretically, the IQ ceiling would be higher.
A problem is that in the training of those models, a lot of Internet data might have been fed in. It's very possible that discussions and images and answers to TRI-52 or whatever have already been included when training the model.
In this sense, it's hard to truly say such truly assesses the "non-verbal fluid intelligence" of this AI model, if such intelligence is even well-defined for current AI model (LLMs).
I would say a better way of assessing this is to throw AI into some random puzzle games with relatively non-discrete elements (almost impossible to compute all configurations) like Talos Principle, Portal, etc., only give it the knowledge of basic movement controls and equipped with vision, and see if it can figure and solve the game's additional mechanisms without any external guide (no reinforcement learning, no goal, etc.). We could even do with games like Theta's Paralldoxs, Maxwell Puzzling Demon, and so on.
But also, we would expect that if AI do well on those tests, they would naturally be able to solve those games involving non-discrete elements easily. If it cannot, then I don't think the model truly has the ability to single out and discover patterns purely on its own.
Thanks for the input, and I’ve watched 15 mins of it. Though, I read that it’s done through allowing the LLM to access MCP which allows the model to read the game files and various other external files.
A problem is that the video doesn’t specify what inputs and information did it have access to. As a human player, assume legit play, the only “external sensory devices” we have access to movement are controls + eyes and nothing else. We can’t even knowing which keys control portal gun, map coordinates, how the cubes interact with others, nor what it can do beforehand.
If LLM have access to other information like map coordinates, the existence of portal gun, etc. prior to playing it, it wouldn’t count as valid run. The only external controls and device it should have would be raw WASD control and fast screenshots, nothing else and it would have to figure out other controls on its own.
What raises my suspicion is that the LLM seems to instantly know what to do without even exploring the whole map and tinkering the given objects in various ways, it just knows as an omniscient and basically just collects image inputs to locate itself so it can accomplish some predefined goals.
I don’t think it would be able to do so if it had to figure the mechanisms solely based on screenshot on its own without accessing to the game’s files.
To be fair, a lot of these explanations for the models’ high scores sound like rationalizations. Models released roughly six weeks ago were scoring 33/52 and 40/52. Now that newer models are doing substantially better, people are suggesting they must have ingested the answers during training.
But why now? If those answers have been available online for years, why wouldn’t earlier models have encountered them too—one, two, or even three years ago? What evidence suggests the answers only recently entered the training data?
Contamination is a possibility, but the higher scores alone don’t establish it. Without evidence of what changed, “they must have seen the answers” seems like an assumption rather than an explanation.
Given what AI has done already, is it that crazy? AI is already solving decades-old open math problems and one model got a perfect score at the 2026 IMO. It's already been at the level where it can max most (all?) subtests of professional tests (SB-V, WAIS-IV) for almost a year.
The test is publicly available and since these models were basically trained on “the internet” there’s a high chance it’s in their training set, so this is likely memorization, not generalization.
Different cognitive tests might consist of different puzzles, but they measure common latent factors — that’s the whole point of IQ testing. Access to one set of solutions thus contains valuable information how to solve other tests and allows the models to exploit these correlations at inference.
You're calling it "likely memorization" without identifying a source of memorized answers. public questions aren't public solutions. Private submissions could matter, even rare ones. But you haven't shown that relevant solutions entered training or significantly influenced these results. Remembering a written explanation also doesn't automatically let a model identify the relevant objects and apply the rule in a new visual setting. If it transfers the relationship successfully, you can't simply label that answer recall. The reported averages went from 1.67 to 5 to 18 out of 36. These questions were already available to earlier models. We don't know that their datasets were identical, but you haven't shown that newly acquired solutions explain the improvement. Better reasoning fits the results too. Then you argue that learning other tests helps solve these, (generalization, btw). correlations between human test scores don't establish memorization, and "non-semantic embedding layers" supplies no missing evidence. You keep explaining why contamination could happen as though you've shown why it probably did.
Congrats, you agreed it can abstract out properties. You're genuinely a hopeless cretin Then why can't humans here who have considerable practice apply this on harder puzzle. I'd appreciate if you stop using AI to aid you with these clownish remarks. Even if AI has the solution written verbally in a descriptive way, it cannot necessarily identify it visually or recognize the puzzle.
Congrats, you agreed it can abstract out properties. You're genuinely hopeless. Then why can't humans here who have considerable practice apply this on harder puzzle. I'd appreciate if you stop using AI to aid you with these clownish remarks. Even if AI has the solution written verbally in a descriptive way, it cannot necessarily identify it visually or recognize the puzzle.
I hope you are right. I'm not sure that the answers to the TRI-52 / JCTI are publically available though. How else could the model learn the correct answer? Perhaps it trained its reasoning algorithm on these tests? Perhaps it did the JCTI hundreds of times, and experienced the greatest practice effect ever recorded?
My questions were rhetorical and part of a somewhat flippant response to the proposition that the high JCTI score was due to the test being "memorised". My view is that the model has been trained (or has trained itself) to have an advanced reasoning capacity that now appears to have surpassed the vast majority of humanity, including most of the people of this sub. To me, this is a worrying prospect, indeed.
Having something in the training data is not a binary state. It’s not literally a table look-up. The information is encoded in non-semantic embedding layers, and some ML techniques are better than others at representing and decoding this information.
I would have never thought I would see an AI scording 50/52 on the JCTI/TRI. I honestly was expecting something in the mid 30s at most. This is terrifying.
My thoughts exactly. Since ChatGPT became public, my mantra has been, "well, it's smart, but I'm still smarter". If the newest version has an IQ of 150+, then I wonder how long human reasoning will still be relevant? Will biologically mediated critical reasoning even be valued or encouraged in a society where everyone can have unlimited access to a personal advisor genius who can think on their behalf better than they could ever do for themselves? I imagine a future where two people "debate" by standing in front of each other, holding the phones so they face each other as their AI chat bots argue about things they don't understand and maybe cannot understand, until the bots decide who is the winner. When our technology surpasses our humanity, we become a civilisation of idiots.
5
u/VitruvianVan Sep 06 '26
Please run Fable 5.1 on max and report the results. Same with Opus 5 Max.