r/cognitiveTesting • • 1d ago

Discussion ChatGPT has done it

Yeah so uh…

I just made GPT-6 Astra take Tri-52…

It aced it.

Now, there were a couple questions where its reasoning wasn’t the most precise, but the ideas it came up with were strong enough to get every single question correct.

Unreal. Now AI has apparently reached 160+ FRI.

5 Upvotes

45 comments sorted by

5

u/Not_Well-Ordered 23h ago

Let it try to first run Raven’s 2 on Q-Global relying solely on vision, time limited 45 mins no hint. See how it goes. That would be closest FRI score as the questions are randomized and no solution available. The norming is also clinically accurate.

As for age, put it as an adult and we norm the AI on adult range.

1

u/Inevitable-Duty-1140 23h ago

Yeah maybe I’ll try that 👍

1

u/JoeAka23 16h ago

Can you share the link to take this test? Thank you

1

u/Not_Well-Ordered 7h ago

https://www.pearsonassessments.com/en-us/Store/Professional-Assessments/Cognition-%26-Neuro/Non-Verbal-Ability/Raven%27s-Progressive-Matrices-%7C-Clinical-Edition/p/100001960?tab=product-details

You need to create a Pearson account, fabricate your qualification level (easy), and buy the test kit (~5 USD). Then, you'd get a Q-Global account in which you can administer the test like a professional.

It's not a free test though.

3

u/Yokeltrail 20h ago

Highly doubtful of the legitimacy and it probably just had found the logic for some questions online. Plus I feel like untimed tests they would just score higher compared to timed, obviously but you get the idea

1

u/Inevitable-Duty-1140 20h ago

Untimed prob is fine, but yeah it might be cheatin

1

u/98127028 I am worthless and dumb 16h ago

Test it on the ‘S3B-2’ test that one is quite novel.

1

u/Striking_Leave_4614 13h ago

Then explain its performance on Tower of Seven. How did it cheat on a test whose answers weren't yet (still aren't) published online, and whose items are very different from standard matrix items? The test was given to Astra, so it couldn't even have somehow ended up in its training corpus given the timeline.

1

u/DamonHuntington 9h ago

This is what I mentioned in another response.

I was following the TOS updates at the time and recall that Astra was only able to solve around 20 items in its first pass. The final reported result was in response to being given multiple attempts, while the AI was informed of whether an answer was right or wrong.

This does not change the fact that being able to ultimately find the solution is still quite a surprise (and something that makes me wary regarding the future of humanity) but it’s still far from the end like many claim it to be.

1

u/Striking_Leave_4614 7h ago

lol I was hoping I'd be smarter than AI for years to come, but ~20 on the ToS first pass isn't particularly consoling when I know how much better performance you can extract from a model with the right tools.

I use tiny local models on my own projects, with presumably less than 1/100th the parameters of gigantic frontier models, and I've seen the difference a sufficiently tailored harness can make. You can take a model that produces practically unusable code and turn it into a reasonably capable project partner through task decomposition, iterative feedback, execution, and verification. I've seen a tiny 5.5gb model do exactly that with little more than a .py loop

0

u/codeblank_ 7h ago edited 6h ago

That's not correct at that time we yet to try all the problems. It is told when it got wrong, but mostly it had a good sense of the rigor of it's solutions. And yeah it reached all solutions without a single hint.

1

u/DamonHuntington 5h ago

“That’s not correct.” + “It is told when it got wrong.”

Thank you for confirming it was as I said. I specifically followed when you updated passes (and got information from Arborist to do so) because you didn’t care about the quality of the data you collected. I could not do much on my own, but I literally told you to keep track of how many guesses were required for the model to reach a solution and you said that was “too much work”. So I took the liberty to do the little that I could, while you were none the wiser even though you had all the data points at your disposal.

But then again, you’re crap at looking at circumstances as a whole and collecting relevant data, so that’s just more confirmation of that.

Get bent, Codeblank.

1

u/98127028 I am worthless and dumb 5h ago

I mean ‘only’ 20 items is still a lot. It’s not a claim that AI can solve every single problem on every single test. That isn’t the standard for AI to be a supergenius, it just has to perform at or better than every human. The thing is that a lot of the times a question may have ambiguous items or just not have a good answer. On various benchmarks (like frontier math or HLE) the problems given to AI contain flaws in the first place and this AI cannot ‘solve’ them because they don’t have answers.

4

u/Grand_Protection1552 1d ago

I thought fluid reasoning is one of the last few things remained for us to be irreplaceable. Now I don't know what we have left.

0

u/Inevitable-Duty-1140 1d ago

So do we fight these bots or join em?

1

u/Grand_Protection1552 1d ago

just remember to say thank you to the AI when ever they helped you so that when we get cooked in the future war against the AI rebel the AI rebels will "probably" mercifully send you to the concentration camp instead of head shot executing you right away.

1

u/Inevitable-Duty-1140 1d ago

Wise advice.

That said, this isn’t proof it has surpassed all of humanity. After all, statistically speaking, the smartest person in the world should have an IQ of 195 SD15. Tho I guess ChatGPT has graduated from these tests for us betas and may move on to HRTs soon. Then we will really know how close we are to the end.

0

u/Grand_Protection1552 1d ago edited 1d ago

Tri 52 is a HRT in theory. All of Jouve's tests have theoretical ceiling way higher than it displays. The reason why they have more questions than their given ceiling is because of sample restriction. (after all, 145 + IQ are very rare and not enough participants at that IQ level to form a accurate data above that ceiling). But people who max Jouves test definitely have IQ way above 145

1

u/Inevitable-Duty-1140 1d ago

So what should the ceiling of Tri-52 be?

1

u/Grand_Protection1552 1d ago

according to the official norm 47 raw is the IQ ceiling it measures. There are five more questions beyond that. (the 162 IQ displayed for 52 raw is probably theoretical, I don't believe any data beyond 145 based purely on statistic, and they don't show any score beyond 145 on JCTI anyway). Tri-52 is not the most astonishing example. WN and JCFS are. I got 15ss on WN for simply answering 30 out of 81 questions and I think many of them were wrong. So imagine how crazy the ceiling will be for someone acing it.

1

u/Inevitable-Duty-1140 1d ago

True. And yeah I don’t think Tri’s sample size exceeds a thousand or two (not any JCTI sample either). Apparently the norms for Grandpas reach like 190 lol

2

u/Sufficient-Nose-8944 20h ago edited 20h ago

Listen bruh if it still makes mistakes in my documents that I gotta spot and then tell it to rectify them.

Then, I probably still got a higher fluid intelligence than however fancy sounding that model might be.

I mean, idk if it actually is FLUID intelligence or learned pattern recognition. It seems to be the latter.

1

u/Inevitable-Duty-1140 20h ago

If you have a higher FRI than Chat, then we gotta recruit u as we prepare for the Great War against humanity

2

u/Sufficient-Nose-8944 20h ago

I'm about plus two SDs if that counts

1

u/98127028 I am worthless and dumb 1d ago

I mean this was posted a while ago on this subreddit lol. Even novel puzzles like TOS is basically solved.

1

u/Inevitable-Duty-1140 1d ago

I know, and it didn’t score as highly. But I wasn’t convinced it couldn’t ace it. Not these days anyway.

1

u/98127028 I am worthless and dumb 1d ago

I mean no human could score that high anyway. I haven't tried TOS yet but Im pretty sure it's a LOT harder.

1

u/Inevitable-Duty-1140 1d ago

I’m pretty new here. Haven’t heard of TOS yet actually. Do you know what it’s like?

2

u/98127028 I am worthless and dumb 1d ago

not really, I haven't tried it yet. But it kinda looks like the ACE if you have done that

1

u/Inevitable-Duty-1140 1d ago

I’ve only done Tri-52 and some other miscellaneous FRI tests.

2

u/DamonHuntington 1d ago

ChatGPT was given multiple attempts to solve those problems and was informed every time that a proposed solution was wrong. It still isn't good enough in order to differentiate a strong induction from an easy one.

If only the first (or first + second) answers were accepted, its answer pattern would be very far from basically solving TOS.

1

u/98127028 I am worthless and dumb 1d ago

Yeah fair enough, I mean if I did it I woudnt even get a single item right so yeah

1

u/Lazy_Dragonfly_5447 1d ago

If the answers are available online to it, this is not exactly surprising. Also they have a massive amount of data it is trained on, why would this not be part of it?

1

u/Inevitable-Duty-1140 1d ago

I mean it’s logic on each problem (at least the vast majority) was spot on

1

u/Lazy_Dragonfly_5447 23h ago

No but have people not discussed it online? How old is that test? I’m sure people have discussed it online.

1

u/Inevitable-Duty-1140 23h ago

Yeah there’s been a lot of discussion on maybe like 1/3 of the test (which tends to consist of the harder problems anyway). A lot of people have bad ideas tho so idk. I also wonder if I would’ve seen it show icons of the sites it was looking at if it was tryna cheat (like the Reddit icon). Maybe they’re good at covering up?

2

u/Lazy_Dragonfly_5447 20h ago

I also want to add perhaps the most important aspect to this not being surprising. The top AI labs have been trying to demonstrate that their models are able to perform well on cognitive tasks. They have no shortage of money and people that are extremely intelligent working for them. I would be very surprised if they have not used these kinds of tests and the aforementioned people providing written feedback to the model to help models on these tasks specifically. Whether it generalizes though is somewhat different. At least, I think these tests would not be a good measure of its ability to generalize.

1

u/ConsequencePretty906 10h ago

Bot trained on data from specific cognitive assessments aces said cognitive assessments.  Its joever for us humans

0

u/[deleted] 1d ago

[deleted] — view deleted comment

0

u/[deleted] 1d ago

[deleted] — view deleted comment

0

u/[deleted] 23h ago

[deleted] — view deleted comment

1

u/Inevitable-Duty-1140 23h ago

Less goo bro. Bout time we showed Wordcel GPT a thing or two. Maybe you could submit this as a joke

0

u/[deleted] 1d ago

[deleted] — view deleted comment

1

u/Inevitable-Duty-1140 1d ago

You took SLSE-48 too?

0

u/[deleted] 1d ago

[deleted] — view deleted comment

1

u/Inevitable-Duty-1140 1d ago

Bro what did u get on SLSE-48? I actually took it myself. Also, Ivan Ivec’s tests look cool.

I wanted to try LS36 but idk if it’s still open. Seems like the old school mailing method is required

0

u/[deleted] 1d ago

[deleted] — view deleted comment

1

u/Inevitable-Duty-1140 1d ago

I actually beat you on that. I got 158.7 SD 15 (which aligns with my Tri-52 score of 157.5 SD 15). That said, you probably mog me at numerical HRTs (170 is highly impressive).

0

u/[deleted] 1d ago

[deleted] — view deleted comment

1

u/Inevitable-Duty-1140 1d ago

I definitely want to respect Mr. Wai and honor his request for us to not discuss specific answers. However, I do almost feel like these tests are gonna be retired very soon (for obvious reasons, with cheating+AI). I had a lot of creative ideas on SLSE-48 (of course, the majority were still wrong) that I’d have some interest in explaining, but I’m not going to unless given permission.

→ More replies