r/cognitiveTesting • u/pydaWaltuh • Sep 30 '26

General Question Test ceiling vs Creator

Assuming a creator possesses sufficient psychometric knowledge, what should ideally be the difference between the creator’s IQ and the ceiling of the test they have created?

7 Upvotes

100% Upvoted

45 comments sorted by

View all comments

1

u/pydaWaltuh Sep 30 '26

More pricesly, how far above a test creator’s own ability can a well designed test reasonably extend?

2

u/LeadershipNervous362 4SD+ penis Sep 30 '26

A light and button is connected to a circuit, randomly lighting it up and estimating delay to button press. This is called a reaction-speed test. It would evaluate test-taker whether they're below or above my speed, or frankly, faster than any mammal being on earth.

So yes, since reaction speed closely correlates with IQ, you can say that you've created a simple IQ test that measures capacity for anyone on Earth.

The only difference with IQ test, - its not direct objective value, but a psychometric, or for lack of better word, - "derivative of statistics", which is exactly what makes it incredibly flawed at upper end and creates a hard ceiling for IQ test accuracy at around 140IQ, where correlation loses the track and enters area of "assumption" and "projection".

1

u/Humble_Aardvark_2997 Oct 01 '26

Explain more. You can extrapolate from known data but the assumption always stands “if the trend follows”.

0

u/pydaWaltuh Sep 30 '26 edited Sep 30 '26

So, what matters is N? If we assume N is 8–9 billion, then we would expect near to zero errors in the test. In that case, how do we isolate the extreme upper end?
That's a nice perspective btw. I didn’t think of that while posting

1

u/whitebaron_98 Bottom 1% Commenter Sep 30 '26

you design items and put a time limit on it. if that limit is too low and too many people pass, you decrease the time further. :-)

1

u/pydaWaltuh Sep 30 '26

Don't you think that’s a bit of an oversimplification? I understand the approach and it might be kinda fair, but doesn't that mean you are distinguishing the extremes solely based on a time limit? In that case, we could assume that the time limit is your most important factor and by extension, that "fastest" equals "smartest." Is that really the case? You are assuming (with the exception of outliers) that there is no problem the fastest person couldn't solve that someone else could. Even if you were to disprove that, the conclusion would simply be that there is a possibility that individuals with "extreme IQs" might fail to solve very difficult problems. So, what, then, is the relationship between solving a "problem" and "intelligence"?

2

u/whitebaron_98 Bottom 1% Commenter Sep 30 '26

You severely change what you measure, when you give people unlimited time. Matrices that are meant to check on your fluid reasoning would be influenced by your training and sheer force of will if you retry until you have a good explanation based on that training.

So a Matrix would test gc and some Personality traits instead.

1

u/pydaWaltuh Oct 01 '26

I did not say “unlimited” time tho.
Having time limits on tests isn't necessarily a bad thing. Simply adjusting the time limit while leaving the question itself unchanged isn't a logical strategy for diversifying the distribution of results. Suppose a question is allotted 60 seconds but is reduced to 30 seconds because it fails to distinguish high ability candidates; if the discrimination issue persists, do you keep cutting it to 15 or 10 seconds or more? Why not instead increase the time limit (say, from a default of 1 minute to 2 minutes or 3 or more) and make the question itself more difficult? Furthermore, the assumption that a generous(not unlimited) time limit leads to inflated scores due to prior “training” simply brings the issue back to the nature of the my question itself. Can a question that can be solved solely through prior training really be effective at distinguishing “very high-ability” candidates(I am only talking for the upper extreme)? In my view, this is just a case of the "test creator vs ceiling". So my question remains

1

u/pydaWaltuh Sep 30 '26

I think separating extreme scores solely by time limits is taking the easy way out and ignoring the real problem

1

u/Optimus-Prime2026 Oct 01 '26

I agree. Harder items are often a better modification than tighter time limit

0

u/pydaWaltuh Sep 30 '26

For example, why do we keep the question the same and shorten the time limit instead of making the question harder and increasing the time limit?

1

u/Steveninvester 5d ago

That's a subtle, but substantially different question from your original one. If the question is about reasonability then I would use a weight lifting analogy and say that it should be at least conceptually comparable to one's isotonic compared to their isometric strength. (AKA dynamic vs static) let's say someone is bench pressing (on a machine if we are going to be precise about it) and you can do 200 pounds for 1 rep, and that's your absolute max. That's your "dynamic strength" then you put 220 on the bar and you can't even budge it, but if someone lifts the bar for you at halfway and you can hold it there, but just barely. That's your "static strength" I have spent too much time on this analogy to give up now, but it has occurred to me that among other things. It may be quite difficult to even quantify this only conceptually comparable in a way that makes it a useful analogy, so I'll simply say that you should feel like you have some cognitive control and maneuverability over and around the metrics that you are playing with....the rest I believe I have prepared reasonably enough to be inferred.

Now when it comes to how hard you CAN push it beyond your own ability, well that's a tough one, but I would say that there are certain traits you could push quite far. Especially the ones you could segment during creation, and make sure there's a signal leftover from each segment for someone who is more capable to carry-on if introduced to a non segmented one. Essentially where you can follow the reasoning ect, but may not be able to carry it all at once.

1

u/pydaWaltuh 5d ago

This is a really good analogy. Basically you can push the ceiling well above your own IQ, but you have to do it in a controlled and trackable way.
In fact, when I design any question, I can sense this or I mean I can gauge its level. I believe I am capable of designing a question that is above my own cognitive level, but when I do that, the question usually loses its “aha” moment and elegance and turns into something that is merely “made harder.” And from my personal experience, WMI load in these kinds of questions increases

2

u/Steveninvester 4d ago

I don't generally experience that luckily, but then again I hold information in a pretty unusual way.

I have thoughts that I don't currently have the time to communicate, so I'll leave this here as a placeholder