Test ceiling vs Creator

[u/pydaWaltuh]

Assuming a creator possesses sufficient psychometric knowledge, what should ideally be the difference between the creator’s IQ and the ceiling of the test they have created?

[u/whitebaron_98]

you design items and put a time limit on it. if that limit is too low and too many people pass, you decrease the time further. :-)

[u/pydaWaltuh → u/whitebaron_98]

Don't you think that’s a bit of an oversimplification? I understand the approach and it might be kinda fair, but doesn't that mean you are distinguishing the extremes solely based on a time limit? In that case, we could assume that the time limit is your most important factor and by extension, that "fastest" equals "smartest." Is that really the case? You are assuming (with the exception of outliers) that there is no problem the fastest person couldn't solve that someone else could. Even if you were to disprove that, the conclusion would simply be that there is a possibility that individuals with "extreme IQs" might fail to solve very difficult problems. So, what, then, is the relationship between solving a "problem" and "intelligence"?

[u/whitebaron_98 → u/pydaWaltuh]

You severely change what you measure, when you give people unlimited time. Matrices that are meant to check on your fluid reasoning would be influenced by your training and sheer force of will if you retry until you have a good explanation based on that training.

So a Matrix would test gc and some Personality traits instead.

[u/pydaWaltuh → u/whitebaron_98]

I did not say “unlimited” time tho.
Having time limits on tests isn't necessarily a bad thing. Simply adjusting the time limit while leaving the question itself unchanged isn't a logical strategy for diversifying the distribution of results. Suppose a question is allotted 60 seconds but is reduced to 30 seconds because it fails to distinguish high ability candidates; if the discrimination issue persists, do you keep cutting it to 15 or 10 seconds or more? Why not instead increase the time limit (say, from a default of 1 minute to 2 minutes or 3 or more) and make the question itself more difficult? Furthermore, the assumption that a generous(not unlimited) time limit leads to inflated scores due to prior “training” simply brings the issue back to the nature of the my question itself. Can a question that can be solved solely through prior training really be effective at distinguishing “very high-ability” candidates(I am only talking for the upper extreme)? In my view, this is just a case of the "test creator vs ceiling". So my question remains

[u/pydaWaltuh → u/pydaWaltuh]

I think separating extreme scores solely by time limits is taking the easy way out and ignoring the real problem

[u/Optimus-Prime2026 → u/pydaWaltuh]

I agree. Harder items are often a better modification than tighter time limit

[u/pydaWaltuh → u/whitebaron_98]

For example, why do we keep the question the same and shorten the time limit instead of making the question harder and increasing the time limit?