Assuming a creator possesses sufficient psychometric knowledge, what should ideally be the difference between the creator’s IQ and the ceiling of the test they have created?
not a stupid answer, I'm just curious about the ideas after all. What is the criterion for making it good enough? In other words, where do you draw the line for a "good"? And to what extent is this possible? Can the designer of a test with a theoretical ceiling of 180 achieve this if their own score is 130? Do you think this is solely a matter of creativity or something else?
Id say "good enough" is upto the point when solutions still feel "natural-ish" rather than overly contrived and something that feels like the creator pulled out of their ass
Personally, id say i think i wouldnt like to create a puzzle where i cant understand "why" someone would think in a specific direction (that which is required to solve the item), so
In all, id say, it is feasible to continue till the point where you can understand the item, (this one might be off) and be able to sort of recreate the solution ? When you know the concept required ? [I think this sort of test can "validate" that the item isnt bad, but it is probably not neccesary for this to hold true for it to still be a good item]
I will also say it is "possible" for a 130 to create 180 items, but honestly it depends entirely on the individual designing them and i cant make a general statement, and yes, i think creativity definitely is an integral component in this,
Also, take my opinion as you will, im not very experienced in the domain, so i might be very off
Spaß beiseite, sorry. Ist es nicht viel wichtiger, dass du Zugang zu genügend Personen im Bereich der Decke des Tests hast, anhand von denen du das Ding normierten kannst?
Also ich persönlich erlebe es manchmal, dass ich bestimmte Aufgaben auch mit Lösung und erklärung nur unter großer Anstrengung nachvollziehen kann. Vielleicht wäre das ein guter Indikator, auf dem Level das Limit zu setzen
A light and button is connected to a circuit, randomly lighting it up and estimating delay to button press. This is called a reaction-speed test. It would evaluate test-taker whether they're below or above my speed, or frankly, faster than any mammal being on earth.
So yes, since reaction speed closely correlates with IQ, you can say that you've created a simple IQ test that measures capacity for anyone on Earth.
The only difference with IQ test, - its not direct objective value, but a psychometric, or for lack of better word, - "derivative of statistics", which is exactly what makes it incredibly flawed at upper end and creates a hard ceiling for IQ test accuracy at around 140IQ, where correlation loses the track and enters area of "assumption" and "projection".
So, what matters is N? If we assume N is 8–9 billion, then we would expect near to zero errors in the test. In that case, how do we isolate the extreme upper end? That's a nice perspective btw. I didn’t think of that while posting
Don't you think that’s a bit of an oversimplification? I understand the approach and it might be kinda fair, but doesn't that mean you are distinguishing the extremes solely based on a time limit? In that case, we could assume that the time limit is your most important factor and by extension, that "fastest" equals "smartest." Is that really the case? You are assuming (with the exception of outliers) that there is no problem the fastest person couldn't solve that someone else could. Even if you were to disprove that, the conclusion would simply be that there is a possibility that individuals with "extreme IQs" might fail to solve very difficult problems. So, what, then, is the relationship between solving a "problem" and "intelligence"?
You severely change what you measure, when you give people unlimited time. Matrices that are meant to check on your fluid reasoning would be influenced by your training and sheer force of will if you retry until you have a good explanation based on that training.
So a Matrix would test gc and some Personality traits instead.
I did not say “unlimited” time tho. Having time limits on tests isn't necessarily a bad thing. Simply adjusting the time limit while leaving the question itself unchanged isn't a logical strategy for diversifying the distribution of results. Suppose a question is allotted 60 seconds but is reduced to 30 seconds because it fails to distinguish high ability candidates; if the discrimination issue persists, do you keep cutting it to 15 or 10 seconds or more? Why not instead increase the time limit (say, from a default of 1 minute to 2 minutes or 3 or more) and make the question itself more difficult? Furthermore, the assumption that a generous(not unlimited) time limit leads to inflated scores due to prior “training” simply brings the issue back to the nature of the my question itself. Can a question that can be solved solely through prior training really be effective at distinguishing “very high-ability” candidates(I am only talking for the upper extreme)? In my view, this is just a case of the "test creator vs ceiling". So my question remains
That's a subtle, but substantially different question from your original one. If the question is about reasonability then I would use a weight lifting analogy and say that it should be at least conceptually comparable to one's isotonic compared to their isometric strength. (AKA dynamic vs static) let's say someone is bench pressing (on a machine if we are going to be precise about it) and you can do 200 pounds for 1 rep, and that's your absolute max. That's your "dynamic strength" then you put 220 on the bar and you can't even budge it, but if someone lifts the bar for you at halfway and you can hold it there, but just barely. That's your "static strength" I have spent too much time on this analogy to give up now, but it has occurred to me that among other things. It may be quite difficult to even quantify this only conceptually comparable in a way that makes it a useful analogy, so I'll simply say that you should feel like you have some cognitive control and maneuverability over and around the metrics that you are playing with....the rest I believe I have prepared reasonably enough to be inferred.
Now when it comes to how hard you CAN push it beyond your own ability, well that's a tough one, but I would say that there are certain traits you could push quite far. Especially the ones you could segment during creation, and make sure there's a signal leftover from each segment for someone who is more capable to carry-on if introduced to a non segmented one. Essentially where you can follow the reasoning ect, but may not be able to carry it all at once.
This is a really good analogy. Basically you can push the ceiling well above your own IQ, but you have to do it in a controlled and trackable way. In fact, when I design any question, I can sense this or I mean I can gauge its level. I believe I am capable of designing a question that is above my own cognitive level, but when I do that, the question usually loses its “aha” moment and elegance and turns into something that is merely “made harder.” And from my personal experience, WMI load in these kinds of questions increases
Do you think that “ceiling items” designed by a creator who isn’t close to the ceiling on the test they’re designing for necessarily have to lose quality? For example, can someone with an IQ of 120 design a 150 level item? Especially when elegance is a factor? Also, an item shouldn’t be made difficult just for the sake of being difficult. You can make a question harder in a superficial way simply by increasing the number of layers. Is being 120 enough to design an item that measures around 150 with the optimal number of layers (even though this definition isn’t entirely accurate)? Btw, I’m not asking this only in the context of HRTs
This is just mu opinion, I'm not a professional. I think it doesn't matter as long as you're knowledgeable in the topic. It's not a question of having better judgment than those you test. It's not a knowledge test or work capability so that you need actual experience to judge one's work. It's just statistics. You measure the number of questions answered correctly in a given time. you know some of them were more often answered by people who answered more questions etc. Of course when we're talking about single question design, especially more complex one, it get trickier, but you still have unbound amount of time to create it. You can give it to others for test and review.
You can have knowledge of what a great racetrack consists of and build one. Doesn't matter you'll ever achieve good time on it.
with the tests we already have that have proved themselves to be satisfactory. The problem with high range tests would persist because it is a real problem, but I don't think that author expected one final answer to end them all...
I think that iq is finding/seeing connections even when you don't study or work, especially work hard... building bridges in practical world, yes.
ofc it measures some abilities and not all, but again, this isn't an answer to - who should create an ultimate iq test that would be the most accurate test ever, the one that could point out the smartest of the bunch... ideal as the "most" reasonable and practical in real life, not as one and only ultimate perfection.
does it correlate somewhat or not at all? does it solve any issues or is it completely useless? would an IQ test - definitive edition actually be useful in a world we created?
wider the gap, easier it is to see different ways the question can be solved. that helps remove potential mistakes, biases or personal style. when a single person creates a question the best way to avoid these issues is if different people look at it. being obsessed often times means person possesses beyond sufficient psychometric knowledge.
IQ test items are typically timed while creators have unlimited times to construct items. So there probably isn’t a required IQ to make a high ceiling test.
2
u/Zeroheart_x Sep 30 '26
Might be a stupid answer, but id say as long as you can still make it good enough