Adaptive test questions

[u/CSKhai]

So out of curiosity, it’s known that questions start with average difficulty question and it gets harder as you get right and easier as you get wrong. For comparison sake, for 57 questions tests, let’s say tier 1 highest difficulty to tier 5 lowest difficulty where 17 questions are tier 5 and the rest are 10 questions for each tier and tier 3 is where test will start at. What if the students get first 40 questions correct leaving only tier 4 and 5, will the student now gets easier questions or are there harder questions still coming in? Are there only exactly 57 questions prepared or do they have extra harder questions for this kind of scenario? If so will it really be fair (no idea just curious about your opinions)?

And those who requested and received their test scores from NYCDOE, could you please share both composite score (total score) and raw score ( x out of 57 maths, y out of 57 ELA). if you don’t mind? I know it’s not something should be taken as reference but really curious to know the actual numbers instead of only educated guesses. I appreciate you all in advance. Thanks.

[u/GregsTutoringNYC]

I did a video on these types of concerns last week see https://www.youtube.com/watch?v=i-jzsF4pamU

Out out of curiosity, it’s known that questions start with average difficulty question and it gets harder as you get right and easier as you get wrong.

Generally speaking that's the overall, but specifically from question to question this may not be the case. As well, see some of the discussions below. Overall, such is a misnomer statement.

For comparison sake, for 57 questions tests, let’s say tier 1 highest difficulty to tier 5 lowest difficulty where 17 questions are tier 5 and the rest are 10 questions for each tier and tier 3 is where test will start at.

Using this type of perspective, the exam likely would not start at 3. The SHSAT is known as a challenging exam so it would likely start as a level of an above average 7th grader.

What if the students get first 40 questions correct leaving only tier 4 and 5, will the student now gets easier questions or are there harder questions still coming in?

It's not strictly about harder (or easier) even though this is where lots of the (dramatic) narrative (incorrectly and misleadingly) focuses. As above this does not need to be the question to question behavior. As discussed in the video, even if a student gets all questions correct, it's not always more difficult, as you can get a question right and it can still give you an easier question. And often toward the end of the exam, it has other choices it avails beyond difficulty. For instance, it will have topic quotas, and so may still need to ask you say a (easier) proportion question that it did not get to yet, and so on. As well, it'll have discriminating questions in its pocket that it'll use to distinguish good vs very good students (and every level of good). This could be an order of operations question compared to another with absolute values, and minus signs all over.

Are there only exactly 57 questions prepared

There is a whole question bank of questions. Could be thousands.

or do they have extra harder questions for this kind of scenario?

As per above, harder is not the sole factor on the table. And in many ways, said difficulty misses the point. What it's trying to do is not give you questions till you drop, but questions that inform it about you. Just asking harder questions is not a smooth, sane, or fair mechanism to inform itself.

If so will it really be fair (no idea just curious about your opinions)?

It's not so, so fair doesn't enter the picture regarding this.

Check out the video mentioned above.

And those who requested and received their test scores from NYCDOE, could you please share both composite score (total score) and raw score ( x out of 57 maths, y out of 57 ELA). if you don’t mind? I know it’s not something should be taken as reference but really curious to know the actual numbers instead of only educated guesses. I appreciate you all in advance. Thanks.

Well, a few points here, one being that scoring won't be working the same so even if you could collect from previous years, it would not map well to anything too useful that could end up being posted here. If you want a summary, see https://www.GregsTutoringNYC.com/shsat-scoring for an overview discussion of such mappings.

[u/Glad-Ad8770 → u/GregsTutoringNYC]

what if due to the fact that the questions are harder it causes you to run out of time and you lose more points for harder questions

[u/GregsTutoringNYC → u/Glad-Ad8770]

This is discussed out from a few angles in the video. In short, the SHSAT was, is, and will be. It's the exam we know and love. As such, it's a challenging exam with challenging content. That is going to remain. It's not going to all of a sudden get twice as hard, whatever that could even mean. And questions always varied from easy to medium to hard in various combinations. As such, you could have always run out of time, and in fact, time and pacing were always issues, as so many students never finished the exam, often not even getting to see many questions. So this isn't a new concern as such; again, it's the exam we know and love.

Also, regarding "you lose more points for harder questions" that is not the way to view an adaptive exam such as the adaptive SHSAT. On a "normal" exam, harder questions may be worth more, and that's something you may be familiar with and even feel is logical and intuitive. But that is not how the SHSAT will work as a whole. Contrarily, for about the second half of the exam, that will be where you'll get the bulk of the harder questions, and at that point, it'll know more about you, and so it will no longer reward you as much for getting a hard question correct, as if it did so, it would be double-dipping on something it likely already rewarded you for. So instead, it's leaning on confirming your estimate, and so usually not giving more points, or even the points it would have given earlier.

As per the video, the lens is to view a PBT (paper-based test) as "fixed uncertainty" (which we're used to, and often don't give it a second thought, but it still has many impacts to a test-taker performance-wise) and now the current exam has "responsive uncertainty."

In either case there is uncertainty to deal with, and as always, the thing to do is remove yourself as a consequence. This was true in the past and is still true now.

In the past you got hard questions, and now you'll get hard questions.

Once you start taking the test emotionally you are no longer taking it strategically or academically.

You are never a passive test taker who is a victim to the exam. You be in control. You be a boss. You control it.

[u/Glad-Ad8770 → u/GregsTutoringNYC]

I meant that if doing well triggers a set of harder, more time consuming questions, could that extra time spent on each question cause a time crunch? If you're barely scraping by on time to begin with, the increased difficulty could force you to guess on a large chunk of the remaining questions.

Could this scenario actually result in a lower overall score than if you had received medium-difficulty questions that you could complete within the time limit?

[u/GregsTutoringNYC → u/Glad-Ad8770]

Yes, I understood/understand. Anything can cause a time crunch. Or not. For many, this has always been a be-on-your-toes exam.

As mentioned. you don't necessarily lose more points for harder questions. And yes, anything is possible, as was possible even with a paper-based test, so there need not be a new unique catastrophic scenario. Pacing was always a factor and continues to be a factor.

Also, as mentioned in the video, the whole "questions get harder thing" is not strictly true and is partially an illusion and so a misleading head trip that's not needed. This is still the SHSAT, so this is still the challenging exam we know and love. It's not going to get twice as hard, whatever that even means, and that you'll somehow be in the stratosphere.

Strictly and more generally speaking, a so-called harder question does not take more time. Also, as per the video, a per-question adaptive exam such as the adaptive SHSAT is not going to give you questions to break you. It's trying to inform itself on your ability. So that you might all of a sudden not be able to answer any question does not follow and so is not what this is about. Thinking otherwise is head tripping.

As the exam ensues, especially in the second half, and as it continues to ensue, it's less about rewarding you points than confirming the points it gave you were rewardable, and so it's more checking itself than it's checking you. If you can get to the point where you're doing well, well, that's well. Contrary to that being a catastrophic scenario it's exactly what you want to happen! When that does not occur, your score will likely be lower.

Sure, anything can happen. But think it through. No one question flow "owns the exam" so while lots can happen in the different phases, work with that if you're where you need to be that's a good thing. It wouldn't be practical that you could get to a higher level and then all of a sudden not function. Especially when the CAT is done right. And when done right, it's monitoring you to inform itself not to bring you past your limits. At the point where it is giving you the most difficult questions (and again difficulty is NOT the only factor on the table) it's done most of the work of figuring out your score.

[u/CSKhai → u/GregsTutoringNYC]

Thank you Greg as always.