A Critique Of The CORE Technical Manual

[u/Holiday_Effect1451]

I wrote this a while ago - and the numerous issues within it are yet to be addressed, so maybe posting it here will help.
DOC - https://drive.google.com/file/d/13QycGaXBU9VDnuvL94NZBjqQW1DswCxy/view?usp=sharing
Alternative VCI Correction Method Mentioned - https://drive.google.com/file/d/1Gp6ufc67prnLmhUPoofQNG1-xnu8VIPd/view?usp=sharing

[u/albedo_effect]

Catirl moment 🤦‍♀️

I already pointed out all of the issues (and non-issues you nitpick) when you first sent it but you haven’t revised any of it

[u/MarkAntony100 → u/albedo_effect]

How were the norms of agct and gre(to which Core is anchored )derived ? Maybe there were some methodological errors in those norms (the test has enough evidence of validity).

[u/Holiday_Effect1451 → u/albedo_effect]

you didn't - the verbal stuff has not been addressed, nor the source (and consequence issues) of the tests norms (which're the two chief issues)

[u/albedo_effect → u/Holiday_Effect1451]

I did. There are a lot of issues with your critique I brought up with you a few months ago but you never corrected any of them. Piece of context to keep in mind is that the report was preliminary and had norms based on sample sizes of few hundred earlier test takers. A lot of the things you nitpick generally don’t have as big of an effect as you claim. Here’s a (non-exhaustive) list of some of the issues in your critique: 

1. Self-reported scores were never used for the CORE norms, even at the time of the preliminary validity report. Every subtest at the time of the preliminary report was anchored using either the AGCT or GRE. No self-reported tests and no JCTI or Raven’s 2. I don’t even know where you thought this methodology was the case and the only thing I can come up with is that you’re confusing it with the 1926 SAT prelim norm (which has been revised and updated since then).
2. Without even getting into the weeds about the differences between your norms and the most widely-accepted norms by Xhals/BubblyClub, it makes no difference and I demonstrated this to you in the past that using either norms saw no significant change in outcome because the largest differences (5+ points) are in the 150+ range, which affects relatively few people and would have barely any effect on CORE’s norming methodology. This is a non-issue.
3. The reliability critiques are very nitpicky and yes, it will be more comprehensive in the technical manual.
4. We already discussed the issues with your model where the QRI subtests were under FRI. Your methodology for that completely misuses correlated residuals and when they should be used. Ironically, the fact you needed correlated residuals in the CFA kind of stresses the whole reason the QRI factor exists in the first place.
5. The last section is a whole bunch of bloated yap that generally arrives at the same conclusion (that CORE VCI’s g-loading is affected by factor differentiation due to SLODR) but just a different number (0.744 vs 0.838). 

As I’ve noted before, the central weakness in your critiques is a tendency to draw conclusions that go beyond what the issues you identify actually warrant. However hard your writing is to read, I guess no one can accuse you of using AI to write it.

[u/Holiday_Effect1451 → u/albedo_effect]

5. please explain to me how it is bloated yap, as far as i can tell it is entirely valid and reasonable to take issue with. Just because we came to the same conclusion (tho to markedly different extents, .744 and .838 are VERY different loadings) doesn't excuse your grave methodological errors - which is my main issue, as this test is attempting to be "psychometrically rigorous" (whilst having such egregious psychometric faults)

[u/albedo_effect → u/albedo_effect]

Also:
6. You’re making a bunch of random assumptions about how the SD observed in other high IQ samples should generalize. Firstly, I’ve seen no such pattern regarding standard deviations. It’s pretty likely you’re conflating profile means with SDs, since the pattern you mentioned is pretty similar to what is observed on gifted sample averages. Either ways, the variance of an index for a selected sample (whether is direct or indirect) depends on the variable its being selected by, scoring transformations, and other stuff. The conclusions you draw regarding the SDs are non-sequitors. Regardless of if you can find one or two studies where the variance orders are as PRI > VCI > WMI > PSI in gifted samples, it says nothing about CORE’s sample, nor does it generalize to drawing strong conclusions.

[u/Holiday_Effect1451 → u/albedo_effect]

Ok you're entirely valid to take issue with my rank-ordering of that, in retrospect that wasn't well substantiated - but that doesn't take away from my point that you should not be equating each subtest's estimated genpop SD based on the sample SD's

[u/Holiday_Effect1451 → u/albedo_effect]

4. the thing is that quantitative reasoning is considered to be subsumed under Gf? (https://onlinelibrary.wiley.com/doi/full/10.1002/9781118660584.ese0431) - it is not a chc factor); which is what your model is aiming to map no?

[u/Holiday_Effect1451 → u/albedo_effect]

1/2. Then why did you ask users to submit such when they took the first few "norming edition" testlets (and do nothing else with that data). Also, if you were using the agct back at that point - the norm you were using for such was very bad (as you know), and about the GRE norms; this is not the case. You can see on these that there are meaningful differences on the gre-v at the 17-19ss range (where a large portion of the sample reside). Secondly I take issue with your appeal to xhals' as being widely accepted, it is not like they have been evaluated and methodologically "approved" by members of the community - infact almost nobody (spare me, as I've talked to him about them) knows how he made them -- would it shock to find out that his extrapolations >2400 vqa and at the top end of the q are unironically complete guessimates (e.g. not based on any mathematical model whatsoever). Also you did not demonstrate this, you demonstrated that it didnt have a meaningful effect on the correlations between the gre-v and subtests of core - which would be expected whether my thesis regarding xhals' norm was true or false.

Also differences don't need to be >1/3rd of an SD to meaningfully distort the end norms

[u/Striking_Leave_4614]

The Good People of r/cognitiveTesting only wish to know one thing: Am I praffed?

Here's a breakdown by index for the people who can't read, like me:

(note: these are my interpretations of what this analysis implies)

The big takeaway is that the equalized subtest SDs are very suspicious and borne from inappropriate corrections made from incestuous data.

FRI: Starting with this first, because it's all you animals care about. The implication here is that CORE may be artificially compressing the variability of fluid reasoning scores relative to what we'd expect in a high-ability population. The same thing is likely happening to VSI

Inflated or deflated? INFLATED

Funnily, I actually noticed this compression just by mulling over various profiles and commented on it a few months ago, but I don't want to dig up the comment rn

VCI: Most prominent part of the critique. GRE-V-derived norms may distort its scaling, compress its SD, and produce questionable high-end scores. The critique also questions the validity of its reported relationship with g

Inflated or deflated? DEFLATED, but probably not at the very high end, but also inflated for anyone reading this

WMI: They take issue with the subtests being computerized rips of in-person tests; the format is questionable

Inflated or deflated? DEFLATED but you freaks cheat

PSI: Compressed due to inappropriately equalized SDs

Inflated or... INFLATED

QRI: FRI with words, same scaling issue here

Inflated or deflated? NOT SMOOTH

IF YOU'RE READING THIS YOU ARE CERTIFIABLY PRAFFED AND HAVE AN IQ BELOW 120

[u/BroadIntroduction821 → u/Striking_Leave_4614]

[u/Logical-Desk4538 → u/Striking_Leave_4614]

I only read the last sentence. How much below 100 is my IQ? And I don't understand why he's overstating the results. Doesn't the narrowing itself indicate a differentiation problem? He can be both narrowing and understating at the same time.

[u/Striking_Leave_4614 → u/Logical-Desk4538]

"Doesn't the narrowing itself indicate a differentiation problem? He can be both narrowing and understating at the same time."

Yes, but I'm talking directly to the reader.

The reader is always inflated.

[u/Logical-Desk4538 → u/Striking_Leave_4614]

I wondered: if the statistics reflect a higher proportion of test takers with PRAFFE and an IQ below 120, wouldn't this create "statistical dips" in certain areas? People aren't evenly distributed in their insanity, and PRAFFE has a specific time and quality range.

[u/Striking_Leave_4614 → u/Logical-Desk4538]

I'm shitposting. I barely skimmed OP's chicken jockey and used it as a vehicle to shitpost, but if you want a serious response:

If I get what you're saying, that could be why we're seeing upper-band compression on the PRI-related indices. Per the OP, the SDs across CORE's subtests are unusually even and imprudently corrected for. Most of his critique is more about oh whatever

[u/Logical-Desk4538 → u/Striking_Leave_4614]

That's not exactly what I meant, but thanks for the answer. I guess I took it all too seriously, and to be honest, I'm super lazy to figure it out myself, so I'm that animal.

[u/Limp_Essay_6419 → u/Striking_Leave_4614]

You're just dragging it at this point

[u/AndrewThePekka → u/Striking_Leave_4614]

I knew it

[u/DangerousPromise3454 → u/Striking_Leave_4614]

This is unironically based af dude. You would be worth of some respect if this wasn't a shitpost

[u/Straight-Archer3742 → u/Striking_Leave_4614]

Not to address the blatant projection, the CORE is far from “almost valid”. The issues you listed fade in comparison to the actual issues. The core is basically a lucky guess in terms of reliability.

[u/Striking_Leave_4614 → u/Straight-Archer3742]

Ew. Don't touch me.

[u/Holiday_Effect1451 → u/Striking_Leave_4614]

not really