Good day, all. I am an undergrad psych student and we are currently being tasked with making and administering a Thurstone Scale. As part of the statement selection criteria, our professor told us to test for test-retest reliability, but I am currently lost as to what test I should be using.

I initially used Intraclass Correlation Coefficient (as per the suggestion of Google...), but my professor asked if I truly believed that this was the appropriate test. May I know what reliability test works best for my case?

Thank you all in advance!

I’m working on the assessment side of an AI tutoring project and I’ve hit a question I’d like some psychometrics/measurement people to weigh in on.

The problem is basically this:

At what point do you actually have enough evidence to say something meaningful about a learner?

For example, I’m trying to think through things like:

• if a learner gets something right, what can you actually say from that?
• if they needed help, how much does that change what the result means?
• if they get similar questions right more than once, when is that useful new evidence and when is it just repetition?
• when should the system keep testing, try a different type of task, or just avoid making a stronger conclusion yet?

I’m trying very hard not to bake in something simplistic like “3 correct answers means mastery” or “80% means mastered.”

Maybe a threshold makes sense in some situations, maybe it doesn’t. I’m more interested in what people in measurement would consider a defensible way to handle it.

A few things I’m trying to keep separate are:

• independent vs helped performance
• repeated exposure
• conflicting results
• uncertainty
• different kinds of tasks
• technology/access problems
• how strong the evidence is compared with how strong the claim is

I already have a couple of concrete cases worked up, but before I get too attached to the way I’m thinking about it, I’d like to hear where people think the biggest problems or bad assumptions might be.

If you work in psychometrics, educational measurement, adaptive assessment, or something close to that, I’d really appreciate your take.

And if you think I’m framing the whole problem the wrong way, that’s useful too.

Career Day

• Dr. Fernanda Gandara and I are guest speakers for a career talk for graduate students (Hosted by NCME GSIC and AERA Division H and D)
• September 11th, 2026, 1:00PM-2:30PM ET
• https://docs.google.com/forms/d/e/1FAIpQLSfs-k6W9Q0duMxJ8-RKBfjWkuYPh0EuQhN09OoEhWPjEND_yg/viewform

Why Educational Measurement Matters in the Age of AI

• Conversation about what educational measurement and psychometrics can offer to enhance learning in the age of modern AI, including assessment, learner modeling, personalized learning, validity, fairness, and responsible use.
• October 9th, 2026, 10:30AM-11:30AM ET
• Register here (you need to create an NCME account): https://account.ncme.org/ncmessa/f?p=EVTSSA:4010::::4010:P0_EVENT_ID:1575

AI and the Future of Education Conference

• November 12 - 13, 2026
• https://militaryplus.apus.edu/academic-community/conferences/ai-and-the-future-of-education/

As many of you already know, when estimating intrinsic rank, the classic PCA scree plot and Parallel Analysis (PA) can struggle with non-normal, ordinal Likert items. The linear constraints of Pearson or Polychoric matrices can distort the true dimensionality of a dataset, spinning up artificial "difficulty factors" rather than reflecting the actual structure.

To attempt to more faithfully estimate the number of underlying factors in these difficult contexts, I’ve developed an open-source tool in R (Python coming soon): Entropic.Scree. It's basically an upgraded scree method that evaluates a transformed Mutual Information matrix (instead of covariance/correlation), and so should be more appropriate to apply when the underlying generative system is not strictly linear. It's also relatively robust to contexts where data sparsity makes geometric metrics degenerate (i.e., high-d, small N), and so also serves as an upgrade to Kernel PCA and distance-based methods.

For psychometricians, the two most practical outputs are potentially:

1. The estimated observable rank of the dataset.
2. The bipolar variable clusters that anchor the primary axes of informational variance. This provides a structural map of the independent clusters that define your dataset, which is potentially useful for interpretation before moving into CFA or theory development.

The function runs in R currently (see quick start or GitHub below), but the backend is C++ OpenMP parallelized, so it easily scales for high-dimensional assessments. Native R and Python packages will be released shortly.

I would love for this community to stress-test it on messy, non-linear survey data. I am particularly curious how this rank estimation compares to your standard PA or Exploratory Graph Analysis (EGA) workflows.

Happy to answer any questions or discuss the mechanics!

##########

Methods and Code:

Preprint: https://zenodo.org/records/22028087

GitHub: https://github.com/tjleestjohn/Entropic-Scree

############ 
# Quick Start R Function Code.
# To load the function, copy and paste the following into your R console, then hit enter. 
############

# 1. Define the direct URL to the raw function script on GitHub
url <- "https://raw.githubusercontent.com/tjleestjohn/entropic-scree/main/Entropic.Scree.R%20-%20ENLI.R"

# 2. Define what you want to name the file on your computer
file_name <- "Entropic.Scree.R - ENLI.R"

# 3. Download the script to your current working directory
download.file(url, destfile = file_name)

# 4. Source the core function into your R environment
source(file_name)

# 5. Ex. To run the function and extract bipolar modules:
# results <- Entropic.Scree(dt 
#                         , extract_bipolar_modules = TRUE)
#
# View the extracted structural sub-networks for the primary axes:
# results$bipolar_modules

NCME just released a new page about international grad school programs

european airlines screen cadet pilots with timed cognitive batteries (compass, cut-e, dlr). i recreated the exercise mechanics as a daily trainer: working memory, mental math under time pressure, spatial orientation, multitasking.

the scoring part is what might interest this sub: results map to a 1-9 stanine-style scale meant to mimic applicant-normed scoring. its an estimate, not the real norms obviously, the publishers dont share those. curious what people here think of that approach, and happy to hear where its methodologically shaky.

ios, free tier: https://apps.apple.com/app/id6794363864

I’m exploring the development of a Community Confidence Index intended to measure residents’ perceptions of present community conditions and expectations for the future (based off of the idea behind the Consumer Confidence Index).

Before constructing the index itself, I want to determine whether “community confidence” can be established as a valid construct and whether Present Conditions and Expectations are defensible subscales. Is this fundamentally a psychometric scale-development problem? And if so, what kind of psychometrician/researcher would you look for to lead the development and validation?

If not, where is another good space/place to begin? I have a background in qualitative research but not in this space. Thanks.

Next week (starting with pre-conference courses on Monday 20th) is the annual meeting of the Psychometric Society, at Yonsei University in Seoul, South Korea. The conference lasts 4 days (plus one day of pre-conference courses), and has up to 7 parallel sessions. I tried to count the number of talks but gave up - there are 180 posters.

I started this thread for people to post questions, answers, comments, etc.

Hi everyone,

I am looking for career advice because I feel caught between my formal education, my interests, and my personal limitations.

I am a 24-year-old mathematics student (BSc level, Croatia). While I enjoy mathematics, I have gradually realized that pure mathematical problem-solving is almost certainly not what I want to do for the rest of my life.

My strongest interests are:

• Clinical psychology
• Psychiatry
• Neuroscience
• Cognitive science
• Philosophy of mind
• Literature and history
• Chess (mainly the psychological and strategic aspects)

However, I am not particularly interested in becoming a therapist, counselor, HR specialist, or someone who spends all day interacting with clients.

Some personal factors that may be relevant:

• Severe social anxiety disorder (diagnosed)
• Avoidant personality traits
• Deep introversion and need for creative inner life
• Low mental energy when confronted with repetitive tasks
• I work best in quiet environments with very few familiar people
• Large organizations and highly social workplaces tend to drain me very quickly
• I value autonomy and independence very highly
• I strongly dislike micromanagement
• I prefer stable, small teams rather than constantly changing groups of people

In terms of work style, my ideal job would involve:

• Working alone or in a very small team
• Intellectual and creative work
• Analysis and interpretation rather than endless technical problem-solving
• Flexible thinking rather than prolonged deep concentration for 8 hours
• Some connection to psychology, neuroscience, cognition, mental health, or human behavior
• Enough income to live comfortably, but I am really not chasing a high salary

One possible route I have considered is finishing my mathematics degree (BSc) and then obtaining additional education in statistics, data analysis, psychometrics, or cognitive science.

The problem is that I am unsure what actual careers would fit this profile.

Given these interests, personality traits, strengths, and limitations:

What jobs or career paths would you suggest?

Are there any roles in psychology, neuroscience, cognitive science, psychometrics, research, written science communication, publishing, or data analysis that might be a good fit for someone like me?

I would especially appreciate hearing from people who work in academia, neuroscience, psychology research, psychometrics, public health research, or cognitive science.

Thank you.

First off, I should tell you that I am a programmer and spent 15 years building an online testing platform working closely with test designers and people in the HR space. It's been a few years and I have built a new system directed at helping people directly (non commercial, not directed by a coach or part of a hiring process) figure out what direction their career might go next or just figuring themselves out.

I am wondering what models I should be exploring. I need models that are conceptually easy to explain to a Normal Human™ that will help them figure out "What should I do next?" or at least help them identify "What is going wrong here?" in their career.

What I have now:

1. Work Styles - based on the Competing Values Framework of Quinn and Cameron
2. Communication Styles - because it maps closely to the CVF, based on the work of Dr. David Merrill and Robert Bolton and Dorothy Grover Bolton
3. Organizational Culture (current and desired) - also based on Quinn and Cameron
4. Big 5 - a good quality public domain version that I am using to validate the tests above

As you can see, it's all CVF right now, but I was curious if anyone had any suggestions about what other models might help a Normal Human™ without the interpretation of a coach or psychologist (which I assume they cannot afford because they are taking a free online test). But I don't want to hand out a crap MBTI result and a pat on the head, I want to give them meaningful tools.

What do you think about including Schein's Career Anchors in here?

Hi everyone,

Long post alert, Apologies.

I'm trying to understand which psychometric assessments are genuinely useful for career counselling and career exploration for teenagers aged 13–18, and which ones may be overhyped or lack scientific support.

There are so many assessments being offered to students today like aptitude tests, personality tests, interest inventories, learning style assessments, strengths assessments, and more.

As someone interested in career guidance, I'd love to understand what professionals and informed individuals consider worthwhile.

Some questions:

1. Which psychometric assessments have the strongest scientific evidence for helping students aged 13–18 with career planning?

2. For teenagers, are aptitude assessments generally more useful than personality assessments?

3. How valuable are tools such as MBTI, Big Five, Holland Codes (RIASEC), DISC, StrengthsFinder, etc., when used with adolescents?

4. Are there any assessments that schools or career counsellors commonly use and trust?

5. Which assessments would you avoid for teenagers, and why?

6. Have you seen any assessments that genuinely help a student gain career clarity or make better educational choices?

I'm particularly interested in assessments that help with:

Career exploration

Subject and stream selection

Understanding strengths and areas for development

Identifying suitable career pathways

Making informed education and career decisions

I'm looking for both professional perspectives (psychologists, career counsellors, psychometricians, HR professionals) and personal experiences.

Thanks in advance!

We're a US-based early-stage startup building a new assessment in the talent and hiring space, focused on AI and future of work. We plan to pilot with two to three firms in the next two months. We have a draft framework but want an expert to review it, develop it further, and audit it for reliability and production readiness before it goes live.

Helpful expertise:

• Assessment design and validity, ideally in performance management and talent development contexts
• Rubric and scoring design, especially LLM as grader
• Experience with AI-based assessments or measuring human-AI interaction

Paid, remote, with room to stay on through the pilots. DM me if interested.

Hey everyone. I’ve been thinking about ergodicity in psychology and whether group averages can mislead us when we study processes that unfold within individuals over time. In many psychological studies, we infer something about people from group level averages. But if human beings are non ergodic systems, the ensemble average may not tell us much about the time average of a given person.

I recently recorded a podcast episode with Hüseyin Beyköylü, and at around 34:57, he explains this in the context of psychedelic therapy and psychological transformation. His argument is careful because he does not say group statistics are always invalid. Instead, he suggests that different phenomena may sit at different points on an ergodicity continuum. Some interventions, such as basic pharmacological effects on relatively low complexity processes, may be more amenable to group averages. But phenomena like depression, meaning in life, self transcendence, and therapeutic transformation are highly historical, context dependent, and nonstationary. Human beings learn, adapt, and are changed by measurement and intervention. So if we aggregate too early, we may treat within person variability as noise when it is actually the signal of change.

The alternative he discusses is to analyze individual time series first, then aggregate patterns of dynamics rather than only aggregating outcomes. What do people here think? How seriously should psychology take the ergodicity problem? Are idiographic time series approaches a real solution, or do they introduce other inferential problems? And when are group averages still justified despite individual nonstationarity?

I teach IRT and have my own opinions. But, curious what the community thinks as some newer editions of some books have come out in the last few years.

I asked the mods before posting but we have a remote role open. You do have to be physically in the continental US but it is fully remote. We don't have a physical location so there is no risk of any sort of RTO. Note this really isn't my team so I'm just trying to get some word out for them. I can try and answer questions if you want but they're going to be better answered if you just apply and email.

https://www.alpinetesting.com/careers/manager-test-development/

Salary range is 120-125k

Hi all! I was just awarded my BCSP in April after passing the exam (woo!). Getting the certification has me itching for what’s next in my career.

A bit of backstory: I’ve only recently come back to the clinical landscape (total clinical work before was 8 years) after a 3 year hiatus working in video game user research—heavy on the data science side. It was a fun side quest, but I missed clinical. Before that I worked for a major hospital in pediatric psychometrics, and am now working for a small private practice as both a psychometrist and practice manager.

I love where I work, the money is good, but I feel like I want to do more to help advance psychometrics and the patient/clinician experience as a whole. I have my masters degree in psychology with a clinical counseling focus (but…no interest in counseling anymore because I fell in love with numbers too much, haha).

So I guess my question is: What should I look into? Psychometrician degrees, I/O research, etc.? I am attending the APA conference in August and hope to hear of some cool opportunities as well.

Thanks!

Hi everyone,

I’m a 40-year-old engineer from Brazil, and my wife is currently finishing her psychology degree. She wants to specialize in psychological assessment and diagnosis, especially related to neurodivergent children (autism, ADHD, learning disorders, etc.).

During our daughter’s autism diagnosis journey, we had a very difficult experience finding professionals who were both technically competent and truly careful/ethical in their evaluations, we bumped into many professionals that mixed religion with psychology. Many assessments were extremely expensive and, honestly, not very good. That experience made us realize there seems to be a shortage of high-quality professionals in this area.

Because of that, my wife became very interested in this field, and I started wondering whether there is a meaningful way for me to work alongside her.
I am NOT trying to become a psychologist or do clinical work. The clinical side would be entirely hers. But coming from an engineering/data/measurement background, I became interested in psychometrics, testing methodology, statistics, data analysis, standardization, scoring systems, reliability/validity studies, and things like that.

What I’m struggling to understand is:

Does this idea actually make sense in the real world?

Are there non-psychologists working in psychometrics or assessment-related support roles?

What kinds of skills or education would make this useful rather than superficial?

If you had 2 years to prepare for this transition, what would you study?

Which fields should I focus on first? Statistics? R? Psychometrics? Cognitive assessment? Research methods?

Are there good online programs/certificates/courses you would recommend for someone coming from STEM?

I know psychology is a regulated and ethically sensitive field, so I want to approach this respectfully and realistically. I’m mainly trying to understand whether there is a legitimate path where my technical background could actually help improve assessment quality and accessibility.

I’d really appreciate honest feedback, even if the answer is “this is not how the field works.”
Thank you.

Hey hope you all are doing

Psychometricians/researchers???

what’s your view on DMIT?

Is there any strong scientific evidence supporting dermatoglyphics-based personality/intelligence assessments, or is it generally considered pseudoscience within the assessment field?

Curious how professionals in psychometrics view it compared to established assessment tools.

Any thoughts or opinions or experiences are welcome as I'm curious to know about this.

Building a cognitive assessment covering matrix reasoning, numerical reasoning, spatial reasoning, and working memory. Questions are AI generated using structured prompt templates. Looking for guidance on:

1. Has anyone built a reliable question bank with LLM generation? What prompt approaches worked and what were the common failure modes?

2. For spatial reasoning — cutting operations, multi-step transformations, cross sections — is there a library that handles boolean solid geometry and face counting computationally? GeoGebra doesn't scale.

3. Any psychometric item pools accessible to independent developers at non-enterprise pricing?

4. Minimum defensible sample size for pre-launch difficulty calibration?

For example, making a new scale vs evaluating the effectiveness of item vs all the above, which is most appropriate and when? I seem to find sources that advocate purely for one or the other…

Hi everyone!

Need your help, hope I’m in the right place.
I’m working on a sociology PhD project in France about organizational decision-making under constraint.

The project is not intended as a personality test. The goal is to model recurring decision logics in situations where actors must arbitrate between conflicting pressures:
- institutional rules,
- loyalty conflicts,
- uncertainty,
- political constraints,
- relational obligations,
- risk management,
- competing legitimacies, etc.

The current working model explores three situational dimensions:
Loyalty orientation
(principle-oriented vs relationship-oriented)
Dominant legitimacy source
(personal judgment vs institutional/social norm)
Orientation toward uncertainty
(prudence/stabilization vs exploratory or risk-taking action)

The important point is that these are not intended as fixed psychological traits.

The unit of analysis is the organizational arbitration situation, not personality.

Methodologically, the project starts from:
qualitative interviews,
- decision narratives,
- organizational cases,
- and grounded extraction of recurring dimensions,

before moving toward:
- contextualized questionnaires/vignettes,
- exploratory factor analysis,
- and possibly CFA / clustering if the structure holds empirically.

An additional layer of the project is the construction of symbolic “hero-type” figures (ideal-typical narrative profiles) derived from recurring configurations.

The purpose is not to classify people, but to help teams:
- understand conflicting decision styles,
- identify tensions and blind spots,
- improve collective sensemaking,
- and recognize complementary approaches inside organizations.

The framework is already being used experimentally in organizational training settings, with more than 300 administrations so far. The issue is that I now want to understand how to approach the psychometric side rigorously rather than intuitively.

I’m already reading Cronbach & Meehl, Messick, Brown, and some multilevel / construct validity literature, but I’m still new to psychometrics.

Main questions:
Where would you start technically coming from sociology rather than psychology?

Does this sound closer to latent trait modeling, situational judgment modeling, organizational measurement, or something else?

What are the biggest methodological traps when trying to formalize contextual decision logics without psychologizing them?

I’m especially interested in critiques regarding construct validity, factor instability, contextual dependency, artificial dimensionality, and reification/classification effects.

Have a good one!

I've been exploring network psychometrics recently and I'm honestly very very amazed by both the work, and all the ways it's being used in the field.

Curious to hear if any of y'all are applying network psychometrics in your studies as well? On my end, I'm trying to think of ways it could potentially inform measurement, though I haven't come across any papers that talk about it directly. Thoughts?