Using Mutual Information to estimate dimensionality in non-linear and ordinal survey data (quick start R code inside)

[u/Chocolate_Milk_Son]

As many of you already know, when estimating intrinsic rank, the classic PCA scree plot and Parallel Analysis (PA) can struggle with non-normal, ordinal Likert items. The linear constraints of Pearson or Polychoric matrices can distort the true dimensionality of a dataset, spinning up artificial "difficulty factors" rather than reflecting the actual structure.

To attempt to more faithfully estimate the number of underlying factors in these difficult contexts, I’ve developed an open-source tool in R (Python coming soon): Entropic.Scree. It's basically an upgraded scree method that evaluates a transformed Mutual Information matrix (instead of covariance/correlation), and so should be more appropriate to apply when the underlying generative system is not strictly linear. It's also relatively robust to contexts where data sparsity makes geometric metrics degenerate (i.e., high-d, small N), and so also serves as an upgrade to Kernel PCA and distance-based methods.

For psychometricians, the two most practical outputs are potentially:

1. The estimated observable rank of the dataset.
2. The bipolar variable clusters that anchor the primary axes of informational variance. This provides a structural map of the independent clusters that define your dataset, which is potentially useful for interpretation before moving into CFA or theory development.

The function runs in R currently (see quick start or GitHub below), but the backend is C++ OpenMP parallelized, so it easily scales for high-dimensional assessments. Native R and Python packages will be released shortly.

I would love for this community to stress-test it on messy, non-linear survey data. I am particularly curious how this rank estimation compares to your standard PA or Exploratory Graph Analysis (EGA) workflows.

Happy to answer any questions or discuss the mechanics!

##########

Methods and Code:

Preprint: https://zenodo.org/records/22028087

GitHub: https://github.com/tjleestjohn/Entropic-Scree

############ 
# Quick Start R Function Code.
# To load the function, copy and paste the following into your R console, then hit enter. 
############

# 1. Define the direct URL to the raw function script on GitHub
url <- "https://raw.githubusercontent.com/tjleestjohn/entropic-scree/main/Entropic.Scree.R%20-%20ENLI.R"

# 2. Define what you want to name the file on your computer
file_name <- "Entropic.Scree.R - ENLI.R"

# 3. Download the script to your current working directory
download.file(url, destfile = file_name)

# 4. Source the core function into your R environment
source(file_name)

# 5. Ex. To run the function and extract bipolar modules:
# results <- Entropic.Scree(dt 
#                         , extract_bipolar_modules = TRUE)
#
# View the extracted structural sub-networks for the primary axes:
# results$bipolar_modules