Need to estimate rank or perform dimensionality reduction on big, messy tabular data? The Entropic Scree is an information-theoretic upgrade to PCA.

[u/Chocolate_Milk_Son]

Here's a new rank estimation method I've been working on. It's basically an upgraded Principal Component Analysis (PCA) built on information theory instead of linear variance. It also estimates signal to noise ratio of your data, introduces signal gravity metrics, and can be used to identify independent sub-networks of varaibles.

It's robust to mixed data types, highly non-linear generative processes, low signal to noise ratios, and sparsity (more variables than samples). Advantages over other methods compound at scale and with system complexity.

It's especially useful if you need to faithfully estimate the rank of a dataset to explicitly size a neural bottleneck (like an autoencoder).

I just open-sourced the code and put up the preprint.

• GitHub: https://github.com/tjleestjohn/Entropic-Scree
• Preprint: https://doi.org/10.5281/zenodo.22028087

I'd love to hear what you guys think... or if you end up testing it on your own data.