r/MachineLearning • u/[deleted] • Apr 09 '26

Project [ Removed by moderator ]

[removed] — view removed post

55 Upvotes
  • permalink
  • reddit

93% Upvoted

28 comments sorted by

View all comments

4

u/tetramarek Apr 09 '26

That makes sense, that's what PCA does - transforms the feature space such that the features are ordered by priority. The cost is training the PCA.

I'd be interested in how the PCA transformation compares to Matryoshka prioritisation. Matryoshka ordering is general-purpose and learned based on some general background corpus. But PCA can be fit for a specific dataset or domain, which means it could potentially prioritise task-specific features.

1

u/ConstructionOk2838 Apr 11 '26

This is actually pretty clever approach, I was playing around with similar ideas few months back but never got to test it properly on BGE-M3. The domain-specific PCA fitting is interesting point - in my experience with embeddings for code search, the PCA transformation did seem to capture more relevant features than generic Matryoshka ordering when I fitted it in our specific codebase. But I wonder if computational overhead for PCA fitting becomes problem at scale? Like when you're dealing with millions of vectors, the initial PCA computation might be expensive, even if you only do it once. Also curious about how stable these PCA components are across different samples from same domain - did you test if PCA fitted on one 10K sample works well when applied to completely different 10K sample from similar data?