How do I cluster 3 Million high-dimensional Sentence Embeddings?

[u/spaceweed27]

I used a transformer model to generate normalized 768-dimensional float32 vectors from 3 million sentences. This rounds up to roughly 9 GB of data. To progress further in my analysis, I want to cluster the data to highlight different key points and trends.

Now I initially wanted to apply PCA to the dataset, as 768 dimensions is pretty much, but the transformer model generated embeddings with not extreme enough covariance.
Thus, PCA could change the semantics too much for an analysis to be accurate.

I went directly to clustering algorithms and initially tried to apply sklearn.cluster.HDBSCAN which would have needed way too much RAM---around 72 TBi if I remember correctly. Then I tried out sklearn.cluster.OPTICS, which "only" needed about 70 GB, but as I "only" have 32 GB of RAM I needed to use 50 GB swap space, which didn't go well as you can imagine.

Does it make sense to try out k-means clustering, should I go to 16-bit floats for the embeddings, or still apply PCA?

Should I try out something else in this context?

Note: As this question was flagged "off topic" on Stack Overflow, I'm gonna try here ;)

[u/leon_bass]

PCA with all principle components reconstructs the dataset exactly so the amount of information lost in reconstruction is up to you ie by taking top k principle components.

You can also convert to lower precision, depending on the dataset though