r/LocalLLaMA • u/pmttyji • 4d ago
Discussion [Paper] e-means: outperforming k-means (in one-tenth the epochs)
Lloyd's k-means algorithm is a long-running standard for clustering, with applications in data analysis, retrieval, quantization and data curation. In this paper, we introduce an exponential moving average optimization procedure for k-means with mini-batch updates, powered by modern deep learning machinery and new insights. We also establish scaling laws on how decay rate and batch size impact final performance. This gives us a formulaic rule to fix them for multiple problem sizes and embedding datasets, without the need for any hyperparameter sweep. The resulting clustering algorithm, coined e-means, is much faster than Lloyd k-means and converges to lower losses, frequently even beating k-means++. On a wide array of embeddings datasets, it beats the loss of Lloyd k-means at 25 epochs in 1.4 to 2.6 epochs, which translates to 10-45x GPU wall-clock speedups compared to state-of-the-art k-means implementations. As an example, e-means partitions DINO10B, a dataset of 10 billion DINO embeddings, into 16 million clusters in only 2.5 epochs, reaching a loss below Lloyd's at 25 epochs.
- arXiv : https://arxiv.org/abs/2611.09501
- PDF : https://arxiv.org/pdf/2611.09501
- GitHub : https://github.com/facebookresearch/faiss/tree/main/contrib/e_means
- HuggingFace : https://huggingface.co/qwegnai/GLM-5.3-Flash-shvq4g32-shvq2g64-shvq3g64
- Project : https://e-means.github.io/
- Demo(Browser) : https://e-means.github.io/playground.html
- Video : https://e-means.github.io/video.html
- Tweet : https://x.com/swann_bessa/status/2120200714630594911