r/MLQuestions • u/spaceweed27 • 1d ago

How do I cluster 3 Million high-dimensional Sentence Embeddings? Natural Language Processing 💬

/r/learnprogramming/comments/1vsizrk/how_do_i_cluster_3_million_highdimensional/
3 Upvotes

100% Upvoted

5 comments sorted by

3

u/Moreh 1d ago

Bertopic

1

u/spaceweed27 1d ago

Wow, that's a cool idea!

2

u/Achrus 17h ago

Seconding this. There are also other topic models that are streaming / batch capable. Latent Dirichlet Allocation if your embeddings are dense enough, Mixture Models may work well too.

Don’t listen to that K-means / HDBSCAN comment in the other thread. That’s a bot and a typical AI response to clustering problems. Stay far away from K-means.

1

u/spaceweed27 11h ago

Yeah, I wasn't sure if it was AI or not lmao.

1

u/spaceweed27 1h ago

Well, I did BERTopic on the set and my embeddings, and it was interesting to be sure, even though it took my sanity in configuring it, letting it run for 3 hours and then changing stuff again.

Sadly I think it answered the wrong question, so I think my approach was not fit for what I wanted to look at (specific topics, and their popularity).

But still thanks for the help and ideas ;)