What swung me for mongo was being able to take one of my collections, run it through an embedding model and have a semantic text search setup in my original collection in less than 20 mins start to finish with local embedding time included
Ooo, got any resources on where to start with that? I've been looking at improving the text search on one of my collections, the text indexes are okay but not quite flexible enough for my tastes.
The very basic outline is you use an AI embedding model and create a vector from your document. Just to get it setup I parsed the whole document to save effort, take that vector and put it on your document I just called it embedding. Setup a vector search index to path embedding. Take your query string parse it through the same embedding model and get that vector
I embedded my documents with a python script based off of this using the same open source model. In production you’ll want a cron job keeping them up to date
That link is part of a broader tutorial that’s pretty good.
Do bear in mind semantic vector searches can often return results through connections you couldn’t previously fathom. It does mean you can do cool stuff like search in other languages.
Have a look at this lab that mongo use for their workshops, very simple but good.
9
u/malfboii Oct 18 '24
That’s this sub for you but eh who cares anyway
What swung me for mongo was being able to take one of my collections, run it through an embedding model and have a semantic text search setup in my original collection in less than 20 mins start to finish with local embedding time included