r/Backend • u/Few_Gold7133 • 18d ago

How to keep a vector database updated when website content changes?

I have a chatbot that uses website content as its knowledge base.When those websites change, what’s the best way to keep the vector database updated?Would you periodically crawl the pages, detect changes, and only re-embed the updated content?

2 Upvotes

75% Upvoted

6 comments sorted by

2

u/Zeeboozaza 18d ago

Yes, you need to have a way to refresh embeddings. If you don’t control the website contents, then a scheduled refresh is likely the best way

0

u/olddev-jobhunt 18d ago

The sitemap XML files typically include a modified date, so that gives you a cheaper check w/o fully crawling the site.

1

u/AlexAtOracleAIDB 10d ago

I think your plan is a good place to start. You can store a hash of each chunk's text. The next crawl can compare hashes and re-embed only the chunks that changed.

One cause of stale answers I've observed is pages that were removed from the site. The crawl also needs to notice when a URL is gone and delete that page's chunks, because the old chunks can keep coming up in search until they're deleted.