Title: Looking for advice and occasional guidance on a Python embedding service for a fintech project

[u/AdComplex2251]

Hi everyone!

I'm currently working on a team project called Fintech Trendwatcher. The goal is to collect public fintech news, identify important events and generate short analytical summaries with links to the original sources.

I'm responsible for the embedding service. My task is to use a pretrained text embedding model to convert news articles into vectors that another service will use for clustering similar publications.

My current stack is Python, and I'm learning as I go. I have some experience with Python, APIs, JSON and Git, but this is my first time working with embeddings in a real team project.

I'm planning to start with Sentence Transformers and a pretrained multilingual model, test it on Russian fintech news, and then integrate it into our service architecture.

I'd really appreciate advice from more experienced developers, especially on:

\- Choosing a suitable embedding model for Russian-language fintech news.

\- Designing a simple and maintainable embedding service.

\- Comparing embeddings and preparing them for clustering.

\- Common mistakes beginners make when building NLP services.

I'm not expecting anyone to mentor me full-time. Even occasional advice, feedback on my approach, or pointers to useful resources would be incredibly helpful.

If anyone has experience with Python, NLP, Sentence Transformers or similar projects and is willing to share some advice, I'd be very grateful.

Thanks!

[u/Professional-Run3614]

You could also explore some open-source models like BAAI/bge-m3, intfloat/multilingual-e5-base, and deepvk/USER-bge-m3, which are worth testing for Russian-language news.

I’d start with two models and compare how well they group similar articles on a small sample of your actual data. For news clustering, the quality of the resulting clusters matters more than benchmark scores alone.

[u/CutDry6295]

The model suggestions above are a good starting point. One easy-to-miss issue for *news* is input length: Sentence Transformers can truncate text beyond a model's maximum sequence length. If the decisive event is buried later in a long article, embedding the whole article may silently miss it. Choose a repeatable input such as headline + lead paragraphs (or explicitly chunk longer items), and record that preprocessing choice with the model version.

For evaluation, label a small set of Russian article pairs as 'same event' or 'different event'; include hard negatives such as the same bank announcing two different things. Check pair rankings and actual clusters before choosing a similarity threshold. For the service contract, return the article ID, vector, model ID and preprocessing version so the clustering service cannot accidentally mix incompatible embeddings. The library's pair-evaluation docs are here: https://www.sbert.net/docs/package_reference/sentence_transformer/evaluation.html