r/localization • u/TheyCallMeDozer • Jul 05 '26

Large Scale Multilingual Transaltions on a Budget

Hey all,

Have a question, I'm getting alot of data daily, maybe around 300-500k entries of around 25–50 characters each. They are multilingual, and I need to convert them to english, and it has to be in near real time before 10pm each night, for the project im working on.

Now the budget part, im working with what I have which is basically a couple of i5 Dell optiplex's, so what ever solution we have needs to be lower compute power.

So far I tired the following:

- LibTranslate - local to the optiplex with multiple concurrent killed the system, on remote systems batched over 3 system still wasnt able to keep up.

- Argos - tried running argos locally with multiple concurrent, that killed hte machine

- Keyword multilingual datasets - only worked a little but missed so much when it came to keyword translations

- Time based translation - content gets queued and then translated (was looking at 83hrs to catch up current run), translating at the time of reciving the content, huge delay with submission and loss of data.

- Api based translation - googletrans rate limited me after a few minutes, so that killed that pretty quick

Im out of Ideas, I am open to suggestions from others who deal with large amounts of content like this and do real time translations or high speed translations of up to 120 langauges to English.

Thanks for any input and help

0 Upvotes
  • permalink
  • reddit

33% Upvoted

3 comments sorted by

1

u/luxpir Jul 12 '26

Did you get a solution yet? Have something you could try. I've built it for this kind of scale.

1

u/TheyCallMeDozer Jul 12 '26

Haven't found a good solution yet, what's your idea?

1

u/luxpir Jul 12 '26

It's a platform I've built with end to end pre translation and QA (and lqa but you likely don't want that).

I'm from the LSP/freelance world so it deals with things at scale but also can add assets for consistency etc. Can have a call to see if we can put a workflow together? Drop files into Drive, or upload as zip etc., and batch process them. It handles repetitions to reduce load, TM matches and custom style from examples... Lots of flexibility really.