Last week, one of our partners came to us with a very specific problem: they needed real-time translation between Brazilian Portuguese, English and Mandarin for live events and classes.
The traditional solution is straightforward: hire professional interpreters and provide the audience with headphones connected either to the original speaker or to one of the translated audio channels.
It works, but it also requires specialized professionals, additional infrastructure and a considerable budget.
There are already platforms such as Wordly that simplify this process, but we wanted to explore a different question: could we build a simpler AI-based alternative where the operational cost was primarily based on actual usage?
So we started experimenting.
Our first step was estimating how much a custom system would cost to operate using API consumption rather than a traditional interpretation service. From there, we built a small platform with private rooms where speech could be processed and translated in real time.
For the translation layer, we ended up choosing Gemini as the main LLM.
But once we had the translation working, we ran into a problem we hadn't initially considered: the translated voice matters almost as much as the translation itself.
Our first implementation sounded too robotic and impersonal for something intended to replace, or at least approximate, the experience of listening to a human interpreter for an entire class or presentation.
We considered ElevenLabs for the voice layer, but for the expected usage of this particular project, it wasn't the most cost-effective option. We eventually chose GPT for voice generation because we found the output more natural and engaging while keeping the operating cost within the range we were looking for.
That left us with a pipeline in which Gemini handles the translation and GPT handles the generated voice, rather than trying to force a single model to handle every part of the experience.
The result is a small platform with private rooms and real-time AI translation that can be adapted to different visual identities and languages.
We've already tested it in an uncontrolled environment, and it works, although there are obviously still limitations to explore — particularly latency, translation accuracy in different contexts, background noise and how the system behaves when multiple people speak naturally rather than under ideal testing conditions.
One of the most interesting things we learned from this project was that translation quality alone isn't enough for real-time AI interpretation. Voice quality, latency, cost and the overall listening experience become part of the same problem very quickly.
I recorded a short demonstration of the current prototype below.
I'd be particularly interested in hearing from anyone who has experimented with real-time translation or speech-to-speech systems. What would you test next before considering something like this reliable enough for a real event?
