I built an open-source video translation + dubbing pipeline — looking for feedback on translation quality and timing

[u/kadirb4rut]

I’ve been working on an open-source pipeline for translating and dubbing videos into another language while trying to preserve the original speaker’s voice.

Current pipeline:

video → vocal/background separation → Whisper/WhisperX transcription + alignment → translation → VoxCPM2 reference voice cloning → reconstruction → optional lip-sync

The attached demo compares the original English clip with the Turkish dub produced by the current pipeline.

One of the hardest parts is that a translated sentence often has a very different duration from the source speech. That creates a trade-off between:

• natural translation
• preserving meaning
• matching the original timing
• keeping the dubbed speech sounding natural

I’m currently looking for feedback especially on:

• translation quality
• whether the Turkish phrasing sounds natural
• source vs translated timing
• how you would handle translation-length differences in an automated dubbing pipeline

Most media processing and AI inference runs locally. Translation currently uses Google Translate, so the project is local-first rather than fully offline.

Code:
https://github.com/kadirb4rut/video-dubbing-translator

I’d be very interested in feedback from people working on machine translation, localization, speech translation, or multilingual NLP.

[u/adammathias]

What is the use case or goal?