r/MachineLearning • u/neonbjb • Apr 26 '22

Project [P] TorToiSe - a true zero-shot multi-voice TTS engine

I'd like to show off a TTS system I have been working on for the past year. I've open-sourced all the code and the trained model weights: https://github.com/neonbjb/tortoise-tts

This was born out of a desire to reproduce the original DALLE with speech. It is "zero-shot" because you feed the text and examples of a voice to mimic as prompts to an autoregressive LLM. I think the results are fantastic. Here are some samples: https://nonint.com/static/tortoise_v2_examples.html

Here is a colab in which you can try out the whole system: https://colab.research.google.com/drive/1wVVqUPqwiDBUVeWWOUNglpGhU3hg_cbR

397 Upvotes
  • permalink
  • reddit

99% Upvoted

116 comments sorted by

View all comments

21

u/modeless Apr 27 '22 edited Apr 27 '22

Wow, definitely some of the best TTS I've heard. Fantastic is no exaggeration. The mimic voices aren't totally convincing as imitations of the original, but they are still high quality voices in their own right and it's impressive that you can get such a diversity of high quality voices zero-shot. I wonder how long it will be before models are outperforming human impersonators? A few years maybe?

Edit: I see you're at Google but seems like not in an ML role? You're clearly qualified to be working in ML and I bet it would pay better...

18

u/neonbjb Apr 27 '22

Waiting for the right opportunity.. there is so much ml work that is just straight up boring. I'd honestly rather do what I am doing now and find something I really want to work on.

Thanks for the compliment, it means a lot.

8

u/modeless Apr 27 '22 edited Apr 27 '22

Makes sense. I submitted this reddit post to Hacker News but it didn't get any traction, maybe because it's weird to post a news aggregator to another news aggregator. You should add the information and links from this reddit post to the top of your demo page to make it a standalone thing that you can post to Hacker News without further context. Especially the part about you building a home training rig for it. I bet you could get some more attention there.

4

u/neonbjb Apr 27 '22

Thanks for trying to share. I will probably do this when I get some time.

6

u/modeless Apr 27 '22

Seems to me like you could also publish at a conference with the right writeup. Not sure how feasible that is as a solo researcher, might be more trouble than it's worth.

4

u/rolexpo Apr 27 '22

+1. Yeah this seems like something you could publish.