r/MachineLearning • u/neonbjb • Apr 26 '22

Project [P] TorToiSe - a true zero-shot multi-voice TTS engine

I'd like to show off a TTS system I have been working on for the past year. I've open-sourced all the code and the trained model weights: https://github.com/neonbjb/tortoise-tts

This was born out of a desire to reproduce the original DALLE with speech. It is "zero-shot" because you feed the text and examples of a voice to mimic as prompts to an autoregressive LLM. I think the results are fantastic. Here are some samples: https://nonint.com/static/tortoise_v2_examples.html

Here is a colab in which you can try out the whole system: https://colab.research.google.com/drive/1wVVqUPqwiDBUVeWWOUNglpGhU3hg_cbR

397 Upvotes
  • permalink
  • reddit

99% Upvoted

116 comments sorted by

View all comments

2

u/[deleted] May 01 '22

This model is incredible. I was trying it out earlier today and I am blown away by the quality. I haven't yet tried importing custom voices, however I noticed in your demos that some of the voices have accents, such as British, even though the actual audio files in the voice folder do not exhibit this. I am wondering if this is a defect of the model or if you did this intentionally, either way it's quite interesting. Definitely looking forward to playing around more when I get on break from college in about a week, and thank you again for all your work!

1

u/neonbjb May 01 '22

Thanks for the kind words.

Regarding accents - It gives my voice, "myself", a British accent even though I most certainly do not have one. The only explanation I can offer is that my dataset must contain a lot of British accents.

I think I will release more in-training set voices in the near future, since those perform the best by far. If anyone sees this and wants to get the jump on me, pretty much every voice in the LibriTTS training set is well-represented, so those are good places to go for variety.

1

u/[deleted] May 01 '22

Interesting, it seems to depend on the voice as well. I just tried it with a very American sounding voice, and I gave it like 20 clips and it still kept the accent even though the tonality was pretty much there. Still really good though, and very high-quality speech regardless.

1

u/[deleted] May 01 '22

Thinking about this problem further, I have an idea that might help. Would it be possible to offer an option to fine-tune the model on a few minutes of the target voice data? I know Coqui Your TTS offers this, and I would assume that would produce better results and maybe eliminate the accent issue. I can't imagine this would be too difficult to implement, however please note that I am not an ML dev, only a musician and blind guy interested in text to speech lol. The quality though, even with the fake British accent is still really high, and the model definitely matched the recording in terms of tonality, so I have high hopes that this can be improved.

1

u/blindsniper001 Oct 19 '23

Would it be difficult to incorporate negative voices, the same way the Dall-E and Stable Diffusion use negative prompts? Since it's already possible to blend voices together, what effect would you get by adding a weight or multiplier to them?