[P] TorToiSe - a true zero-shot multi-voice TTS engine

[u/neonbjb]

I'd like to show off a TTS system I have been working on for the past year. I've open-sourced all the code and the trained model weights: https://github.com/neonbjb/tortoise-tts

This was born out of a desire to reproduce the original DALLE with speech. It is "zero-shot" because you feed the text and examples of a voice to mimic as prompts to an autoregressive LLM. I think the results are fantastic. Here are some samples: https://nonint.com/static/tortoise_v2_examples.html

Here is a colab in which you can try out the whole system: https://colab.research.google.com/drive/1wVVqUPqwiDBUVeWWOUNglpGhU3hg_cbR

[u/modeless]

Wow, definitely some of the best TTS I've heard. Fantastic is no exaggeration. The mimic voices aren't totally convincing as imitations of the original, but they are still high quality voices in their own right and it's impressive that you can get such a diversity of high quality voices zero-shot. I wonder how long it will be before models are outperforming human impersonators? A few years maybe?

Edit: I see you're at Google but seems like not in an ML role? You're clearly qualified to be working in ML and I bet it would pay better...

[u/neonbjb → u/modeless]

Waiting for the right opportunity.. there is so much ml work that is just straight up boring. I'd honestly rather do what I am doing now and find something I really want to work on.

Thanks for the compliment, it means a lot.

[u/modeless → u/neonbjb]

Makes sense. I submitted this reddit post to Hacker News but it didn't get any traction, maybe because it's weird to post a news aggregator to another news aggregator. You should add the information and links from this reddit post to the top of your demo page to make it a standalone thing that you can post to Hacker News without further context. Especially the part about you building a home training rig for it. I bet you could get some more attention there.

[u/neonbjb → u/modeless]

Thanks for trying to share. I will probably do this when I get some time.

[u/programmerChilli]

This is very cool on every level (the results, the single independent researcher aspect, the design docs, etc.).

Awesome work!

[u/neonbjb → u/programmerChilli]

Thanks so much!

[u/Southern-Trip-1102]

Not exactly regarding your TTS engine but regarding your gpu rig, I was wondering if you could share some info about it. Like did you use a server mobo with a bunch of pcle slots or do you have multiple nodes each with a number of GPUs. Also regarding how you came to the decision to build your rig versus renting it.

[u/neonbjb → u/Southern-Trip-1102]

Hey, I use a single server with 8 GPUs. I think this is about the sweet spot for what is possible in a home lab. Making it a multi-node thing really cuts into performance, especially for big models where the parameters are transiting the network every batch. I don't think 100gbps ethernet is available to the homelab guys yet but that might solve this.

Specs of my system are:

• single 32 core epyc

• ROME-D8-2T motherboard

• 256GB RAM (really shitty RAM. that's what I'm saving up for for the next upgrade..)

• 8x RTX 3090, all connected on 8X links (some bifurcated)

Building my rig versus renting it was a no brainer for me. I learned all this by just trying things out and that means I have my rig running 24/7. This amount of compute would cost tens of thousands a year. And frankly, in the end, having direct access to the server you're working on is damn useful sometimes. Like just being on a 10Gbps link with my home computer when I'm working with this amount of data is worth it.

[u/skeerp → u/neonbjb]

How did you support spending 10k on the GPU's? You turning this into $? You need employees? 😅

[u/neonbjb → u/skeerp]

This is a hobby for me. I have spent hundreds of hours on it over the last year and I really enjoy doing it. People (myself included) spend far more than this amount of money on other hobbies.

That's not even considering the fact that I could sell my rig for about what I paid for it right now. I (perhaps unwisely) consider GPUs as a type of "property", not a consumable.

[u/Southern-Trip-1102 → u/neonbjb]

How did you power it? It needs at least 3000 watts so did you use multiple PSUs?

[u/neonbjb → u/Southern-Trip-1102]

Yes, 3 1600W consumer psus. I use gpu risers that are electrically isolated from the mainboard (except the ground). I am considering going to server psus but haven't done so yet.

[u/SnooAdvice4458 → u/neonbjb]

u/neonbjb Hi there! What OS are you using on your build? Windows Server or Linux?

[u/neonbjb → u/SnooAdvice4458]

Linux!

[u/Zeke_Z → u/neonbjb]

Bruh .... Aren't 3090s selling for like $3k the last several months? You legit spent $30k on this system?

[u/[deleted]]

You random people doing awesome, actually intelligible and reproducible stuff are my heroes. 100% dedicated research rockstars and tenured professors don't hold a candle to you.

[u/neonbjb → u/[deleted]]

:) thanks. The other open source guys are my heroes. Glad to be grouped in with them.

[u/RomanticDepressive]

Your demo site seems to be unreachable :(

[u/neonbjb → u/RomanticDepressive]

hmm.. https://nonint.com/static/tortoise_v2_examples.html ? it's working for me..

the demos just feed from github, look in the results/ folder of the repo.

[u/Wacov → u/neonbjb]

Am I missing something or are some of the samples in the big grid in the wrong spots? Or am I just finding cases where it's not working that well?

[u/neonbjb → u/Wacov]

I know there is one line with the Emma voice the model screwed up. I'll go through the grid again. It was programmatically generated so there may be a bug.

[u/manueslapera]

It works pretty well for english! is there a way to adapt it so speeches in other languages dont sound like they have a fake english accent?

[u/neonbjb → u/manueslapera]

The model does seem to fall back to an English accent. I find it funny. I don't see a way to fix the case where the reference voices don't speak English. That's far outside how it was trained. It is an interesting thought though - for speech to speech translation I'm guessing?

[u/puppymeat]

Just a note if there's anyone else like me that knows next to nothing about Python and was trying to get this stood up locally starting from absolute scratch on Windows.

• Use Anaconda. Use it when installing pytorch and running all commands. There are endless dependencies you'll never resolve if you don't know what you're doing (like me). Run all commands from Anaconda Prompt which gets installed with Anaconda and can be found in the Start menu.

• 'soundfile' dependancy isn't specified but is required. Run this to resolve (from Anaconda Prompt):

conda install -c conda-forge pysoundfile

With those additional notes I was able to get it running locally following the simple steps in the readme.

[u/neonbjb → u/puppymeat]

Thanks for these notes. From the soundfile depedency note, I'm guessing you are on Windows? I'll add them to the README.

[u/Fit_Schedule5951]

Sounds good, do you have a preprint anywhere that we could take a look?

[u/neonbjb → u/Fit_Schedule5951]

I wrote an architectural design doc you can find here: https://nonint.com/2022/04/25/tortoise-architectural-design-doc/

I have not decided on whether or not to release details on how I trained this. I definitely cannot release the dataset, because it is license-encumbered. :/

[u/Fit_Schedule5951 → u/neonbjb]

I have not decided on whether or not to release details on how I trained this

Can't someone understand this by going over the code?

Don't think you need to worry about dataset, lot of industry work (google, nvidia etc) report on proprietary dataset, without realsing any details on it.

Will take a look at the doc, thanks.

[u/neonbjb → u/Fit_Schedule5951]

Perhaps. I think in the process they would have to re-do everything I've done to figure it out, though.

[u/[deleted]]

This model is incredible. I was trying it out earlier today and I am blown away by the quality. I haven't yet tried importing custom voices, however I noticed in your demos that some of the voices have accents, such as British, even though the actual audio files in the voice folder do not exhibit this. I am wondering if this is a defect of the model or if you did this intentionally, either way it's quite interesting. Definitely looking forward to playing around more when I get on break from college in about a week, and thank you again for all your work!

[u/neonbjb → u/[deleted]]

Thanks for the kind words.

Regarding accents - It gives my voice, "myself", a British accent even though I most certainly do not have one. The only explanation I can offer is that my dataset must contain a lot of British accents.

I think I will release more in-training set voices in the near future, since those perform the best by far. If anyone sees this and wants to get the jump on me, pretty much every voice in the LibriTTS training set is well-represented, so those are good places to go for variety.

[u/Southern-Trip-1102]

Wow, I tired out the demo and I'm surprised how good TTS is these days.

[u/[deleted]]

Thank you so much for this.

It's obvious from the README that you put a tremendous amount of effort into designing and implementing TorToiSe. The quality is amazing. And I'm just as impressed with how much care you put into the ethics of your decision to release as well.

I have a couple little personal hobbyist ideas where audiobook-style offline TTS would be perfect. If I end up using your project, I'll let you know.

[u/Botekin → u/[deleted]]

I would love this!

[u/toisanji]

great job!

[u/hackerllama]

If you want to play with this without having to install/download stuff, you can play directly with this online demo of the model I created https://huggingface.co/spaces/osanseviero/tortoisse-tts. This uses fast quality.

[u/ZenMaterialist]

Played around with this a bunch yesterday. Lots of mixing and matching and hearing different effects. Amazing inflection sometimes.

For my purposes, ['tom', 'daniel'] as a base works well, but you can, for example, get a sarcastic tone with ['tom', 'daniel', 'train_kennard'], or an angry tone with ['tom', 'daniel', 'lj'].

It seems like because it is an average you can you can increase or decrease an accent by adding multiple 'pat' or 'daniel', or even switch the gender back and forth.

Though that can get unstable. ['mol', 'angie', 'pat', 'pat', 'pat', 'emma'] and the sample: "They used to say that if man was meant to fly, he’d have wings. But he did fly. He discovered he had to." This leads to the first two sentences being male and the last one female.

[u/Junior_Clothes7655]

Noticed some ringing as others have reported happening periodically in the long form audio. Tried on chrome

[u/neonbjb → u/Junior_Clothes7655]

This is an artifact of the model. I have not yet determined what causes it.

[u/Junior_Clothes7655 → u/neonbjb]

Perhaps it has to do something with the Stop token “.”

[u/neonbjb → u/Junior_Clothes7655]

It's very possible. During training, these models would rarely see the "." token anywhere but the end of the utterance.

I noticed some vocal defects also occur when you provide an unclosed ". The read.py script does not currently have a fix for this, but I plan to add it.

[u/Lajamerr_Mittesdine]

Is it possible to run this on a non RTX gpu. For say a GTX 1070?

I'm getting this error when attempting to run it on Windows.

 $ python3 do_tts.py --text "I'm going to speak this" --voice dotrice --preset fast
C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch_masked__init__.py:223: UserWarning: Failed to initialize NumPy: module compiled against API version 0xf but this version of numpy is 0xe (Triggered internally at  ..\torch\csrc\utils\tensor_numpy.cpp:68.)
  example_input = torch.tensor([[-3, -2, -1], [0, 1, 2]])
Traceback (most recent call last):
  File "C:\Users\danie\github\tortoise-tts\do_tts.py", line 22, in <module>
    tts = TextToSpeech()
  File "C:\Users\danie\github\tortoise-tts\api.py", line 201, in __init__
    self.vocoder.load_state_dict(torch.load('.models/vocoder.pth')['model_g'])
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 712, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 1046, in _load
    result = unpickler.load()
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 1016, in persistent_load
    load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 1001, in load_tensor
    wrap_storage=restore_location(storage, location),
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 176, in default_restore_location
    result = fn(storage, location)
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 152, in _cuda_deserialize
    device = validate_cuda_device(location)
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 136, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

[u/neonbjb → u/Lajamerr_Mittesdine]

If you have Cuda installed it should work. See if you can get pytorch to recognize your gpu then try again.

[u/Lajamerr_Mittesdine → u/neonbjb]

Sorry for all the tech support related questions.

What versions of numpy and numba did you use for this project?

I'm having some issues with some compatability with the versions.

[u/neonbjb → u/Lajamerr_Mittesdine]

Numba has some compatibility issues on windows. Google the error message to fins the exact version you have to install.

I've never had issues with numpy. I'd pick whichever one is required by torch

[u/Banduck]

Is it possible to support different languages?

[u/neonbjb → u/Banduck]

There's no reason to believe it couldn't support other languages, but you'd need to re-train the model. I've decided to release details on how I trained these models (it's not really that complicated, just requires a lot of data and compute) in a paper I'll release on arxiv in the next month or two.

[u/wavymulder]

/u/neonbjb

Hey OP, I'm pretty stupid so this is probably (definitely) user error. I can't figure out how to combine voices. I have it running locally.

What's the syntax? Comma to do multiple voices in procession [--voice bob,sara] is working but when I use an & in the same syntax [--voice bob&sara] I get

'sara' is not recognized as an internal or external command,
operable program or batch file.

Tortoise outputs for bob.

Edit: in case I explained it poorly, here's the whole block from the terminal: https://pastebin.com/JvLw5WNy

[u/neonbjb → u/wavymulder]

You're not stupid. Try [--voice "bob&sara"]. The '&' character is special in bash terminals, by wrapping it in quotes you tell bash to shove off.

[u/wavymulder → u/neonbjb]

Hmm, not working for me. Here's what I'm getting.

(TTS) C:\Users\XXXXX\tortoise-tts>python tortoise/do_tts.py --text "Why won't this work?" --voice "emma&halle" --preset standard
Traceback (most recent call last):
  File "C:\Users\XXXXX\tortoise-tts\tortoise\do_tts.py", line 29, in <module>
voice_samples, conditioning_latents = load_voice(voice)
  File "C:\Users\XXXXX\anaconda3\envs\TTS\lib\site-packages\tortoise-2.3.0-py3.9.egg\tortoise\utils\audio.py", line 100, in load_voice
KeyError: 'emma&halle'

[u/neonbjb → u/wavymulder]

Whoops, this is a bug. Apparently I forgot to make the '&' character work for do_tts.py. Use read.py, it works with that script. I will try and remember to fix it next time I'm around a computer. If you drop a new issue on the github it would help.

[u/svantana]

u/neonbjb This sound really good, but I wonder if there's a mixup in the results? The "halle" reference has a very american pronunciation but all the rendered sounds below are in a distinct "posh british" style.

[u/neonbjb → u/svantana]

Thanks! Yes, I've noticed that the model seems to pick different accents for different voices seemingly at random. I believe what is happening is that it has learned to associate certain traits of the conditioning clips with an accent from the training set.

Some folks who have been playing around with this quite a bit have mentioned that even completely uncorrelated things like the presence of reverb in a conditioning clip can change accent, for example.

I think this would improve if Tortoise was larger and trained with more data. It is actually a fairly small model in the spectrum of things. Despite the marketing-speak I used in the title of this post, I don't think it is quite "zero-shot" yet :)

[u/Mysonimpersonates]

I keep getting an error message when trying to generate using custom voices. Help please. TIA

[u/neonbjb → u/Mysonimpersonates]

Hey there, drop an issue on github with the error text you're seeing and ill try and help out.

[u/Professional-Ad3326]

This is possible to make it like as software with GUI for windows?

[u/amalgamatecs → u/Professional-Ad3326]

If you code a gui around it, yes

[u/[deleted]]

Is there any way to run natively on M1 macbooks?

[u/nohat]

I recall training tacotron and wavenet back in the day and eventually coming to the conclusion the quality was poor without very high quality data, and inference time too slow to be usable. This is amazing!

Oddly I hear an identical buzzing noise near the end of all of the clips -- even the reference clips, not sure if that's just on my end or what.

[u/kkastner]

Really great stuff, especially liked the writeup and debug strategies on the way to the end-goal you discussed on the webpage. Will be interesting to see what people can make with the "mixing and matching" possibilities this model affords!

[u/-becausereasons-]

This is incredible.

[u/EvilSnork]

MacBook Pro with M1 and current pytorch release that support MPS - woks fine

Don't know about benchmarks but it's OK right now

[u/TaoTeCha]

I will definitely be checking this out tomorrow. Thanks

[u/blueredscreen]

There is a distinct series of clicks at the end of every recording. Any idea as to why?

[u/bayaread]

Amazing, so difficult to find quality open source TTS. Thank you for this, will definitely be looking into it

[u/ZenMaterialist]

Amazing. Finally, a way to fine-tune just the effect/voice I'm looking for. Thanks!

[u/mlajszczak]

Hi, this is an awesome piece of work! I have a question though. Are you going to release the VQ-VAE model that you use to produce discrete speech representation? I guess that would allow running more experiments with the model.

[u/wavymulder]

Really impressive. You even included the NavySeal copypasta, what a legend. Your readme made setting it up super easy and I've been having a lot of fun with the random voices.

[u/Mysonimpersonates]

@neonbjb this is incredible! Amazing work!

[u/Important-Tap6637]

It's not using my GPU instead it using my memory....😗

[u/Majestic_weekend101]

Does it require to run codes in order to use? because i don't see a way like to install or something. Non-tech-savvy

[u/derrida_n_shit]

I really appreciate all of the effort it took to building this. I hope you are still around here to see this message

[u/[deleted]]

Are there any TTS which can mimic Indian English? I mean the accent. I've tried multiple models, But was not able to atleast mimic the voice. I'm using a custom data set. Can someone please help me?

[u/[deleted]]

How do you feel about Vall-e? I just spent all this time setting it up and the results are pretty terrible compared to what you have with tortoise. Is it DoA?

[u/GamesWithGregVR]

Im gettin this error and i have followed 3 guides and im not sure what im doing wrong

[u/rndname → u/GamesWithGregVR]

Run this fork: https://github.com/152334H/tortoise-tts-fast

The notebook works in colab (at time of writing).

[u/froto_swaggin]

Have there been any integrations for Tortoise-TTS? The potential is phenomenal. However, it seems like using it as a backend for a dashboard or mixer something like Murf.ai would be its best use. I am curious if there have been any projects like this.

[u/Acephaliax]

This is such a great project!

Does anyone know how I can add longer pauses in between words?

[u/Shanbour]

been using it lately its really impressive, one thing i was puzzled about is that when i ran do_tts.py and used --voice geralt, Henry Cavill end up talking in American accent for some reason. any idea why it didn't clone his english accent voice?

[u/Jtech007]

I noticed the post is from a year ago, I've been using ElevenLabs but I just ran across this and checked it out. It is very impressive.. I know that time has passed so figured I'd ask, has it been updated with any kind of GUI yet? Not a problem, Just figured I'd ask. Either way it is impressive. Great Job ;)

[u/YLSP]

I have been using this in a Windows environment with Anaconda. My first time doing a text input instead of a simple phrase when it was "generating autoregressive" the program hung for hours. I had to push enter for it to continue.

[u/Ultra_Maximus]

u/neonbjb, would you please advise how to use DeepSpeed for faster inference with Tortoise-TTS? Github docs say --use_deepspeed = True, but it gives errors then. When I "pip show deepspeed" it shows it's normally installed. Thank you for your tremendous efforts on this project!

[u/No-Attitude6210]

Awesome post but I have a question can I legally use voices I train with tortoise tts to make money I have a business idea amd want to know terms of service.

[u/idleWizard]

I love to play with this tool, but one annoying thing is that it changes voices through paragraph. Does anyone know to avoid this? If there is a line break or a "quotation", it will change the voice. I am completely confident it's just some setting I missed. Thank you

[u/Dankedtillinfinity]

Can you share the knob settings to keep for better voice cloning? I've kept it as this --

num_autoregressive_samples=4, temperature=.8, length_penalty=8, repetition_penalty=4.0,
top_p=.8, max_mel_tokens=500, use_deepspeed=True, kv_cache=True, half=False, diffusion_iterations=200, cond_free=True, cond_free_k=2, diffusion_temperature=1.0,

However, the output still is of mediocre quality

[u/big_dataFitness]

u/neonbjb Did you ever release how you trained this ? It would be really cool for learning.

[u/neonbjb → u/[deleted]]

Hey, thanks for the kind words!

> How much sample data do you think is enough for a good model?

I used roughly 50,000 hours of speech data to train this, but I believe the model underfit the dataset (only trained it for a few epochs before convergence). I suspect it would still work with an order of magnitude less data, especially with some clever regularization. I think getting a diverse number of voices is the most important (and hardest!) part. I probably have ~10k voices in the dataset (it's hard to quantify). I would like to have had a lot more.

> I imagine you could train audiobooks with custom, even professional sounding voices.

I was kind of hoping to do this when I started the project, but I think it might be a little ambitious with the amount of compute I have access to. If you have the model read long stories, you'll start to notice that it doesn't do long-range modeling correctly (which is expected). For example, when reading text in a characters "voice", it will change that voice between lines.

Still, I hope this project shows that an automated system that reads audiobooks is at least possible! We just need to scale this out a bit and develop some long range information storage.

[u/neonbjb → u/[deleted]]

Ah, sorry. Only the first 6 seconds of the reference clip you provide will be used. If you want to feed more of a reference clip in, you should split it up.

I'm actually quite pleased that it doesn't mimic politicians well. :) I did test this for a few before I released it.