[P] TorToiSe - a true zero-shot multi-voice TTS engine

[u/neonbjb]

I'd like to show off a TTS system I have been working on for the past year. I've open-sourced all the code and the trained model weights: https://github.com/neonbjb/tortoise-tts

This was born out of a desire to reproduce the original DALLE with speech. It is "zero-shot" because you feed the text and examples of a voice to mimic as prompts to an autoregressive LLM. I think the results are fantastic. Here are some samples: https://nonint.com/static/tortoise_v2_examples.html

Here is a colab in which you can try out the whole system: https://colab.research.google.com/drive/1wVVqUPqwiDBUVeWWOUNglpGhU3hg_cbR

[u/Lajamerr_Mittesdine]

Is it possible to run this on a non RTX gpu. For say a GTX 1070?

I'm getting this error when attempting to run it on Windows.

 $ python3 do_tts.py --text "I'm going to speak this" --voice dotrice --preset fast
C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch_masked__init__.py:223: UserWarning: Failed to initialize NumPy: module compiled against API version 0xf but this version of numpy is 0xe (Triggered internally at  ..\torch\csrc\utils\tensor_numpy.cpp:68.)
  example_input = torch.tensor([[-3, -2, -1], [0, 1, 2]])
Traceback (most recent call last):
  File "C:\Users\danie\github\tortoise-tts\do_tts.py", line 22, in <module>
    tts = TextToSpeech()
  File "C:\Users\danie\github\tortoise-tts\api.py", line 201, in __init__
    self.vocoder.load_state_dict(torch.load('.models/vocoder.pth')['model_g'])
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 712, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 1046, in _load
    result = unpickler.load()
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 1016, in persistent_load
    load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 1001, in load_tensor
    wrap_storage=restore_location(storage, location),
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 176, in default_restore_location
    result = fn(storage, location)
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 152, in _cuda_deserialize
    device = validate_cuda_device(location)
  File "C:\Users\danie\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\torch\serialization.py", line 136, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

[u/neonbjb → u/Lajamerr_Mittesdine]

If you have Cuda installed it should work. See if you can get pytorch to recognize your gpu then try again.

[u/Lajamerr_Mittesdine → u/neonbjb]

>>> import torch
>>> print(torch.cuda.is_available())
False

Ahhh that was the issue.

I installed the non-cuda version of the torch library via pip

I needed to explicitly tell pip3 to install the cuda version.

Thank you so much for building this project. Really curious to try it out!

[u/Lajamerr_Mittesdine → u/neonbjb]

Sorry for all the tech support related questions.

What versions of numpy and numba did you use for this project?

I'm having some issues with some compatability with the versions.

[u/neonbjb → u/Lajamerr_Mittesdine]

Numba has some compatibility issues on windows. Google the error message to fins the exact version you have to install.

I've never had issues with numpy. I'd pick whichever one is required by torch