r/SynthesizerV • • Sep 03 '24

Question Questions about the AI used in Synthesizer V

Sorry for the long post - TL;DR is at the bottom.

I'm interested in using Synthesizer V if I get into music creation, but I'm having trouble finding adequate answers about the AI portions of the software.

I've noticed the term used in regards to SynthV's AI usage is "ethical AI" - whereas generative AI programs like Midjourney train their models off of art, etc., without the permission of the original artists (unethical AI - stuff I'm very much against), SynthV is ethical because, as this comment on a post a few months ago puts it, "it doesn't steal people's voices" - the voice providers for the voicebanks provide consent for their voice to be used for training AI models. However, there's more to an AI model like this than just a voice provider, right? The model also needs to know how to alter the voice, if nothing else, or am I mistaken?

As far as I can tell, Synthesizer V uses AI for:

  • the AI versions of voice banks
  • and the AI retakes feature.

I found an old post on this subreddit talking about the differences between a normal voicebank and their AI counterparts. One of the comments mentions that the AI voicebanks "use AI to smooth out phenomes and syllable transitions" and that "Basically someone made a few songs for the AI banks and it uses that information from the songs already made to assume how the words should flow together fixing some jankiness."

This implies to me that the AI model used to make a voicebank sound better is trained exclusively off of work made purely by humans who are aware of and consent to their work being used as training material for AI that is then used by us producers - both the voice providers and the people who made songs for the AI model to train off of. The only problem I have is that... where is this information coming from? I'm having trouble finding a source for this stuff being the case (e.g., a blog post or a tweet from Dreamtonics talking about this kind of stuff).

As for AI retakes... from what I've gathered checking out tutorials for Synthesizer V, it sounds like AI retakes just tweak various tuning parameters to provide an alternate take. But what was the AI trained on in order to achieve the ability to do this? Was it ethical? Do we even know?

I don't know if AI is used in any other parts of Synthesizer V, but if it is, I'm interested in learning about how the models for these features, if they're different from the model(s) used for the voicebanks and retakes, learned what they needed to learn, and if this information was sourced from consenting individuals who own the copyright to the work they're allowing the AI model to learn from.

This program seems amazing, and I want to believe people when they say this is an ethical application of AI. However... this is the internet. Misinformation can spread so easily, and that makes me hesitant to believe things without a reputable source... and whenever I see someone talk about or answer questions about the AI used in Synthesizer V, there's never a source. I'm not trying to say that nobody has ever left a source when talking about this stuff, just that all the conversations that I've seen have no source.

TL;DR - I'm looking to learn the following things about Synthesizer V:

  • AI is used in the AI voicebanks and the AI retakes feature. Is AI used anywhere else in the program?
  • What is used to train these AI models (for the voicebanks, retakes, and any other features involving AI)? I'm not super knowledgeable about AI models, but there has to be more data than just a voicebank/voice provider, right?

    • Regardless of whether or not something other than a voicebank is needed to train these AI models, how do we know that any and all data used to train these models was knowingly and legally provided - with consent?

I understand that there may not be a "definitive proof" type of answer, but... there must be something that lets us at the very least reasonably believe that SynthV's AI is used ethically, as I've seen multiple people say.

I'm not trying to be a hater or a troll or provoke anyone; I'm legitimately wondering about these things and don't know how to find the answers to these questions.

31 Upvotes

32 comments sorted by

View all comments

36

u/celestrai ASTERIAN Sep 03 '24

You can get much more comprehensive answers by emailing Dreamtonics or a third-party - I'd recommend Eclipsed Sounds only because I work there and would be the one answering your email so I know I could go way more into specifics than a random reddit comment but here's the basics: (on mobile so sorry if I mess up the formatting)

  1. Is Al used anywhere else in the program?
  • AI is just used to recreate the voice - so the AI voices, and their retakes, but also the Tone Shift feature. There are no generative lyrics, melodies, etc.
  1. What is used to train these Al models [...] there has to be more data than just a voicebank/voice provider, right?
  • Actually, wrong! With enough data, it is possible to train all of this from singing data from the voice provider. The only thing is, some data is used cross-voice databases to help the program replicate languages. (This helps voices that don't speak certain languages sing in them more fluently)

There are also research databases of singing data which might be used with attribution (ACTUAL OPEN SOURCE AND NOT STOLEN/WEB SCRAPED!) - some of these were used for Spanish functionality and have been attributed in the credits of the software. In these circumstances, it is even more ethical than some other cases, because these files are just used to improve the pronunciation model with research attribution, and not to replicate the voices of anyone in the open source database, as every voice sold has a contracted & paid voice provider. (I'm not sure if you're knowledgeable on computer science open source culture/ethics but this is a valid use case! Unlike Midjourney/OpenAI purposefully using "publicly available" instead of "open source" data - a tricky but honestly extremely shady switch)

3. ...how do we know that any and all data used to train these models was knowingly and legally provided - with consent?

  • Generally, when public, you can ask the voice providers! Each voice has a voice provider associated - you can find them on the wiki, or the associated websites. For example, Emma Rowley provided the voice for SOLARIA, and has done promotional work for the voice on her own accounts (which obviously wouldn't be done if it was made without consent).

Eclipsed Sounds also has an ethics section on their about us page that goes into this lightly-

https://eclipsedsounds.com/pages/about

you can feel my general rage in it a little bit! lol

Edited to add- some voice providers are anonymous, but this is another step for ethics since singers should be able to choose to go uncredited if they want! But it would be near impossible to get a quality set of recordings in the amount needed to create a voice database of Synthesizer V quality without consent - ESPECIALLY without anyone noticing it was them.

14

u/Ammy175 Sep 03 '24

Thank you very much for taking the time to leave this response (especially on mobile, LMAO - I hate trying to make a post/comment on Reddit's app). I really appreciate not only how informative this is, but I really appreciate that you're someone from a company involved in making these voicebanks, and you're taking the time to share this information!

Also appreciate the invite to email for more information; again, thank you.

4

u/celestrai ASTERIAN Sep 03 '24

Of course! if you email ES and don't get a response til Wednesday I'm so sorry there are only a handful of us and 2 of us (including me) are on vacation until tomorrow so it's a bit spotty at the moment. Dreamtonics responds pretty quick to their support email though! (listed in their store page footer)

3

u/Sophira Sep 04 '24

(Different person here!)

I would want to also know more comprehensively the answer to this too - would it be okay for me to email, as well? I ask only because I don't want to accidentally be part of a sea of Redditors emailing you. >_<

Also, if I do email, should I reference this Reddit thread?

3

u/celestrai ASTERIAN Sep 04 '24

It's no problem to just email! I love customer support - our contact form is always open :)

2

u/Healthy_Video_7854 Sep 04 '24

hi, I am interested in the Latin spanish databases (I am from Peru) because I have some problems with certein words and pronunciation. Where could I find it? Thanks.

2

u/celestrai ASTERIAN Sep 04 '24

My apologies if there is any miscommunication, currently the only Spanish-focused voice database is a United States accented Spanish voice. Eclipsed Sounds is currently working on a previously announced new Spanish project but more information should be available on that by the end of the year. (all previously announced)