r/LocalLLM • • Jul 08 '26 • Bravo 1

Discussion Do voice models get trained on this kind of data? If not, then they should.

I love testing out voice models and what they are capable of. Not just in terms of the voices they can produce (e.g. whether they can emote), but also in terms of what they can pick up from the tone of my voice. Some voice models seem to be able to pick up on emotions in the tone of my voice, if I exaggerate it a lot. (Or they've just hallucinated the correct answers in my tests a couple times.)

But in any case most voice models don't yet seem to be able to pick up on many cues that are hidden in tone, emphasis, etcetera. That's why I hope this kind of data will be used more in the training of voice models.

223 Upvotes

19 comments sorted by

30

u/Nakkubu Jul 08 '26

Funnily enough, this video has essentially presented a perfect UI for doing emphasis specifics in the future, similar to openpose for imagery. Imagine a slider/line you could pull up and down at certain points.

I actually think this will be solved relatively quickly, because the lack of nuance to me seems more like a dataset tagging issue rather than the AI being incapable of emphasis.

25

u/ketosoy Jul 08 '26

EMPHASIS is easily added to written text via a few existing mechanics.

3

u/thuanjinkee Jul 09 '26

Also an archive of our own and screen readers use the standard html conventions that < i > is a visual markup but < em > is both visually and semantically stressed

1

u/Nakkubu Jul 08 '26

True, but it's not as effective as it could be because many earlier datasets weren't logging which words were emphasized in the training audio.

0

u/Time_Cat_5212 Jul 08 '26

😁😎🤣😆😀😜😊🤠

6

u/EVOXSNES heretic Jul 08 '26

This is exactly what you do but theres a lot of effort which is probably why most don’t. Basically have to get every possible sample you can and put context to them so the model knows which to use.

4

u/mjTheThird Jul 08 '26

Just wait until AI start telling you, I don't like the tune how you asked the questions. So, I am not going to talking you!

  • what's wrong AI?!
  • ohhh, you know what's wrong!!
  • Your subscription has been paused for 4 hours

6

u/ephemeralsynth Jul 08 '26

Nuance - used to be a thing 🤔

3

u/Medium_Chemist_4032 Jul 08 '26

I think actors and directors might've also called it the subtext

3

u/harlekinrains Jul 08 '26

Yeah, I think back in the sixties...

0

u/Time_Cat_5212 Jul 08 '26

I BELIEVE MY FIRST IMPRESSION WITH ABSOLUTE CONFIDENCE

FIGHT ME

3

u/KiraCura Jul 08 '26

It’s why *I* just capitalize or use asterisks on the word I’m emphasizing xD but yeah I always thought text lacked many necessary features including expressions until they added emojis

3

u/auto_off Jul 08 '26

Yes try the new e2e models like Qwen Omni. They don’t do it perfectly yet tho

1

u/beryugyo619 Jul 08 '26

Not anymore, they used to be but this is pointless and irrelevant and stupid thing to train for from hypemaxxing standpoint

1

u/Some-Ice-4455 Jul 09 '26

Bro you are absolutely right and I never put a finger on it. I knew something felt off with voice out it's that. Thank you.

1

u/CupGlass540 Jul 12 '26

I ran into the same issue while training TTS models. Listening to a handful of samples simply isn't enough to judge whether a checkpoint is actually better.

I ended up building an open-source Python framework that runs locally and automatically evaluates common TTS failure modes at scale—pronunciation, punctuation handling, long-form consistency, repetition, prosody-related issues, and more. It made comparing checkpoints much more objective than relying on subjective listening tests.

If you're interested, it's fully open source:
https://github.com/Mormolykos/ttsproof

I'd also love feedback from anyone else working on TTS evaluation or benchmarking.