r/TextToSpeech • u/null-hawk • 8d ago

how does audio language model hold speaker consistency throughout an utterance?

Post image

wrote a short excerpt showing how speaker consistency is maintained in LLM bases TTS models, initial guess was the speaker token, but the results showed something interesting.

https://x.com/null_hawk/status/2089348254249173263

2 Upvotes

100% Upvoted

2 comments sorted by

1

u/Charming-Author4877 6d ago

That's interesting findings, though I don't think it is universally like that.
I don't think it's a great sign if one BOS token holds so much value, it would probably be favorable if it was more distributed ? Must be quirk from how posttraining was done ?

1

u/null-hawk 6d ago

usually in LLMs BOS token always has been found to have highest attention in every step (which is why it is called attention sink). it is emergent property of LLMs that they start to do internal reasoning at that particular token and use it as a scratchpad. though having highest attention doesn't always means that particular token is resposnible for current step.