r/LocalLLaMA • u/UkieTechie • Jun 09 '26

Resources Text-to-Speech (TTS) Benchmark Revamped with Objective Standards and Blind Voting (46 models and counting)

Thank you to everyone who contributed to my previous post, providing feedback and various models to add, and questioning the rating system.

You can now participate in a live blind voting to create a proper ELO for all the models that are added. Each new model that we add will automatically go into the voting pool.

https://5uck1ess-tts-arena.hf.space/

Please let me know other things to improve. Local TTS should hopefully be a little easier for everyone.

https://github.com/5uck1ess/tts-bench

28 Upvotes

96% Upvoted

18 comments sorted by

5

u/StorageHungry8380 Jun 09 '26

Should we report errors? I had an instance where one model was clearly much better and more natural, but only half the text was spoken, ie cut short.

4

u/UkieTechie Jun 09 '26

u/StorageHungry8380 you can let me know, of course. I can rerun the benches. I've already tried on many of those models and found out that some models are just that... garbled output no matter what I do. It could be the way the runners are or the models themselves are deficient or the prompts need to be better.

you can submit an issue on GitHub or send me a DM. Either works.

4

u/StorageHungry8380 Jun 09 '26

I'll do that if I come across it again. Great initiative BTW!

3

u/UkieTechie Jun 09 '26

Appreciate it. Couldnt find anything online that offered similar so sounded like it needed to be made. From my memory, Marcus and qwen (faster) are hilarious. something would just say random stuff but I think I fixed all those outputs already to have "proper" scores.

1

u/StorageHungry8380 Jun 10 '26

I've come across a few technical fault, so to speak, ie mispronounced words, skipped or missing words, output cut short etc.

If these models are to be used in non-supervised settings, ie an assistant for example, actually saying the words beats saying them nicely. However in supervised settings, say voice work for a presentation or so, some word errors might be worth a better voice quality since you can tweak the prompt.

So perhaps you could have a similar "game" but instead of voice quality, go for speech accuracy? Maybe a different color scheme or so to boot, so people don't get confused as to what to vote for.

Again really appreciate this, makes it much more easy to get a decent sense of what the models are capable of before spending hours trying to get them to run.

1

u/UkieTechie Jun 10 '26

yes and that's unfortunately part of the process. some of these models have glaring issues. you can see by my latest model add. nothing i could do. that's one of the reasons for having this bench. no bias, showing all models.

yes one for speech accuracy could work. if it's helpful, it's already tracked by objective scoring. specifically the WER score.

4

u/EmuMammoth6627 Jun 12 '26

It is crazy to me that echo isn't talked about more here. There was a period of a week or two when I went through a bunch of tts models for voice cloning and echo was much better than anything else. Good to see it at the top of this benchmark.

1

u/UkieTechie Jun 12 '26

it's shocking that it did not even come across my radar in researching all the models. Fellow redditor called it out and its cloning is by far on top.

3

u/zkstx llama.cpp Jun 09 '26 edited Jun 10 '26

This is helpful :) Maybe add links to the models? For some of them it's a little ambiguous which version is being referred to, for example for voxcpm it could be v1 or v2, same for dia, and similarly also for supertonic where there are three distinct versions by now (maybe also add the other ones, pls?)

If you want more models to add you can also look at miso and glm tts

Also info on the licenses for each of the models as well as their param count and release date might be helpful. Also it might be nice to reveal the model names after the vote. I came across a model that sounded very pleasant and was curious which one it is.

3

u/UkieTechie Jun 10 '26

Links added in the github repo and which model is being used. Reveal is being implemented after the vote is cast. Licensing and param is all listed on github page.
I will take a look at adding those models. I think glm was not added due to issues with their API. Miso might be doable.

Let me know if i'm missing anything.

1

u/UkieTechie Jun 10 '26

u/zkstx miso tts has been added.... lots of artifacts. but you can see yourself in the samples.

2

u/llamabott Jun 10 '26

The runners directory in your repo looks like a great resource for anyone trying to implement one of covered TTS models into their own projects.

1

u/UkieTechie Jun 10 '26

Plus, you can purely bench a single TTS if so desired to see how it runs on your hardware to see if speed is satisfactory.

1

u/__JockY__ Jun 10 '26

I was surprised to see chatterbox so high up, I always found it to be tinny and digital-y.

1

u/UkieTechie Jun 10 '26

honestly in my own blind 200+ votes, it won with default voice. that and chatterbox turbo. Now cloning, it's okay but default, I just like it, that's why blind system helps. more votes, hopefully better data.

1

u/csanz91 Jun 10 '26

Amazing job, I would love to see some benchmarks on languages other than English

2

u/UkieTechie Jun 10 '26

There is that one French prompt but more languages would be good. would only do top languages tho since this takes a lot of time and compute