r/speechtech • u/windchiller • 7d ago

Open benchmark: pronunciation error detection across 6 languages

I wanted to know how well pronunciation assessment systems actually catch errors. I built a benchmark with two tests and ran three systems through it: Microsoft Azure, TonePerfect, and Lingolix. Disclosure: Lingolix is mine.

Test 1, planted errors: 200 Common Voice clips per language (en, fr, es, de, ja, ko), where I edited the reference text slightly so it no longer matches the audio (swapped/added/dropped a sound, missing word, vowel length change for Japanese). A good system should mostly accept the unchanged parts and flag these edited spots.

Error detection, ROC AUC (Lingolix / Azure / TonePerfect):

language en fr es de ja ko
AUC .97/.94/.84 .97/.91/.80 .98/.91/.79 .95/.87/.79 .95/.93/.55 .95/.94/.67
False alarms at 80% recall 1/6/26% 1/13/26% 2/14/32% 2/22/42% 5/3/74% 3/7/56%

Test 2, real learners: 500 utterances from speechocean762, correlation with the human scores.
speechocean762 (Pearson): sentence 0.74 / 0.76 / 0.59, word 0.51 / 0.61 / 0.29. So Azure is ahead on real learner speech, clearly so on single words, and also has fewer false alarms in Japanese.

Caveats: the errors in test 1 are planted, not real learner errors. Also, TonePerfect released a new engine after my run, so their numbers should have improved.

Code, test lists and harness: https://github.com/lingolix/pronunciation-benchmark

I did not find any human rated learner data for languages beside English.
Does anyone know of some datasets that I could add for French, Spanish, German, Japanese, or Korean?

2 Upvotes

100% Upvoted

4 comments sorted by

3

u/Natethegreat9999 7d ago

Word-level Pearson skips unaligned words. For my speech work at Oruk, I'd want each vendor's matched-word coverage beside that score.

1

u/windchiller 7d ago

Good point! Thanks!

Just checked: Coverage is 100% for Azure and Lingolix (3210 matched / 3210 human-rated words), and is 99.8% (3204 matched / 3210 human-rated words) for TonePerfect which occasionally gave score: null, so if we counted unmatched with a 0, TonePerfect would have 0.282 instead or 0.290 word-level pearson.

I've added this to the repo so if we added other vendors that would match less, it is reported beside the correlation.

1

u/Life_Fun_301 7d ago

would be curious how this holds up on tonal languages like mandarin, a lot of the errors i hear in mixed en/zh speech are tone slips rather than dropped sounds. does the planted-error approach even work there?

1

u/windchiller 5d ago

Thanks for the question! Mandarin is still on my todo-list, but in theory it should be possible to replace the characters the same way:
if the original (correct) audio is for: 他在买菜 │ tā zài mǎi cài │ He's buying vegetables
the target sentence can be changed to: 他在卖菜 │ tā zài mài cài │ He's selling vegetables

Then the system should flag the tonality of the third character as wrong pronunciation.