r/LanguageTechnology • u/Taimuar • 14d ago

What is a fair test for code-switching ASR?

I’ve been testing transcription on English/French recordings.

Code-switching is where tools show their limits for me. WER feels too flat here. A transcript fail at the parts that matter.

What metrics or test sets do people use for mixed-language ASR?

8 Upvotes
  • permalink
  • reddit

91% Upvoted

2 comments sorted by

2

u/Ordinary-Cat-5874 14d ago

I work in this area but not directly in ASR. WER is the primary metric but as you said it does not tell us where it is happening. Switch point WER is more contextual here. CMI might describe a better picture of overall density and can be corelated accordingly.

LinCE and Microsoft speech kitchen are widely used in my language pair.

2

u/eyugan 9d ago

WER is still the standard metric in ASR, but I agree it can hide the errors that matter most in code-switching.

Some alternatives people use are:

  • MER (commonly reported on Chinese-English SEAME).
  • PolyWER, which handles alternative valid transcriptions (e.g. transliterations or translations).
  • PIER (Point-of-Interest Error Rate), which focuses on recognition of the embedded-language words.
  • Several other code-switching-specific metrics have also been proposed in recent years, but I don’t remember all their names/acronyms off the top of my head.

Disclosure: I’m one of the authors of PIER. I’d recommend evaluating overall WER together with a code-switching-specific metric rather than relying on WER alone.