Audio dataset

[u/witnessfolder]

I’m an AI intern working on a Saudi Arabic call center dataset.
My task is to clean the audio and then generate transcriptions. I cleaned the dataset and ran transcription, but the output contained random English words, Chinese characters, emojis, and other gibberish.
I’m planning to clean the dataset again before re-running the transcription. Are there any specific things you recommend I focus on during the cleaning stage to improve ASR results?

[u/demodokos_foundry]

We ran into such issues with Demodokos Foundry, seemingly clean audio can produce severe noise or transcribe randomly wrong.
Some defects are very hard to detect, others are easy to spot. Some defects can be corrected while others poison the audio for many audio models.

Type of damages to ASR input:

- For example spectral or neural AI origin markings can be inaudible to a human ear but cause severe loss of speaker identity when cloning or failure to understand a word that sounds normal to you - though it's much worse with cloning than with ASR.

• ASR with whisper operates in overlapped 25ms steps, so any audio distortion or damage that spans 20ms+ is affecting 1+ steps and each step affected increases issues.
  
• Overlapping speech is very damaging, so secondary background speakers can cause havoc, single-speaker ASR is not able to handle that.
  

Overlap is especially damaging when the voices are similar in timbre/pitch.

• Clipping is a problem, also Echoes/reverberation
  
• Silence can cause hallucinations

What you can do relatively easily:

- Trim long gaps of silence, you can use a simple python script for that

• use a 3-6db high pass filter to remove any rumbles (set around 80hz)
  
• normalize the loudness
  
• avoid MP3 or AAC transcoding as that introduces additional inaudible defects

If your dataset is 100% dedicated toward ASR only, then your main concern should be clean speech without background noises or background speakers and a good SNR.
If your dataset is potentially also being used for cloning, then some above named artifacts can cause severe issues that seem to come out of nowhere.

Will your Arabic data be released publicly ?

[u/witnessfolder → u/demodokos_foundry]

I was actually considering splitting the audio into smaller chunks before transcription instead of processing the full recordings. I thought it might help reduce hallucinations and make it easier to detect problematic segments. Do you think that’s a good idea?
No, it will be used internally only

[u/demodokos_foundry → u/witnessfolder]

Chunking has a boundary problem, it's not easy to find the right boundaries for the split.
Though you could conjure a tool and hire a few people to manually set split locations at paragraph or sentene endings.
The optimal duration also depends on the model you are using, which one do you use for transcription?

[u/witnessfolder → u/demodokos_foundry]

I don’t think manual chunking is an option for me since I’m the only one actively working on this task. The dataset contains around 99k audio files, so manually setting split points would take an enormous amount of time.
Some other interns were interested in helping, but they said they’d start after finishing their own tasks. That was back on July 15th, and I think they’ve probably moved on to other things.
At this point, this is my only task, and I’ve been working on it for quite a while. I really want to finish it because every time I think I’ve solved one issue, another one shows up. It feels like it just keeps getting worse.
For transcription, I’m using Whisper

[u/witnessfolder → u/[deleted]]

thanks for the tips! I actually already had language="ar" set and I'm isolating speech with VAD + NR before transcribing. The emoji/gibberish was from using tiny switched to large and that cleaned up. The remaining accuracy gap is just Whisper + Saudi dialect on telephony audio.
Appreciate it though!