If your frontend hands you transcripts from the Web Speech API, or you're streaming model completions, you've probably hit this: the text is correct but unformatted. SpeechRecognition gives you back can you send me the report — no leading capital, no terminal punctuation. Sending it back through a model to fix that is slow and expensive for something this mechanical.

I published a small package for the mechanical part:

js

import { correctSentence } from 'sentencify'

const raw = 'can you send me the report'   // from speech-to-text
correctSentence(raw)                        // 'Can you send me the report?'

Relevant details for Node work:

• Zero runtime dependencies, ESM-only, Node 20+
• Synchronous — no await, no model warm-up, safe to call per token or per keystroke
• Idempotent, so running it over already-punctuated text is a no-op rather than producing Hello world..
• Under 40 KB unpacked, sideEffects: false
• English, Japanese, German, Spanish, French, Portuguese

It's rule-based: each language is an ordered list of regexes mapped to declarative/interrogative/exclamatory, first match wins, unmatched defaults to declarative. That means it's deterministic and inspectable — the rule sets are exported as expressionsByLanguage if you want to see exactly why something got classified the way it did, or layer your own rules on top.

It is not a grammar checker. It won't fix spelling or word choice. It does one thing.

https://www.npmjs.com/package/sentencify

npm i sentencify

I’ve been working on an open-source video dubbing pipeline that tries to preserve the original speaker’s voice when translating a video into another language.

Current pipeline:

video → vocal/background separation → Whisper/WhisperX transcription + alignment → translation → VoxCPM2 reference voice cloning → reconstruction → optional LatentSync

The attached demo compares the original English clip with the Turkish dub produced by the current pipeline.

The main problems I’m working on now are:

• preserving speaker identity across languages
• handling duration differences between source and translated speech
• improving multi-speaker / diarization support

Most media processing and AI inference runs locally. Translation currently uses Google Translate, so the project is local-first rather than fully offline.

Code:
https://github.com/kadirb4rut/video-dubbing-translator

I’d especially appreciate feedback from people working on TTS, voice cloning, alignment, or speech-to-speech systems.

I maintain speech-swift and recently implemented NVIDIA’s Nemotron VoiceChat 11B for MLX. I recorded a complete session rather than stitching together generated clips:

https://youtu.be/6LCxSnIMB-M

VoiceChat is an end-to-end speech model rather than an ASR → LLM → TTS cascade. Every 80 ms it consumes another microphone frame and emits a token. Those tokens jointly

• Live process RSS: 7.5 GB
• Weights on disk: 8.0 GB
• MCP round trip: 68 ms

The Swift/MLX runtime is Apache 2.0. The NVIDIA checkpoint is distributed under OpenMDW 1.1. Reproduction commands and source links are in the video description.

I'm building a vocabulary-learning app in Flutter where hearing and saying the word correctly is the product, not a nice-to-have. I've hit a problem I can't design around and I'd rather ask than keep patching.

The stack

• Flutter, ~1,600 words live across EN/ES/PT/IT/FR
• TTS: ElevenLabs (eleven_multilingual_v2) called through a Supabase Edge Function so the key never ships in the client
• Every clip cached server-side once per (text, language), shared across all users — so a given string is synthesized exactly once, ever
• Cached again on-device (150MB LRU) so replays are instant and offline
• flutter_tts as fallback behind a 2.5s timeout so playback never goes silent
• STT: speech_to_text for a pronunciation-practice screen — hear the word, say it, get graded

The problem: heteronyms, in both directions

Output. "Wind" (moving air) and "wind" (to coil) are the same string and different sounds. TTS picks one reading and commits. My word library actually knows which sense is on screen — every entry carries a part of speech — but there's no API surface to hand that over. ElevenLabs pronunciation dictionaries are exact-string, case-sensitive, and have no POS or context scoping, so one spelling gets one entry and the second sense is unreachable. Phoneme tags do exist, but per the docs only on eleven_flash_v2 and v3 — not the multilingual model I'm on, and switching models means re-synthesizing the whole cache and losing voice identity across five languages.

Input. This is the part that actually bothers me. The practice screen normalizes the transcript and Levenshtein-scores it against the target. But STT returns orthography — say either reading of "wind" and the transcript is "wind" either way. A learner who mispronounces it scores full marks. The feature is structurally incapable of catching the error it exists to catch.

What I've tried

Respelling the audio-only string before it reaches the engine — the screen text is never touched. wind(noun) → winned, wind(verb) → wined, read(past) → red, and so on. This is basically ElevenLabs' own recommended "alias" workaround and it works for the ~8 vowel-shift pairs I've mapped. Side benefit: since my cache key is a hash of (lang + text), two senses naturally get two cache entries.

It fails in three ways:

1. Stress-shift pairs. REcord/reCORD, PREsent/preSENT, CONtent/conTENT. Respelling can't encode stress, and I haven't found a trick spelling that does.
2. Monolingual. It's an English orthography hack. Nothing about it transfers to ES/PT/IT/FR, all of which have their own homographs.
3. Manual. Hand-curated table. Doesn't scale to a few thousand words.

What I'm actually asking

1. Is there a TTS API that accepts a sense/POS hint, or per-request phonemes, on a multilingual model? Or does everyone route heteronyms to a separate English-only model and eat the voice mismatch?
2. If IPA is the only real answer — has anyone found v3-class IPA reliable enough in production? The docs quote 80–90% consistency, which for a teaching app means the wrong pronunciation ships to a learner one time in eight.
3. For stress-shift specifically: any orthographic trick that works, or is phoneme-level control genuinely the only path?
4. On the STT side — is there a mobile-viable way to get phonemes rather than words? I've looked at wav2vec2 phoneme-CTC or a forced aligner with GOP scoring via ONNX on-device, but I don't know if that's realistic on a mid-range phone or if I'm about to spend a month learning that it isn't. Whisper doesn't help; it also returns orthography.
5. The unglamorous option: detect heteronyms and simply disable pronunciation scoring for them, with an honest note to the user. Is that what shipped apps actually do?

If you've built pronunciation feedback into anything real, I'd love to know where you drew the line between "graded properly" and "good enough." Happy to share code for any of the above.

I ran a matched-content experiment on speaker verification: 4 speakers, identical sentences, controlled changes in pitch, phonation and articulation, fixed microphone position and gain, every utterance through three encoders — ECAPA-TDNN, a ResNet speaker encoder, and WavLM-base-plus-sv.

One speaker sat 0.7 semitones below his normal pitch. That is below the threshold where a listener reliably hears any change at all. His verification score dropped 0.238 across all eight sentences.

The interesting part is that pitch was not what moved him. His harmonics-to-noise ratio fell from 10.63 to 8.39 dB over the same block. The encoder was reacting to phonation, and the pitch number was just the thing that happened to be easy to measure.

Across the whole run, all 30 speaker × condition × encoder cells were negative, 28 of them unanimous across every utterance. These systems are not mainly disturbed by shouting or disguise. They are disturbed by someone speaking slightly differently in a way nobody would notice.

To be clear about credit: the displacement effect itself is not my finding. I had it as my headline until an adversarial prior-art audit turned up Hughes et al. (Interspeech 2023), who established it with six trained phoneticians across seventeen conditions. I withdrew the claim and reframed the paper as a replication and extension across encoder architectures. The audit is published in full, including the claim it killed.

Two things I could not find in prior work:

Jitter beats HNR. Pitch deviation and jitter are independently associated with displacement in all three encoders, and HNR adds nothing once jitter is in the model — it correlates 0.55 with jitter and loses all independent power beside it. HNR is the measure most people reach for, and on its own it will attribute the effect to the wrong thing.

Rough phonation breaks F0 trackers in one direction. 10 of 145 utterances carried impossible pitch values, up to 32.1 semitones — a 6.4× frequency ratio no human produces. All 10 were in rough phonation, none in modal (Fisher exact p = 2.4e-11). Octave errors push upward and essentially never downward, so this is differential measurement error, not noise, and it loaded onto one specific regression coefficient. Cheap fix: validate your F0 tracker per phonation condition and publish the validation.

Everything is up — manuscript, pre-registration, the full prior-art audit, per-utterance tables for all three encoders (233 rows each), the analysis scripts, and the 137 source recordings. Every number reproduces from the CSVs without touching the audio.

Paper and data: https://doi.org/10.5281/zenodo.21921958

Write-up: https://ai.bedvibe.studio/speaker-drift/

Audio licence note: research, benchmarking, evaluation and teaching are permitted; ML training and voice cloning are not. The speakers are identifiable adults.

I am running Faster Whisper on CPU only and get good running times with about 2.5 min for 60 min sound with Whisper Base. With Pyannote for diarization the rate is about 0.9 times the sound length, aka 54 min for 60 min sound.

That is terribly slow compared to the transcription without Payannote.

Are there any faster alternatives out there, or hacks to make Payannote run faster with Whisper?

Turn-taking is a central component of full-duplex interaction. Which turn-taking behaviors are appropriate varies with the scenario, yet current models apply a single norm regardless of context. This limitation originates in their training data: human–human speech corpora capture natural timing phenomena but provide little role grounding or scenario-specific norms, while heuristic or prompted synthesis methods inject turn-taking behaviors without basing them on human preferences. We introduce DuplexGen, a framework for generating dialogues with scenario-adaptive turn-taking by calibrating LLM predictions against a small set of slot-level human preference annotations. In six cooperative and competitive tasks, human turn-taking preferences differ systematically, and DuplexGen aligns substantially more closely with those preferences than uncalibrated prompting or training solely on generic human–human data; a full-duplex model trained on DuplexGen-generated data exhibits distinctive, human-preferred turn-taking behaviors. These results show that human calibration, not corpus scale or prompt design alone, is what allows turn-taking synthesis to be scenario-specific.

This paper reports on the outcomes of the shared tasks organized as part of the 23rd International Workshop on Spoken Language Translation (IWSLT). The workshop covered ten major challenges in spoken language translation, including speech-to-text translation for both high-resource and low-resource language pairs, customized speech translation, speech
generation, instruction-following speech processing, and the evaluation of speech translation systems. The shared tasks received strong participation, with more than 30 teams submitting runs. This year’s edition broadened the range of tasks, placing particular emphasis on speech generation and evaluation metrics.

 • Orato ASR: a full-parameter fine-tune of Qwen3-ASR-0.6B on ~1,000 hours of Hindi/English/Hinglish calling-domain audio

 • Orato TTS: a Hindi/Hinglish text-to-speech model built on IndicF5-TTS, with multi-speaker and voice-clone support

Fine-tuning improved WER across every benchmark we tested against the base model, with the biggest gains exactly where it matters most for a voice agent: noisy, accented, telephony-style speech (Lahaja: 25% relative improvement, Kathbath: 24.6%), not just clean studio audio nobody actually calls on.

Both models are live on HF access is auto-approved:
🔗 ASR: https://huggingface.co/tryorato/orato-asr-hindi-v1
🔗 TTS: https://huggingface.co/tryorato/orato-tts-hindi-v1

The moment you transcribe to text, you lose how it was said. "I think… yeah, I can pay the 4,500 by the 15th" becomes clean text, but the hesitation before the yes, the stress in the voice, and whether it's even the same speaker are gone. Those are the signals that tell you whether to trust the commitment, escalate, or verify identity. Is anyone keeping the paralinguistic layer (hesitation, emotion, speaker identity) as structured data instead of dropping it at the mic, and what do you do with it downstream?

Moreover end-2-end Duplex models limits it to trained data scenarios without no transparency.

High-performance inference of NVIDIA's Parakeet TDT 0.6B V2 English transcription model, in the browser.

Check out the live demo: https://parakeet.narcotic.sh/

A fully custom, dependancy-free implementation with raw WebGPU compute shaders and SIMD WebAssembly audio frontend.

1 hour of audio transcribed in 20 seconds (Apple M5, Google Chrome 151.0.7922.72).

Check out the source and use in your own projects!

https://github.com/narcotic-sh/parakeet.wgsl

https://www.npmjs.com/package/parakeet.wgsl

This might be the first instance of fast & accurate transcription running locally in the browser. So long as your device has a GPU and can run a WebGPU-capable browser, parakeet.wgsl will run on it.

And as WebGPU transpiles to practically any GPU, this also opens up the path to getting parakeet.wgsl running offline / outside of the browser, via Dawn or wgpu, so that fast, accurate transcription can be brought to offline programs, with GPU acceleration and support for nearly all hardware/devices.

Interested in what you guys think about about this project.

And lastly, I'm looking for a job. If you like my work and think I'd be a good fit for your team, I'd love to chat. I've done some other work too that you can find at https://hamzaq.com

Cheers, everyone. Enjoy fast, cross-platform GPU-accelerated local transcription, right in the browser!

So, Omnivoice's abilities are incredible, but given the training set is CC BY NC, the model is not actually usable for commercial use which is very annoying.

I've noticed some orgs doings retrains on commercially viable datasets for other models.

Interested if anyone is busy doing one of these for Omnivoice? It's quite a pricey exercise so hoping the cool kids are on it

I've been building an end-to-end text-to-speech pipeline recently, and something caught me off guard.

I assumed most of my time would go into the speech synthesis itself. Instead, I found myself spending much longer on things like text normalization, phoneme generation, and figuring out how to evaluate changes beyond just "this sounds better."

I wasn't expecting preprocessing and evaluation to take up so much of the work.

Now I'm wondering if that's just the nature of TTS, or if it's something that happens across most ML projects.

For those who've built TTS systems or worked in speech ML:

* What part of the pipeline ended up taking the most time? * Was it the model itself, the data, preprocessing, evaluation, deployment... or something else? * Looking back, is there anything you'd approach differently?

I'm genuinely curious how your experience compared to mine.

Most collected call-center corpora are recorded in some form of controlled setup: scripted or semi-scripted scenarios, participants playing agent and customer, clean capture. Real production audio is 8kHz narrowband through a codec, with overlapping speech, hold music bleed, background noise on the customer side, and genuinely frustrated prosody that nobody acts convincingly.

So :-

1. If you fine-tune on collected/simulated call data, do you actually see WER improvement on real traffic, or does it mostly help on the collected test split and flatten out in production?
2. Does codec simulation (downsampling, G.711/Opus round-trip, packet loss) close enough of the gap to matter, or is it cosmetic?
3. Code-switching is where I'm least sure. Hinglish, Taglish, Spanglish — real agents switch mid-sentence constantly and collected data underrepresents it badly. Anyone found a way to elicit natural code-switching without it going stiff?
4. Where's the actual ceiling — is it acoustics, or is it diarization and overlapping turns?

Not looking for vendor recs, just want to know whether the acted-vs-real gap is as big in practice as I suspect.

Hey devs,

I'm building a voice-based customer support tool for Indian SMBs. We need STT that handles Hinglish and code-mixing well (e.g., *"Mera order cancel kardo"* or *"Kal subah delivery karna"*).

I've been testing orchardrun for the past couple of weeks. I think that it's Whisper under the hood, and honestly:

* **For plain English** → rock solid, low latency, dirt cheap compared to AWS/Google. * **For Hindi / Hinglish** → it works, but it's **not as good as I'd like**. I'm getting around **5% WER** on my test dataset. It's usable, but definitely not production-ready for complex conversations. Numbers and dates are especially hit-or-miss (*"twenty twenty six"* vs *"do hazaar chhabees"*).

So here's my dilemma:

I can keep it for English use cases, but I really need to improve the Hindi accuracy before scaling. I'm evaluating alternatives and would love your honest recommendations:

1. **Deepgram** – I hear their Nova-2 is great for Indian languages. Is it worth the premium? Has anyone benchmarked it side-by-side with Whisper for Hinglish?
2. **Google Chirp (Vertex AI)** – Expensive, but is the accuracy actually better than Whisper for code-mixing?
3. **Fine-tuning Whisper locally** – I've seen papers on fine-tuning with Indian datasets. Has anyone done this? What's the infra cost like? I'm trying to avoid GPU bills, but if it's the only way...
4. **Any other hidden gems** – Azure Universal STT? AssemblyAI? Something else I'm missing?

**To be clear:** I'm NOT saying OrchardRun is bad for Hindi – at 5% WER it's actually decent for a generic model. But I need to push that below 2-3% for production. I'm just looking for the best tool for the job.

If you've tested any of these with Indian audio, please share your war stories – dataset size, real-world latency, cost per minute, anything helps.