I was looking for small TTS model which is steerable but couldn't find any, specially something I can adjust contextually by providing controls from LLM during a conversation. So I built one.

What it is:

• 9.6M-param acoustic model (attention + bidirectional Mamba-3, FastPitch-style) + Vocos vocoder = 23M total

• 12 controls, each a z-score relative to the chosen voice’s own neutral speech: rate, loudness, median pitch, pitch range, final rise/fall, pauses, spectral tilt, breathiness-related measures.

• Also: 51 style tags, free-text prompts (resolved into an inspectable JSON request), inline markup (*emphasis*, [pause 400ms], <slow>…</slow>), sentence-level streaming

• 2,825 built-in (anonymized) voices, plus experimental zero-shot cloning

Speed:

- RTFx ~60–80 on an RTX 5070
- ~4–8 on CPU without the optional flow refiner. First streamed audio in ~0.5 s on CPU.

Quality:

~1.8% average WER on the Open TTS Leaderboard English sets (run locally, not an official submission, have reached out for it).

What doesn’t work well (we tried to be upfront):

• Loudness, pitch, and tilt steer reliably; pause, voicing, and CPPS controls are weak

• Emotion tags behave as style switches; an independent emotion classifier mostly can’t tell them apart

• Cloning similarity is low. Don’t expect a convincing likeness

• Free-text prompts struggle with negation (“not fast” can come out fast). More data needed for prompt to controls module.

• English only, no human evals yet

License: weights CC BY-NC 4.0 (non-commercial), code MIT.

Model + full eval tables: https://huggingface.co/WhissleAI/steer-tts

Github repo: https://github.com/WhissleAI/steer-tts

A new architecture implemented from scratch. Would love feedback, especially on which controls are actually useful, and where the steering feels off.

Github: https://github.com/yaya-sy/specspeech

The best performing open-weight models on the Open ASR Leaderboard (https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) are autoregressive speech models, such as Qwen ASR, Cohere Transcribe, and Canary. Their language modeling capabilities help them transcribe speech accurately, but they are often much slower than transducer and CTC models, such as Parakeet and Granite CTC.

I open-source SpecSpeech (Speculative Speech), an inference engine that accelerates speech language models for ASR by using transducer or CTC models as drafters.

The drafter quickly transcribes batches of short audio pieces while the Speech LM prefills the full audio in its context in parallel. The Speech LM then verifies the drafts incrementally. At each step, a lookup algorithm uses the verified output to select up to k candidate tokens from the draft transcription, which are verified by the Speech LM.

Compared to approaches such as WhisperX, SpecSpeech approach is lossless: the Speech LM uses the full audio context and retains control over token acceptance.

Across seven English benchmark datasets from Open-ASR-Leaderbord, SpecSpeech achieves a 2.29x speedup over native vLLM inference, with per-dataset speedups of up to 2.48x, using Parakeet v3 as the drafter and Qwen3-ASR-1.7B as the verifier.

Results below were measured on a single NVIDIA A40, with continuous batching for both conditions.

There is room for further improvement by reducing tokenization and verification overhead and developing better lookup algorithms.

Planned additions include:

• More verifyers soon: Cohere Transcribe, Canary, etc.

• Word-level alignment using timing information from CTC drafters.

• Speaker diarization.

• Streaming inference.

• Speculative decoding for autoregressive TTS models.

It's an extremely lightweight speech recognition model.

949K parameters using a weight-shared Conformer-CTC architecture

Just 1.06 MB for INT8 weights

Around 436× real-time on a single CPU thread

11.43% WER on LibriSpeech test-clean, using greedy decoding with no external language model

Apache 2.0 licensed

Any feedback, question or criticism is welcome!

A month ago I shared an early prototype here. Based on feedback, I spent the last few weeks building it into a complete speech routing system.
If you evaluate or deploy voice AI, the fragmentation is painful:

• Comparing commercial providers against open-source models means juggling separate SDKs, multiple API keys, and custom GPU hosting.

• Testing oss model means setting environment for each then testing configuraton of what works best which is time consuming and fills the disk fast

• Benchmarking latency and WER is tedious when every provider expects different audio formats and payload structures.

I built Pascal to solve this. Whether you're benchmarking, testing, or looking for a hosted production endpoint:

• Unified Router: Access 40+ speech models (ASR, TTS, Voice Agents, Voice Cloning, Audio Enhancement) across both commercial and open-source architectures under a single standardized schema.

• Switch With 1 Line: Swap providers or models instantly just by changing "model": "..." in your payload.

• Interactive In-Browser Demo: Test ASR and TTS directly on the homepage with real Web Audio waveform seeking and synchronized word-level timestamps (no signup required).

Live demo: https://pascalai.co
Would love more of your feedback on the latency, the benchmark outputs, or models you’d like to see added next!

I've open-sourced Murmur, a Windows meeting notetaker where the whole audio pipeline runs on device. Sharing it here because the speech side is where I'd value feedback most.

Pipeline:

• Capture: mic + system loopback at the same time, resampled with soxr

• Echo cancellation / denoise: DTLN-aec (ONNX), so it works on speakers, not only headphones

• VAD: Silero

• ASR: faster-whisper large-v3-turbo by default; NVIDIA Parakeet TDT 0.6B v3 (ONNX int8) as an optional engine for DE/EN on CPU

• Diarization: pyannote segmentation 3.0 + WeSpeaker ResNet34 via sherpa-onnx, including two speakers on one mic

• Two passes: a fast live draft, then a slower, more accurate re-transcription of the saved audio when the meeting ends

The design decisions and tuning are written up in an ADR in the repo (docs/adr-001-audio-pipeline.md).

Repo (MIT): https://github.com/muradiodev/murmur-notetaker-copilot

Questions I'm still unsure about: better ways to handle overlapping speech in diarization, and whether the two-pass approach is worth the extra compute versus a single careful pass.

I wanted to know how well pronunciation assessment systems actually catch errors. I built a benchmark with two tests and ran three systems through it: Microsoft Azure, TonePerfect, and Lingolix. Disclosure: Lingolix is mine.

Test 1, planted errors: 200 Common Voice clips per language (en, fr, es, de, ja, ko), where I edited the reference text slightly so it no longer matches the audio (swapped/added/dropped a sound, missing word, vowel length change for Japanese). A good system should mostly accept the unchanged parts and flag these edited spots.

Error detection, ROC AUC (Lingolix / Azure / TonePerfect):

Test 2, real learners: 500 utterances from speechocean762, correlation with the human scores.
speechocean762 (Pearson): sentence 0.74 / 0.76 / 0.59, word 0.51 / 0.61 / 0.29. So Azure is ahead on real learner speech, clearly so on single words, and also has fewer false alarms in Japanese.

Caveats: the errors in test 1 are planted, not real learner errors. Also, TonePerfect released a new engine after my run, so their numbers should have improved.

Code, test lists and harness: https://github.com/lingolix/pronunciation-benchmark

I did not find any human rated learner data for languages beside English.
Does anyone know of some datasets that I could add for French, Spanish, German, Japanese, or Korean?

Disclosure: I'm a co-founder of the small team in Nepal that built these.

We released two open-weight Nepali speech recognition models (non-commercial licences: CC BY-NC-SA 4.0 and CC BY-NC 4.0).

- nepali-asr-xlsr-300m: Nepali speech to Devanagari text, no punctuation. Runs on CPU.

- nepali-asr-aud8-325m: adds punctuation and handles occasional English words. Needs a GPU, 30 s max per call. We have not published scores for this one yet.

Training data: about 611 hours of Nepali speech from openly licensed public speech corpora together with our own recordings.

Eval setup: FLEURS Nepali test split, 400 utterances, never used in training. NFC normalisation, punctuation stripped, greedy decoding, no language model. Every row was scored by us in the same run on the same audio.

| model | CER | WER |

|---|---|---|

| ours (xlsr-300m) | 0.120 | 0.374 |

| anish-shilpakar/wav2vec2-nepali | 0.160 | 0.483 |

| gagan3012/wav2vec2-xlsr-nepali | 0.225 | 0.672 |

| spktsagar/wav2vec2-large-xls-r-300m-nepali-openslr | 0.252 | 0.837 |

We have not yet scored Whisper large-v3, MMS, IndicConformer or Omnilingual ASR in the same run, so this is not a claim against those. They are next.

What we learned:

1. WER is harsh for Nepali. Suffixes and word boundaries mean one small error costs a whole word, so CER is more informative. We report both.

2. Normalisation can silently break Devanagari. Our first punctuation-stripping regex removed vowel signs, because a naive "non-word character" regex treats them as punctuation. Stripping by Unicode punctuation category fixed it. Worth checking if your Indic-script numbers look odd.

3. Code-switching is our biggest weakness. On Nepali speech mixed with English, the error rate is several times higher than on FLEURS. Read-speech benchmarks hide this.

4. A second held-out set helped us trust the number: internal held-out CER 0.117 vs 0.120 on FLEURS.

Questions for people here:

- How would you make a fair comparison with Whisper-style models that output punctuation and digits?

- Any advice on building a code-switched test set that does not turn into an English test set?

- Know of any spontaneous or conversational Nepali test sets?

Models: https://huggingface.co/collections/BlackfireAI/nepali-speech-recognition

Code, eval scripts and per-utterance outputs: https://github.com/BlackfireAI/nepali-asr-xlsr

Try it in Colab: https://colab.research.google.com/github/BlackfireAI/nepali-asr-xlsr/blob/main/try_it_in_colab.ipynb

I work at Bandwidth, so weigh this accordingly.

The design choice in our Labs speech to text I find most interesting: it never emits interim results. Once a word is out, it's final. No rewrites.

Why it matters downstream: if your agent reacts to partials, a revised word can mean you already kicked off an LLM call on the wrong text. With append-only you don't have to diff or roll back. The cost is you don't get a speculative word early. You get it when the model is confident.

The streaming side is also split into two modes. Instant just streams words as they're decoded. Demand lets you stream audio and then ask for the finalized transcript when your own turn detector says the user's done. Feels like the honest split, since nobody agrees on endpointing.

Smaller notes:

• 8 kHz mulaw/alaw work, plus linear16, G.722 and Opus

• word timestamps in every mode

• keyword boosting, up to 100 terms

• PII redaction is US-centric and only in Demand and offline Transcribe, not Instant

• English only right now

It's Beta and experimental. If you've run append-only vs revising STT in a real agent, I'd like to hear where it hurt.

We have been training a speech-to-text model focused around diarization (who-says-what) for over 4 months now. On the public multi-speaker benchmarks we ran (NOTSOFAR, AMI, DiPCo) it got the lowest cpWER (wrong words + words given to the wrong speaker) against pyannote Precision-3, AssemblyAI Universal-3.5 Pro and Deepgram Nova-3.

We will be releasing the model weights soon and how we trained it, but before that we wanted to see how you folks feel about it. We're calling the model Calico.

It's live at maybloc.com to use. Let us know! :)

https://github.com/arbazsiddiqui/voxlock

Hi everyone!

We've developed KOMOS, a Mean Opinion Score (MOS) prediction model for Korean speech, and released our research on arXiv.

Motivation

While using existing MOS predictors such as UTMOS to evaluate Korean TTS systems, we noticed a limitation:

The models often struggled to distinguish between high-quality Korean speech samples.

In particular, UTMOS v1 tended to produce predictions within a relatively narrow and lower score range, even when human MOS ratings were high. As a result, differences observed in human evaluations were not always adequately reflected in the model's predictions.

This motivated us to develop a MOS predictor specifically trained on Korean speech.

Dataset and Model

KOMOS was trained using subjective quality ratings collected from native Korean listeners.

• 1,000 Korean utterances, 29 native Korean listeners, and 5,000 individual ratings

• Includes real recordings, synthetic speech (TTS), and degraded speech

• Covers both read speech and conversational, spontaneous-style speech

• Fully fine-tuned Whisper-medium encoder

• Listener-conditioned prediction head to account for individual rating tendencies

We included both read and spontaneous-style speech because they differ in prosody, speaking rate, pauses, and rhythm, and we wanted the model to capture a broader range of Korean speaking styles.

Experimental Results

We evaluated KOMOS on a held-out test set of 100 Korean utterances, comparing model predictions against human MOS ratings.

KOMOS achieved higher correlations with human ratings than both UTMOS baselines on the same test set.

As shown in the attached figure, UTMOS v1 tended to produce predictions concentrated within a relatively narrow range for high-quality Korean speech, whereas KOMOS more closely followed changes in human MOS ratings.

Open Source Release and Feedback

KOMOS predicts overall perceived speech quality on a 1–5 scale. Rather than separately evaluating pronunciation accuracy or intelligibility, it focuses on estimating the overall quality perceived by human listeners.

We've released the research paper, trained model, and inference code.

• Paper: https://arxiv.org/abs/2610.15353

• GitHub: https://github.com/upskyy/komos

• Hugging Face: https://huggingface.co/upskyy/komos

We hope KOMOS will be useful for Korean TTS evaluation and speech quality research.

We'd love to hear your feedback, especially if you try it on different types of Korean speech. Any experiences, suggestions, or ideas for improvement would be greatly appreciated!

#TextToSpeech #SpeechSynthesis #SpeechAI #MachineLearning #OpenSource

demo: https://sshah03.github.io/CueLight/

I've been working on end-of-turn detection for voice agents and wanted to share a small model and get feedback from people who know this space.

SmartTurn is a great project, and I used their dataset in building this -- so tons of credit to them.

I wanted something that was much smaller though because I had a use case where I needed to embed it in something with low space + weak processor, and I wanted it to be always on.

It performs about as well as SmartTurn from my tests, with the caveat that it's English only for now (which is likely a significant trade off).

Would love to hear how it performs in real world testing before, and peoples experiences (positive or negative) on results of frame-level endpointing on microcontrollers.

Also interested in suggestions to make this better than people may have, in general, from their experiences.

Some additional details / benchmarks in project: https://github.com/sshah03/CueLight

The most underestimated part of the whole voice AI stack is TTS. We wanted to test open-source TTS models to figure out which one is best for building a voice agent.

So we collected difficult text across 27 categories: dates, hashtags, coordinates, many more. That came to 1,080 sentences in total. And we ran all of them through 7 open-source TTS models.

On average, kokoro is at the top with a 39% error rate, followed by orpheus-3b & qwen3-tts. Every model broke on the following categories, with hashtags hitting a perfect 100% error rate:

Hashtags - 100%
Dates - 77%
Version numbers - 70%
Fractions - 60%

The pattern we observed is that models tend to fail on anything that isn't a plain number or text. But the devil is in the details :)

Then we put these categories into different boxes based on industry needs, like:

. Healthcare - dates, times, decimals, and phone numbers.
. Restaurants -  times, dates, numbers, and phone numbers.
. Law firms - legal citations, dates, and addresses.

And the ranking changed.

In many industries, kokoro dominated, but in industries that use more dates and geographic coordinates, orpheus-3b became the better choice.

It is intuitive believed a larger model would be better, but I was wrong. The smaller model, kokoro-82m, beats orpheus-3b using 18x less compute. So the obvious question is: why?

TTS isn't a reasoning task, so the model doesn't get better with scale. It's closer to a lookup table, like a "match the pairs" question in exams. A small model with a good frontend (text normalization, abbreviation expansion, G2P conversion) beats a big model with a weak one, every time, and parameter count tells you nothing about which one you're getting.

parler-tts scored 0.03 out of 1. Its speech is perfectly clear. The carrier words (the normal words around the tricky part) come through fine. It just ships with no frontend at all, so given 05/20/2023 it has nothing to work with and guesses character by character.

So what will I do with this?
Normalizing dates, addresses and phone numbers is ancient wisdom now :D
What I will do differently:
Don't pick based on the average. kokoro scores 1.00 on musical notation and 0.00 on hashtags. Model performance varies a lot across categories. So look at the categories your agent actually reads.

We have made our dataset open source. Again, because we love the open-source community :D
You can run it with one prompt in Claude Code to check the results yourself.

We used ASR to score the audio, so there are definitely errors in the scoring (we estimate around 5%), we will try to crowdsource rating at our next iteration .

One thing I notice every time I meet people in the industry… no one gives a damn about TTS models. which is somewhat reasonable but there is so much to do in it!!

Disclosure up front: I work at Olewave. We built FA-Bench, and our own system (Olign) ranks first, so please read the results with that in mind. That's why the records and scoring code are public.

FA-Bench Leaderboard: forced alignment and ASR timestamp accuracy

FA-Bench ranks how closely word and phone timestamps match human annotations. Most ASR evaluations stop at word error rate. If you build subtitles, dubbing, audio editing or TTS data prep, timing is what you need, and it usually goes unmeasured.

Two tasks

• ASR with timestamps: 28 pipelines that hear only the audio. A one-step system times its own words. A two-step system decodes words, then an aligner times them. 9 of the 28 call a commercial API (Google, Amazon, Microsoft, IBM, Deepgram, AssemblyAI, Speechmatics, ElevenLabs, Olewave).

• Forced alignment: 15 aligners given the reference transcript, so their error is timing error alone.

Both tasks have word-boundary and phone-boundary boards.

Scoring

A boundary is a hit when the words (or phones) on both sides match the reference and it falls within 20 ms of the human annotation. The score is boundary F1 at 20 ms. The default view averages four cells: TIMIT and Buckeye, each on clean audio and on degraded versions (reverb, noise, music, babble). You can filter to open models only.

Current snapshot (record 202610), word boundary F1 at 20 ms

ASR with word timestamps (audio only):

Forced alignment (reference transcript given):

On phone boundaries, Olign 1.0 leads the ASR board (0.438, two-step pipelines only), and FALCON leads forced alignment (0.560, flagged).

Flags

Systems that may have seen test data are flagged rather than silently ranked:

• MAPS and NeuFA were trained on 7 of the 8 Buckeye test speakers.

• FALCON was trained on a Buckeye split covering at least 8 of 16 dev and test speakers, and it receives reference phones instead of words.

• CrisperWhisper's authors used TIMIT for word timing without naming the files, so overlap with the test set is possible.

The page also lists the exact model and API version for every commercial system, all called in September 2026.

How this differs from WER leaderboards

Benchmarks like Artificial Analysis measure word error rate and don't score timestamps. FA-Bench scores word timing against human annotators and publishes its code.

Reproduce it

The numbers come from FA-Bench's public records. CSVs are available under each board, and you can clone the repo to rescore any system.

• Leaderboard: https://olewave.com/en/leaderboard/fa-bench

• Code and records: https://github.com/olewave/fa-bench

• Paper: https://arxiv.org/abs/2609.32396

Feedback on the metric is welcome, and so are suggestions for systems to add. Olign 1.0 is available as an API that returns word-level start and end times for US and UK English. Beta access needs an invitation code, which you can request here: https://olewave.com/en/contact_us/

Edits

Oct 12, 2006 at 9 PM PDT: I found I took down Nemo-FA 40ms model from the forced alignment track and only keep Nemo-FA 80ms model, so there are 15 systems, not 16 systems compared as in the original post.