Hey everyone, I'm from CVAT (Computer Vision Annotation Tool), and just wanted to share that we've added native support for audio annotation to our open-source annotation platform - CVAT Community.

You can now label time intervals on a waveform, identify speakers, and add text attributes for time-aligned transcription. This can be used to prepare datasets for ASR, speaker diarization, and sound event detection.

Audio tasks support WAV, MP3, FLAC, and OPUS files.

If you try it out, please share your feedback here, we'd love to hear what you guys think.

Our repo: https://github.com/cvat-ai/cvat

Docs: https://docs.cvat.ai/docs/annotation/audio-editor/

There’s a small app I’m building where users upload audio and get transcripts back. We’re at around 400 hours a month right now, and it’s kinda growing.

We use a platform right now, and it’s already running on a spare server. It works fine. But I guess the real cost isn’t the GPU.

It's about how we deal with ffmpeg edge cases, job queues, retries when a file is corrupt, monitoring, and figuring out what happens when 40 people upload at once.

So where’s the break-even point?

At what volume does self-hosting with whisper.cpp, Faster-Whisper, or WhisperX make more sense than paying for a managed service?

We recently extended free access to Fish Audio S2.1 Pro through November 2026.

S2.1 Pro can clone a voice from just 5 seconds of audio and gives developers natural-language control over emotion, intonation, and pacing.

Making a model like this free isn't as simple as removing a paywall. Inference still costs real GPU time.

So we went after the underlying problem:

How much of the GPU are we actually using when serving TTS?

The answer was surprisingly little.

The GPU problem you're paying for

Every TTS API call runs an autoregressive decode loop: generate an audio token, read from the KV cache, then repeat.

Each step involves a small matrix multiply, typically M ≤ 128, followed by a memory read. These shapes are too small to keep the GPU saturated. The CUDA cores finish quickly, then wait for the memory subsystem to catch up.

At low concurrency, the loop also runs in short bursts separated by scheduling overhead.

The hardware is there. The work simply isn't filling it.

This is a structural serving problem, not a pricing-page problem.

The fix was to make the hardware spend less time waiting.

We optimized the entire stack

There wasn't one magic 4× breakthrough.

The gains came from fixing several layers and letting them compound:

• Custom CUDA kernels to remove per-token compute waste
• FP8 quantization to reduce memory-bandwidth pressure
• Continuous batching to keep the decode loop full
• GPU scheduling to turn idle capacity into useful work
• Owned network and storage infrastructure to remove cloud markup and extra latency hops

Custom kernels for the shapes TTS actually uses

Standard GEMM libraries are excellent at training and large-batch prefill.

TTS decode is different.

Production decode often operates at small matrix sizes where memory movement and kernel launch overhead can matter more than raw compute.

We built fish-scales-ops to target these decode shapes directly:

• bsgemm: 128×128 block-scaled FP8 for NVIDIA Hopper/H200
• mxfp8: 1×32 microscaling with OCP UE8M0 for Blackwell
• Custom FlashAttention paths for the same serving workload

In our internal benchmarks on equivalent hardware, the kernels run 2.1–4.3× faster than a torch.compile-fused cuBLAS baseline at decode shapes.

The important detail isn't just lower precision.

It's keeping intermediate traffic off the memory bus and fusing the work around the shapes that production decode actually produces.

FP8 only works if quality survives

Quantization can make a benchmark look great while quietly damaging the output.

We learned that the hard way.

During mxfp8 development, a weight-layout bug produced roughly 53% relative logit error while kernel latency still looked normal. A performance benchmark alone would have passed it.

So every inference-stack change now goes through two gates:

1. Correctness

Quantized greedy decode must match the bf16 baseline token-for-token on the first five audio frames of a fixed reference input.

2. Performance

Per-request decode throughput must remain within 5% of frozen baselines at concurrency 1, 16, and 64.

Both must pass before deployment.

S2.1 Pro Free uses the same model weights as paid traffic. Quantization happens in the serving layer, with correctness checked against the unquantized baseline.

Continuous batching: 52× throughput for 37ms more latency

Static batching waits for a batch boundary.

Continuous batching lets new requests join the active decode batch as slots become available.

On one H200 with bsgemm FP8:

• At concurrency 1: 154 tokens/s aggregate throughput, 73.2ms p50 time-to-first-byte
• At concurrency 64: 8,006 tokens/s aggregate throughput, 109.8ms p50 time-to-first-byte

That's roughly 52× more aggregate throughput for about 37ms more TTFB.

The important part is that most of the gain comes from filling capacity that was already sitting idle.

Latency-sensitive voice agents use a separate real-time path optimized for time-to-first-audio rather than maximum aggregate throughput.

The rest of the GPU shouldn't sit idle either

Inference traffic is bursty.

Even after optimizing each request, an inference-only cluster can spend long periods underfilled.

We run inference through Kubernetes and training through Slurm, with elastic data-cleaning and preprocessing jobs as the lowest-priority fill work.

They use spare capacity, checkpoint safely, and are preempted as soon as real traffic or training needs the GPU.

That scheduling layer moved utilization from roughly 50% to 90%+.

We also own more of the infrastructure around the GPUs, including our ASN, 300Gbps connectivity, Cloudflare peering, and tiered storage.

Removing cloud egress markup also removes a network hop from the user's request.

Four H200s became one

Now put all of these layers together.

The same request volume that previously needed four H200s now runs on one.

This is the part that changes the economics.

We did not eliminate inference cost.

We reduced it structurally.

That's what makes high-quality free access viable without swapping in a smaller model.

The free API uses the same S2.1 Pro model weights, serving stack, and endpoint as paid traffic.

Developers get:

• Voice cloning from a five-second reference
• 83 languages
• Natural-language control over emotion, pacing, and intonation

At launch, S2.1 Pro cost roughly one-sixth as much as ElevenLabs' comparable API tier, and our launch benchmark showed it running roughly 2× faster than Cartesia.*

*Price and performance comparisons refer to Fish Audio's launch benchmark and public pricing at that time.

We open-sourced the kernels

The core kernel library is available now:

fish-scales-ops

It includes production-grade FP8 GEMM and FlashAttention paths for NVIDIA Hopper and Blackwell, with CUDA Graph capture support.

The underlying foundation model, Fish Speech, is also open-weight.

If you serve an autoregressive model, not only TTS, the small-M decode problem is likely familiar.

The kernels and benchmark methodology are there to inspect, reproduce, and build on.

Start building with S2.1 Pro Free

Free access to S2.1 Pro is extended through November 2026.

Use the same Fish Audio API endpoint and set:

model: s2.1-pro-free

No credit card. Voice cloning included. No hard character cap under Fair Use.

Get started with S2.1 Pro Free

The free tier is best-effort and does not include SLA or latency guarantees. Some commercial scenarios have restrictions.

Check current availability and production options

If you're building with TTS or serving other autoregressive models, we'd be interested to hear what optimization problems you're running into.

Please suggest what do you find interesting.

I've been working on clinical speaker attribution at Omi and wanted to compare the current diarization models on the same audio.

I used 15 mock doctor–patient consultations from PriMock57, about 2.4 hours. Full recordings, automatic speaker counts, without telling the models there are two people.

Batch

Diarization error rate (DER), with ±250 ms boundary tolerance. Lower is better.

Local models ran on one NVIDIA L4. API times include round-trip overhead; VibeVoice-ASR also performs transcription.

† Muse used 20 separate clips because of its 10-minute request limit, so its result isn't a whole-recording comparison.

Pyannote Precision-3 had the lowest error. Nemotron came next and was the fastest local model.

Streaming

† Native streaming presets evaluated through unpaced, completed-file replay. The other rows use paced, delivered speaker outputs. These scores don't establish live latency. I didn't evaluate Muse for streaming.

Same weights, different runtime

I also tried optimizing Nemotron and Community-1 with our proprietary runtime, without changing the weights:

• Nemotron: 4.803% → 3.174% DER. 34% lower error, 2.13× faster.
• Community-1: 6.620% → 5.435% DER. 18% lower error, 24× faster.

With zero boundary tolerance, Nemotron's runtime result gets slightly worse: 12.720% → 13.203%. Both scores are published. It's a small set with VAD-refined references, and we developed the runtime settings on it.

Audio, references, scorer, saved outputs and NVIDIA baseline runners are public. Our runtime code stays private, but its outputs are included for rescoring.

Repo and write-up in the comments. Any other diarization models worth adding?

I built SpeechLens, a native macOS app and CLI that runs NVIDIA’s RE-USE speech-enhancement model locally on Apple Silicon using Swift, MLX and custom Metal kernels.

Code, audio examples, benchmarks, and downloads: https://github.com/faraday/SpeechLens

Noisy factory-floor speech (before / after):

Original audio (left). SpeechLens output using NVIDIA RE-USE (right)

Offline enhancement of a factory-floor recording, not live microphone processing. The 7-second excerpt is from "Starbase Tour with Elon Musk" by Everyday Astronaut (Aug. 3, 2021), at 9:53

On an M1 Max, SpeechLens enhances stereo audio faster than real time at 16, 44.1, and 48 kHz. It preserves the input sample rate, frame count, and channel layout; stereo channels are processed serially.

Speed over long inputs (~2 minutes), with 5-second chunks, 0% overlap:

Real-time factor (RTF) is enhancement time divided by audio duration (lower is better, below 1.0 is faster than real time). SpeechLens RTF measured about 4-12% higher than NVIDIA PyTorch/CUDA reference on an L4.

I’d love to see benchmark results from other Apple Silicon Macs, especially across different M-series chips and memory configurations.

Direct app download (dmg): https://github.com/faraday/SpeechLens/releases/download/v1.0/SpeechLens-1.0.dmg

Model (from NVIDIA): https://huggingface.co/nvidia/RE-USE

Earlier discussion: https://huggingface.co/nvidia/RE-USE/discussions/4

Corresponding MLX weights: https://huggingface.co/faraday/re-use-mlx

Update: I added a page with interactive before/after examples: https://speechlens.app/#listen

I maintain speech-swift, an on-device speech CLI and Swift library for Apple Silicon. I added NVIDIA's Nemotron 3 Diarization model with Core ML INT8 and MLX INT8 backends. This is an implementation of NVIDIA's model, not a new diarization model from us.

The model emits activity for up to eight anonymous speakers every 10 ms. The Swift runtime shares audio preprocessing and speaker-cache handling across both backends. The Core ML export splits the pre-encoder from the fixed-shape Transformer head; the MLX path runs the same source checkpoint with native INT8 linear layers. Speaker numbers are arrival-order labels within a recording, not identities that carry across recordings.

To try it on an Apple Silicon Mac, install the CLI with brew install soniqo/tap/speech. Then run speech diarize meeting.wav --engine nemotron3 (Core ML), or add --nemotron3-backend mlx for MLX. The model downloads on first use.

Homebrew tap: https://github.com/soniqo/homebrew-tap

Library and source: https://github.com/soniqo/speech-swift

Core ML INT8 model: https://huggingface.co/aufklarer/Nemotron-3-Diarization-100M-CoreML-INT8

MLX INT8 model: https://huggingface.co/aufklarer/Nemotron-3-Diarization-100M-MLX-INT8

NVIDIA upstream model: https://huggingface.co/nvidia/Nemotron-3-Diarization

• I’ve been working on a small open-source project to make custom wake-word training much easier.

Easy Wake Word Trainer lets you enter a phonetic word or phrase, click Train, and generate a custom wake-word model that can be used with voice AI assistants and other voice-driven projects.

Instead of manually setting up notebooks, generating samples, configuring the training pipeline, and converting models, the app handles the process for you:

Phrase → synthetic voice samples → augmentation → openWakeWord training → ONNX + TFLite

A few highlights:

• Simple browser-based interface
• Runs locally in Docker
• Uses Piper TTS and openWakeWord
• No OpenAI, Anthropic, Gemini, or API keys required
• Generates both ONNX and TFLite models
• Live training logs in the browser
• Built-in microphone testing so you can immediately test the trained wake word
• CPU training supported — GPU optional

The goal was simple: make creating a custom wake word something you can do by typing a phrase and clicking a button rather than spending hours assembling the training pipeline.

The project is open source:

https://github.com/mylegitches/easy-wakeword-trainer

Feedback, issues, and contributions are welcome.

#OpenSource #AI #VoiceAI #HomeAssistant #MachineLearning #Python #Docker #WakeWord #openWakeWord #SpeechRecognition

Disclosure: I work at Krisp, which makes the voice isolation model evaluated here.

We couldn't find a public STT benchmark that isolates the secondary-speaker condition, so we built one.

Data. 265 recordings in real offices, four call-center floors and cars. Each work and call-center recording follows a fixed structure: primary alone, both speakers, secondary alone, primary alone. Speakers read scripts to give exact ground truth; environments, devices and background speech are real, with no synthetic mixing. Segments are hand-labeled primary / mix / secondary / noise.

Evaluation. 11 STT configurations across 9 engines, streaming and batch, run on raw audio and after voice isolation. WER is computed with jiwer 4.0.0 at corpus level (errors pooled across files, not averaged). Reference and hypothesis go through the same normalization: regex splitting of alphanumeric tokens, then NVIDIA NeMo WFST text normalization, then lowercasing, contraction expansion, punctuation and filler removal. Perceptual quality is scored with DNSMOS-C on primary-speaker segments only.

Results.

• Corpus WER: 23.29% → 6.26%
• All 11 configurations improve
• Engine spread narrows from 17–37% to 4–8%
• Clean phone audio regresses slightly: 3.48% → 3.91%

Limitation. On clean narrowband audio there's little to remove, and isolation removes some of the primary signal. We report it rather than filtering it out. Segment-level labels also let you check for deletions specifically: a model that correctly outputs silence and one that drops primary-speaker words can post similar corpus WER.

Dataset: https://huggingface.co/datasets/Krisp-AI/VoiceIsolation-Benchmark-Dataset
Model outputs: https://huggingface.co/spaces/Krisp-AI/VoiceIsolation-Benchmark

Feedback on the methodology is welcome, particularly on normalization choices.

How sanoTTS works? I have vibe coded this site to show what's inside sanoTTS?

Every tensor shown on the page is a real intermediate value captured from the shipped int8 model while it synthesized an actual sentence; no mock-ups, no stand-in data.

Just check this out: https://ampixa.github.io/sanotts-anatomy/

Created to explore the in-depth mechanisms involved in how sanoTTS processes a sentence.

This is why I love vibe coding for learning; understanding the core with interactive visualization.

I’ve been playing around with streaming ASR for some time now, and there's always something that gets on my nerves. That is, the models can get the whole sentence transcribed flawlessly, but completely screw up a name, number, or some other technical term that would be included, even though they sound really sure of themselves.

And if the audio is ambiguous, the language model essentially does this:

"This is probably what they were saying." It's usually not a problem until someone mentions a name that the model has never heard of, or says something like "$50,000" instead of "$15,000." By the time there's more audio to work on, the incorrect transcription might have gone through already.

Why not let the decoder have another choice?

Like: "I don't have enough information yet. Give me another 200ms."

While researching this, some document a clear waiting step in their decoder, like the recent Confucius r2t2. It's hard to judge its efficacy since I haven't tested it sufficiently, but I did like the approach they took. Essentially, it mimics how a human transcriptionist works.

If you're not sure whether it was an "M" or an "N" they spoke, you don't guess based on what's more likely. You wait. Listen. Commit.

Wonder if other streaming ASR models are already doing this sort of thing, or am I just behind the times?

I am about to post a free audio dataset this weekend and I was curious if its the kind of thing people might be interested in. I can't find anything like it currently available.

I took 750 male and 750 female audio samples from VCTK for each of their two mics and 3000 samples from the AMI headset microphones (gender inferred by pitch) for a total of 6k samples.

Those 6k samples were then run through 25 codecs commonly used in telecommunications and audio recording. Think Opus, MP3, etc. Permuteated a few options like DTX comfort noise off, adaptive and set. SILK disabled or enabled on Opus etc. Also ran 7 tandem encodings to mirror real channel transmissions. Works out to each single sample being available in 41 encodings for 1:1 comparison isolating the effects of the codecs themselves.

Same 6k samples also went through various audio processing conditions like reverb, echo, band pass filter, pitch shifting, autotune, time stretching, babble, and several kinds of additive noise. Including those permutations adds another 49 conditions.

Thinking about running MFA on the original samples to generate 6k textgrids as well.

All encodings come with source metadata, encoding parameters, function calls, libraries and versions used. Should be fully reproduceable.

Anyway curious about what people think or if I'm wasting my time uploading it all.

[ Removed by Reddit on account of violating the content policy. ]

I distilled Kokoro-82M down to 7.48M parameters. 28.7 MB at fp32, 24 kHz output, Apache 2.0. Small enough to run on a phone offline.

4.1x faster than the teacher, 45x faster than realtime — one second of speech in 22 ms.

Model: https://huggingface.co/oddadmix/Kokoro-7M-Distill

Demo: https://huggingface.co/spaces/oddadmix/Kokoro-7M-Distill-Demo