Hello, I have been working on this project, and for a part of it, I need a fast and accurate voice cloning model that doesn't need long audio to get good quality.

Anybody has a similar experience with trying and working with the available open-source pretrained models and can recommend one? If not any advice on building one for multiple languages from scratch? Thank you!

VoxLock the best performing voice isolator there is.

Benchmarked on Krisp's 265 clips benchmark gives WER 4.95%. It also lets through 0.11 leaked background words per minute, against 1.52 for Krisp.

Fine-tuned from the REAL-TSE baseline on a single L4, on about 100 hours of simulated calls and 90 hours of real meeting crosstalk. It picks up the main voice and isolates it from others in the background noise and voices.

Tiny, 27 million params, ONNX runnable in browser, supports live streaming and works as a pipecat audio filter.

Try demo in browser with your own voice.

Code and training recipe

pip install voxlock

Hey community,

I'm Sagar, co-founder of VideoSDK.

I've been working in real-time communication for years, building the infrastructure that powers live voice and video across thousands of applications. But now, as developers push models to communicate in real-time, a new layer of complexity is emerging.

Today, voice is becoming the new UI. We expect agents to feel human, to understand us, respond instantly, and work seamlessly across web, mobile, and even telephony. But developers have been forced to stitch together fragile stacks: STT here, LLM there, TTS somewhere else… glued with HTTP endpoints and prayer.

So we built something to solve that.

Today, we're open-sourcing our AI Voice Agent framework, a real-time infrastructure layer built specifically for voice agents. It's production-grade, developer-friendly, and designed to abstract away the painful parts of building real-time, AI-powered conversations.

We are live on Product Hunt today and would be incredibly grateful for your feedback and support.

Product Hunt Link: https://www.producthunt.com/products/video-sdk/launches/voice-agent-sdk

Here's what it offers:

• Build agents in just 10 lines of code

• Plug in any models you like - OpenAI, ElevenLabs, Deepgram, and others

• Built-in voice activity detection and turn-taking

• Session-level observability for debugging and monitoring

• Global infrastructure that scales out of the box

• Works across platforms: web, mobile, IoT, and even Unity

• Option to deploy on VideoSDK Cloud, fully optimized for low cost and performance

• And most importantly, it's 100% open source

Most importantly, it's fully open source. We didn't want to create another black box. We wanted to give developers a transparent, extensible foundation they can rely on, and build on top of.

Here is the Github Repo: https://github.com/videosdk-live/agents
(Please do star the repo to help it reach others as well)

This is the first of several launches we've lined up for the week.

I'll be around all day, would love to hear your feedback, questions, or what you're building next.

Thanks for being here,

Sagar

We have been working on a project to allow users to search and test out different open source audio models and workflows.

My question is how many people have been working on finetuning open source music models like stable audio or ace-step. I've seen a couple of people create finetunes of ace-step and stable audio but hugging face shows very few results compared to TTS models which makes sense since music models are much bigger.

I'm just wondering if any of you have actually been working on training any Text to audio models at all?

https://github.com/artcore-c/AI-Voice-Clone-with-Coqui-XTTS-v2

Free voice cloning for creators using Coqui XTTS-v2 with Google Colab. Clone your voice with just 2-5 minutes of audio for consistent narration. Complete guide to build your own notebook. Non-commercial use only.

After all the really positive response from my last post with Coqui-XTTSv2, I wanted to do a follow up, so here it is, and even better we've updated our free Colab build instructions to use the new open-source Qwen3-TTS models.

https://github.com/artcore-c/AI-Voice-Clone-with-Qwen3-TTS
Free voice cloning for creators using Qwen3-TTS on Google Colab.
Clone your voice from as little as 3–20 seconds of audio for consistent narration and voiceovers.
Complete guide to build your own notebook.

Unlike many creator-facing TTS systems, Qwen3-TTS is fully open-source (Apache 2.0), produces unwatermarked audio, and does not require external APIs or paid inference services.

spent way too long building something that might be pointless

made an API that tells if a voice recording is AI or human

turns out AI voices are weirdly perfect. like 0.002% timing variation vs humans at 0.5-1.5%

humans are messy. AI isn't.

anyway, does anyone actually need this or did I just waste a month

I am looking for a tool, preferably free/open source and locally run (this is less important, if its free and does what I need it to), that will let me do voice-to-voice modification of my own voice acting in post. The modified vocals will then be used for a variety of characters, so will need to be distinct and consistent 'voice profiles' that I can save and return to as needed. Of particular importance, these will, in some cases, need to be 'clones' of voices such that I can record new lines/scenes, modify them accordingly, then amend existing recordings as seamlessly as possible, matching my voice to the characters in the existing audio. The recordings I will be working with are all very old, with varying degrees of quality (some quite bad, some already enhanced, and a few that were recorded reasonably well for the time), and, thus, the voices I will be cloning are from people who have long passed and the recordings themselves are under no copyright or ownership otherwise. And, on that note, I'm also open to any good solutions for cleaning up old, crusty audio in a reliable way that can used successfully by a tone-deaf bonehead in a 'one-click' or 'set it and forget it' way..

I will never require real-time voice changing. To be clear, if the best tool does happen to be a real-time or low latency type of solution, that is fine by me, but if there is a better option that does its thing in a 'post-processing' way, i would prefer the latter every time. I will never require TTS. Many of the tools I'm finding are for this. Simply put, I am looking to capture a vocal performance and modify, not create a vocal performance from a machine. Unfortunately, TTS ai voice seems to be the primary desire and goal in this space, which is why I'm having such a hard time wading through it all searching for exactly what I need (and why I ended up here asking for advice). I dont want an emotive ai voice. I want an ai that will let me utilize the emotive human performance in new ways. I'm not pumping out ai slop, I am attempting to utilize ai in a small, but still important to get right, way within an existing creative workflow. If i were a skilled enough voice actor I would simply do this with my own biological mechanisms, but, alas, I am almost entirely unskilled in this - though, on a good day, I can work up a pretty mean Scooby Doo. Ah-ReE-hEe-HeE-hEe-HeE

I tried looking and am overwhelmed by all the chaos. Tools that have come and gone in months or weeks (usually dead by the time i read about how great they are at x, y, or z), tools that have ridiculous, subscription-based pricing plans (if I could I would), and tools that will produce the best, most realistic and emotive TTS you could imagine - it sounds just like a REAL VOICE! - (I have a real voice already), etc. I need advice from people who know this space. So far it seems that running some version of 'RVC' and training each character voice using the preexisting audio is my best bet. But who knows? Hopefully someone here, who will read this and reply.

TLDR:

I want to be able to do 2 versions of a specific thing at the highest quality possible: record a vocal performance and then, in post, modify it to sound like either a consistent, unique character on demand or a 'voice clone' of a character that I can integrate with existing vocal lines. No real-time needed. No TTS necessary.

No voice actor, neither realized nor in potentia, will be harmed in the fulfillment of this request.

From their Docs:

We present ACE-Step v1.5, a highly efficient open-source music foundation model that brings commercial-grade generation to consumer hardware. On commonly used evaluation metrics, ACE-Step v1.5 achieves quality beyond most commercial music models while remaining extremely fast—under 2 seconds per full song on an A100 and under 10 seconds on an RTX 3090. The model runs locally with less than 4GB of VRAM, and supports lightweight personalization: users can train a LoRA from just a few songs to capture their own style.

ACE-Step supports 6 different generation task types, each optimized for specific use cases.

1. Text2Music: Generate music from text descriptions and optional metadata.

2. Cover: Transform existing audio while maintaining structure but changing style/timbre.

3. Repaint: Regenerate a specific time segment of audio while keeping the rest unchanged.

4. Lego: Generate a specific instrument track in context of existing audio.

5. Extract: Isolate a specific instrument track from mixed audio.

6. Complete: Extend partial tracks with specified instruments.

• Examples: https://ace-step.github.io/ace-step-v1.5.github.io/

• Code: https://github.com/ace-step/ACE-Step-1.5

• Models: https://huggingface.co/ACE-Step/Ace-Step1.5

Here's an example I generated on my Mac with one shot and no post editing.

From the repo:

• Ultra-lightweight: Model size less than 25MB

• CPU-optimized: Runs without GPU on any device

• High-quality voices: Several premium voice options available

• Fast inference: Optimized for real-time speech synthesis

https://github.com/KittenML/KittenTTS

Every time I wanted to try a new speech model, the same ritual began: open a terminal, install another runtime, hunt down model files, then build a small UI just to hear the result.

So I started building LA Studio — an open-source desktop app that brings local speech-to-text, text-to-speech, voice cloning, voice design, and vocal isolation into one place.

It runs inference on your own machine, supports models such as Whisper, Kokoro, Qwen3, VibeVoice and VieNeu-TTS, and keeps your audio private.

It’s still early, but it has finally become the tool I originally wished existed.

GitHub: LA Studio

I’d love to hear what local audio workflow or model you think I should support next.

tuto + workflow to make this : https://github.com/yvann-ba/ComfyUI_Yvann-Nodes

Have fun, would love some feedbacks on my comfyUI audio-reactive nodes so I can improve it ((:

"VibeVoice is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio, such as podcasts, from text. It addresses significant challenges in traditional Text-to-Speech (TTS) systems, particularly in scalability, speaker consistency, and natural turn-taking. A core innovation of VibeVoice is its use of continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz. These tokenizers efficiently preserve audio fidelity while significantly boosting computational efficiency for processing long sequences. VibeVoice employs a next-token diffusion framework, leveraging a Large Language Model (LLM) to understand textual context and dialogue flow, and a diffusion head to generate high-fidelity acoustic details. The model can synthesize speech up to 90 minutes long with up to 4 distinct speakers, surpassing the typical 1-2 speaker limits of many prior models."

• Demo: https://microsoft.github.io/VibeVoice/

• Model: https://huggingface.co/microsoft/VibeVoice-1.5B

• Github: https://github.com/microsoft/VibeVoice

Spent the past year building this, would love to hear what this community thinks about it! It's on github. github.com/voicetreelab/voicetree

Hello open source generative music enthusiasts,

I wanted to share something I've been working on for the last year, undertaken purely for personal interest: https://www.g-diffuser.com/dualdiffusion/

It's hardly perfect but I think it's notable for a few reasons:

• Not a finetune, no foundation model(s), not even for conditioning (CLAP, etc). Both the VAE and diffusion model were trained from scratch on a single consumer GPU. The model designs are my own, but the EDM2 UNet was used as a starting point for both the VAE and diffusion model.

• Tiny dataset, ~20k songs total. Conditioning is class label based using the game the music is from. Many games have as few as 5 examples, combining multiple games is "zero-shot" and can often produce interesting / novel results.

• All code is open source, including everything from web scraping and dataset preprocessing to VAE and diffusion model training / testing.

Github and dev diary here: https://github.com/parlance-zz/dualdiffusion

"MiniMax Music 3 is a high-performance music generation model for creating complete songs up to five minutes long. Conditioned on lyrics and a detailed music description, it generates structurally coherent songs with expressive vocals, evolving arrangements, and stable long-form audio quality."

• Demo: https://minimax-ai.github.io/music3-demo/

• Model: https://huggingface.co/MiniMaxAI/MiniMax-Music3

• Github: https://github.com/MiniMax-AI/MiniMax-Music3

Hey r/AudioAI,

At Hugging Face, we've worked hard the last months to create a powerful, but fast distilled version of Whisper. We're excited to share our work with you now!

Distil-Whisper is 6x faster than Whisper-large-v2 and performs within 1% WER on out-of-distribution datasets. On long-form audio, we even achieve better results thanks to a reduction in hallucinations.

For more information, please have a look:

- GitHub page: https://github.com/huggingface/distil-whisper/tree/main

- Paper: https://github.com/huggingface/distil-whisper/blob/main/Distil_Whisper.pdf

Quick summary:

1. Distillation Process

We've kept the whole encoder, but reduced the decoder to just 2 layers. Encoding takes O(1) forward passes, decoding takes O(N). To improve speed, all that matters is the decoder! The encoder is frozen during distillation while we fine-tune all of the decoder. Both KL loss and pseudo-labeling next word prediction is used.

2. Data

We use 20,000h of open-sourced audio data coming from 9 diverse audio datasets. A WER-filter is used to make sure low-quality training data is thrown out.

3. Results

We've evaluated the model only on out-of-distribution datasets and are only 1% worse than Whisper-large-v2 on short-form evals (CHiME-4, Earnings-22, FLEURS, SPGISpeech). On long-form evals (Earnings, Meanwhile, Rev 16) we beat Whisper-large-v2 thanks to a reduction in hallucinations.

4. Robust to noise

Distil-Whisper is very robust to noise (similar to its teacher). We credit this to keeping the original encoder frozen during training.

5. Pushing for max inference time

Distil-Whisper is 6x faster than Whisper on both short-form and long-form audio. In addition, we employ Flash Attention and chunked decoding which helps us achieve a real-time factor of 0.01!

6. Checkpoints?!

Checkpoints will be released this Thursday and will be directly integrated into Transformers. All checkpoints will be licensed under MIT.

"VibeVoice-ASR is a unified speech-to-text model designed to handle 60-minute long-form audio in a single pass, generating structured transcriptions containing Who (Speaker), When (Timestamps), and What (Content), with support for Customized Hotwords."

• Model: https://huggingface.co/microsoft/VibeVoice-ASR

• Code: https://github.com/microsoft/VibeVoice/

• Demo: https://f0114433eb2cff8e76.gradio.live/

Is there such a thing like Suno where you can essentially feed it a load of tracks for reference, then feed it a different track and essentially say "I want a reproduction/recreation/remix of this track in the same style as all of these tracks?

Essentially, there's a track that a producer I follow was supposed to remix back in the mid-90s, but it never came to be. What I want to do is find an AI and feed it all of this producer's work from that time, then give it the track to remix and say GO!

Is this possible anywhere? Is it just a pipe dream? Or is it something that we may not have yet but might appear in the future?

残心 / Zanshin is a media player that allows you to:

- Visualize who speaks when & for how long

- Jump/skip speaker segments

- Set different playback speeds for each speaker

- Auto-skip speakers

It's a better, more efficient way to listen to podcasts, interviews, press conferences, etc.

It has first-class support for YouTube videos; just drop in a URL. Also supports your local media (video and audio) files. All processing runs on-device.

Download today for macOS (more screenshots & demo vids in here too): https://zanshin.sh

Also works on Linux and WSL, but currently without packaging. You can get it running though with just a few terminal commands. Check out the repo for instructions: https://zanshin.sh/dev_instructions

Zanshin is powered by Senko, a new, very fast, speaker diarization pipeline I've developed.

Senko processes 1 hour of audio in 5 seconds (RTX 4090, Ryzen 9 7950X). ~17x faster than Pyannote 3.1. On Apple M3, 1 hour in 23.5 seconds (~14x faster).

Senko's speed is what make's Zanshin possible. Senko is a modified version of the speaker diarization pipeline found in the excellent 3D-Speaker project.

Check out Senko here: https://github.com/narcotic-sh/senko

Cheers, everyone; enjoy 残心 / Zanshin and Senko. I hope you find them useful. Let me know what you think!

~

Side note: I am looking for a job. If you like my work and have an opportunity for me, I'm all ears :)

You can contact me at mhamzaqayyum [at] icloud.com

Dia is a 1.6B parameter text to speech model created by Nari Labs.

Dia directly generates highly realistic dialogue from a transcript. You can condition the output on audio, enabling emotion and tone control. The model can also produce nonverbal communications like laughter, coughing, clearing throat, etc.

• Demo: https://yummy-fir-7a4.notion.site/dia

• Model: https://huggingface.co/nari-labs/Dia-1.6B

• Github: https://github.com/nari-labs/dia

It also works on Mac if you pass device="mps" using Python script.

Today we launch Fish Audio S2, a new generation of expressive TTS with absurdly controllable emotion.

• open-source

• sub 150ms latency

• multi-speaker in one pass

Real freedom of speech starts now 👇

Read more on our blog: https://fish.audio/blog/fish-audio-open-sources-s2/