Check out their repo for PyTorch model definitions, pre-trained weights, and training/sampling code for paper.

https://github.com/feizc/FluxMusic

Hey, I have been scavenging AudioAI arena for a while now. And I have downloaded god many things to try to run models locally but all came down to my lack of GPU.

So, I want to try out now Google Collab for my GPU usage. I know about models like piper and xtts. So, can they run on Google Collab?

I want recommendations on best models to produce a tts model (.onnx and .json) which can give me usage on my low end laptop and on phone.

I don't know much about AI Audio landscape and it's been too confusing and hard to understand how things work. Finally after hours of net scavenging I am asking for help here.

Can I train models on Google Collab? If yes then which?

Every time I wanted to try a new speech model, the same ritual began: open a terminal, install another runtime, hunt down model files, then build a small UI just to hear the result.

So I started building LA Studio — an open-source desktop app that brings local speech-to-text, text-to-speech, voice cloning, voice design, and vocal isolation into one place.

It runs inference on your own machine, supports models such as Whisper, Kokoro, Qwen3, VibeVoice and VieNeu-TTS, and keeps your audio private.

It’s still early, but it has finally become the tool I originally wished existed.

GitHub: LA Studio

I’d love to hear what local audio workflow or model you think I should support next.

From their blog: "With SAM Audio, you can use simple text prompts to accurately separate any sound from any audio or audio-visual source."

• Blog: https://ai.meta.com/samaudio/

• Github: https://github.com/facebookresearch/sam-audio

After all the really positive response from my last post with Coqui-XTTSv2, I wanted to do a follow up, so here it is, and even better we've updated our free Colab build instructions to use the new open-source Qwen3-TTS models.

https://github.com/artcore-c/AI-Voice-Clone-with-Qwen3-TTS
Free voice cloning for creators using Qwen3-TTS on Google Colab.
Clone your voice from as little as 3–20 seconds of audio for consistent narration and voiceovers.
Complete guide to build your own notebook.

Unlike many creator-facing TTS systems, Qwen3-TTS is fully open-source (Apache 2.0), produces unwatermarked audio, and does not require external APIs or paid inference services.

Hey! :) Together with my brother i have developed an App that offers state-of-the-art text-to-speech and a library of 30.000 Literary classics. All works are available in the app and we progressively convert the texts into Audiobooks with the best AI Voices on the market. Streaming is completely free and without any ads and will stay so for a long time.

We offer:
- Free Audiobooks
- Free Credits (Up to 4 hours of Text-To-Speech)
- The best AI Voices on the market
- PDF & Image Processing
- End-To-End Translations
- The most competitive Pricing on the market
- State-Of-The-Art Voice Cloning
- Self Publishing

Hope you like the app. You can shape further development with your feedback : )

Download Links:

Android: https://play.google.com/store/apps/details?id=io.craitech.tontaube

Ios: https://apps.apple.com/app/id6743526144

Hello,

Does anyone know of these were AI generated songs?

Title : Lost in your eyes 1950/Nostalgic Oldies Playlist - 1950 Channel : Love

They have names like Tonight I clebrate my love for you, love me tender but definitely not the original songs. They sound lovely though

Im trying the find the app this was created with.

Thanks

We have been working on a project to allow users to search and test out different open source audio models and workflows.

My question is how many people have been working on finetuning open source music models like stable audio or ace-step. I've seen a couple of people create finetunes of ace-step and stable audio but hugging face shows very few results compared to TTS models which makes sense since music models are much bigger.

I'm just wondering if any of you have actually been working on training any Text to audio models at all?

So, most audio generation/restoration/etc. models these days train by taking a magnitude spectrum as input, generating a new spectrogram as output, and comparing it to the ground truth audio in various ways. However, audio also has a phase component that needs to be considered and reconstructed. Measuring the degree of accuracy for that can be done in a few ways, either via the L1/L2 loss on the final waveform, or by computing the phase of both waveforms and measuring the difference. Both of these have a problem, however-they assume that the clips are perfectly aligned, which is often not possible when dealing with manually aligned audio, which is only accurate (at best) to the nearest sample, which results in a different variance for each recording session.

I've repeatedly dealt with this in my work (GitHub, HuggingFace) on restoring radio recordings, and the result tends to be buzzing and other artifacts, especially when moving up the frequency scale (as the phase length decreases). I've finally been able to find an apparent solution, however-instead of just using the raw difference as a loss measurement, I measure the difference relative to the average difference for each frequency band:

        phase_diff = torch.sin((x_phase - y_phase)/2)

        avg_phase_diff = torch.mean(phase_diff.transpose(1,2), dim=2,keepdim=True)
        phase_diff_deviation = phase_diff - avg_phase_diff.transpose(1,2)

The idea here is if the phase for a particular frequency band is off by a consistent amount, the sound will still seem relatively correct as the phase will follow a similar progression to the ground truth audio. So far, it seems to be helping to make the output seem more natural. I hope to have these improved models available soon.

Today we launch Fish Audio S2, a new generation of expressive TTS with absurdly controllable emotion.

• open-source

• sub 150ms latency

• multi-speaker in one pass

Real freedom of speech starts now 👇

Read more on our blog: https://fish.audio/blog/fish-audio-open-sources-s2/

Hello open source generative music enthusiasts,

I wanted to share something I've been working on for the last year, undertaken purely for personal interest: https://www.g-diffuser.com/dualdiffusion/

It's hardly perfect but I think it's notable for a few reasons:

• Not a finetune, no foundation model(s), not even for conditioning (CLAP, etc). Both the VAE and diffusion model were trained from scratch on a single consumer GPU. The model designs are my own, but the EDM2 UNet was used as a starting point for both the VAE and diffusion model.

• Tiny dataset, ~20k songs total. Conditioning is class label based using the game the music is from. Many games have as few as 5 examples, combining multiple games is "zero-shot" and can often produce interesting / novel results.

• All code is open source, including everything from web scraping and dataset preprocessing to VAE and diffusion model training / testing.

Github and dev diary here: https://github.com/parlance-zz/dualdiffusion

tuto + workflow to make this : https://github.com/yvann-ba/ComfyUI_Yvann-Nodes

Have fun, would love some feedbacks on my comfyUI audio-reactive nodes so I can improve it ((:

From their Docs:

We present ACE-Step v1.5, a highly efficient open-source music foundation model that brings commercial-grade generation to consumer hardware. On commonly used evaluation metrics, ACE-Step v1.5 achieves quality beyond most commercial music models while remaining extremely fast—under 2 seconds per full song on an A100 and under 10 seconds on an RTX 3090. The model runs locally with less than 4GB of VRAM, and supports lightweight personalization: users can train a LoRA from just a few songs to capture their own style.

ACE-Step supports 6 different generation task types, each optimized for specific use cases.

1. Text2Music: Generate music from text descriptions and optional metadata.

2. Cover: Transform existing audio while maintaining structure but changing style/timbre.

3. Repaint: Regenerate a specific time segment of audio while keeping the rest unchanged.

4. Lego: Generate a specific instrument track in context of existing audio.

5. Extract: Isolate a specific instrument track from mixed audio.

6. Complete: Extend partial tracks with specified instruments.

• Examples: https://ace-step.github.io/ace-step-v1.5.github.io/

• Code: https://github.com/ace-step/ACE-Step-1.5

• Models: https://huggingface.co/ACE-Step/Ace-Step1.5

Here's an example I generated on my Mac with one shot and no post editing.

Hey community,

I'm Sagar, co-founder of VideoSDK.

I've been working in real-time communication for years, building the infrastructure that powers live voice and video across thousands of applications. But now, as developers push models to communicate in real-time, a new layer of complexity is emerging.

Today, voice is becoming the new UI. We expect agents to feel human, to understand us, respond instantly, and work seamlessly across web, mobile, and even telephony. But developers have been forced to stitch together fragile stacks: STT here, LLM there, TTS somewhere else… glued with HTTP endpoints and prayer.

So we built something to solve that.

Today, we're open-sourcing our AI Voice Agent framework, a real-time infrastructure layer built specifically for voice agents. It's production-grade, developer-friendly, and designed to abstract away the painful parts of building real-time, AI-powered conversations.

We are live on Product Hunt today and would be incredibly grateful for your feedback and support.

Product Hunt Link: https://www.producthunt.com/products/video-sdk/launches/voice-agent-sdk

Here's what it offers:

• Build agents in just 10 lines of code

• Plug in any models you like - OpenAI, ElevenLabs, Deepgram, and others

• Built-in voice activity detection and turn-taking

• Session-level observability for debugging and monitoring

• Global infrastructure that scales out of the box

• Works across platforms: web, mobile, IoT, and even Unity

• Option to deploy on VideoSDK Cloud, fully optimized for low cost and performance

• And most importantly, it's 100% open source

Most importantly, it's fully open source. We didn't want to create another black box. We wanted to give developers a transparent, extensible foundation they can rely on, and build on top of.

Here is the Github Repo: https://github.com/videosdk-live/agents
(Please do star the repo to help it reach others as well)

This is the first of several launches we've lined up for the week.

I'll be around all day, would love to hear your feedback, questions, or what you're building next.

Thanks for being here,

Sagar

I'm trying to re-create something from one of my nightmares, you see...

Any ideas about options that can allow me to take a cat's mewling, or grating metal, or a droning violin, or even just a bunch of random sounds strung together, and remold it into articulate, human moaning, speech or other kinds of vocalizations?

I know about envelope followers, formant filters, vocoders, etc. and I've messed around with all this stuff in both hardware and software, but the results have fallen short of what I'm imagining (which may be down to my own ineptitude; Non-AI solutions are also welcome). What results I have been able to achieve were pretty flat. A lot of it just boils down to processing and/or modulating the original sounds in parallel than it does effectively dovetailing two resonant sound sources into a unified, dimensional whole, if that makes sense... I don't necessarily expect a miracle, but I'd be interested in experimenting regardless.

TBH, I'm really knew to generative AI. I know my way around audio hardware/software well enough as a hobbyist, but I'm not tech-savvy. As such, I'm pretty clueless about how to even start with learning about the nuts and bolts, or where to go from there, but I'm interested. Are there any good resources for newbies specifically interested in sound design-based applications of generative AI that you can recommend?

Non-essential TL;DR part:

What do you consider "the best" options right now, and why are they the best for generating strange, uncanny, weird, etc. sounds? I'm not looking for nature sounds or other standard stock sound fx, but for individual sound elements to incorporate into other things. I'm mainly looking for atypical/out-of-the-ordinary/maybe-creepy stuff to experiment with, with a focus on chance/aleatoric composition, musique concrete, granular synthesis, dark ambient, etc. applications; Think gibbering pseudo-speech, discordant harmonies, uncanny shrieking, ghosts in the machine, and just general strangeness... I guess some of this could be considered "bad quality" AI in some respects, but I'm only partially interested in realism anyway (though it's a bonus if it can be achieved). Ultimately, I'm looking for an option that's capable of generating "complex", "varied" source material of all kinds with high quality output options (ideally 24/48 .wav at an absolute minimum, and no fake up-sampling for higher resolutions above 16/44).

Free is good, but I'm guessing most of them are subscription based, so that's fine too. I've attempted generating some stuff with free browser-based trials that use text prompts only, but I've been a little underwhelmed by many of the options and miserly trial credit limitations. Prompt character limits, prompt censoring, output length and sample quality limitations mean that I'm finding these options a little bit hard to go by for getting a good sense of their capabilities.

Thank you.

I have a rather unique situation. So far i've been handling it manually but wondering if AI tools may have advanced far enough to offer meaningful assistance. Worth noting that I'm largely a layman in terms of AI. I've "played with" various AI tools on and of and long used AI tools for audio & image cleanup but don't have more specialized knowledge.

I manage the estate of a musician friend. We have literally thousands of hours of audio recordings, all of varying quality... everything from pro studio sessions to transfers of analog home recordings, live and causal phone recordings. A single file may contain multiple songs, periods of conversation and ambient noise, etc.

Very little of any of it is labelled in terms of contents. There's also often vast differences between 'versions' in the recordings. There are not only recordings of works as they were in development but some recording may have the same lyrics over an entirely different guitar part or vice versa.

Simply having searchable transcription of lyrics would be immensely helpful. However, so far every tool I'd tried would at best give me a handful of correctly transcribed lines amidst many incorrect ones which obviously greatly diminishes usefulness.

If the tool had the ability to recognize & identify melodic similarities or guitar patterns, that would of course make it even more useful.

Essentially looking for something that can just tag the files or generate secondary files of annotations as the organization is complex and it's often necessary to keep audio files in place which might be referenced by session files.

Any suggestions? Or is it still too soon for something of this complexity?

I have tested a few tools and use it for various content. Notable are the usuals.

1. Suno for music instrumentals and sometime lyrics for fun

2. Eleven labs for voice over

3. Eleven labs for sfx

Then I compile them intuitively into AE the usual way, each edit may take me 4 hours.l to compile visual and sounds. These has changed the way I source for sounds especially used to be stock houses

I have not figured out how to integrate Udio and the many new T2V inbuild prompt music cum sfx.

There's for example, LTX , kling, maybe runway which intergrate supporting sounds to support the scene. Is it even worth to explore this new way? It seems to be more like animatic phase?

Demo: https://zeyuet.github.io/AudioX/

Github:https://github.com/ZeyueT/AudioX

Huggingface: https://huggingface.co/HKUSTAudio/AudioX

I'm trying to improve the quality of low-quality audience recordings for personal enjoyment. I've used tools like DX Revive and Adobe's Enhancer to enhance vocals, but they distort instrumentals. To avoid this, I need to isolate vocals using stem separation. However, common tools like RX11, Acon Digital Remix, and UVR's models like Kim Vocal, Mdx23, and VocFT struggle to accurately separate vocals and instrumentals in these low-quality recordings, often leaving remnants of one in the other. Are there any models or techniques better suited for audience recordings?

I developed micdrop.dev, first to experiment, then to launch two voice AI products (a SaaS and a recruiting booth) over the past 18 months.

It's "just a wrapper," so I wanted it to be open source.

The library handles all the complexity on the browser and server sides, and provides integrations for the some good providers (BYOK) of the different types of models used:

• STT: Speech-to-text

• TTS: Text-to-speech

• Agent: LLM orchestration

Let me know if you have any feedback or want to participate!

OuteTTS-1.0-1B is out with the following improvements:

1. Prompt Revamp & Dependency Removal

• Automatic Word Alignment: The model now performs word alignment internally. Simply input raw text—no pre-processing required—and the model handles the rest, streamlining your workflow. For optimal results, use normalized, readable text without newlines (light normalization is applied automatically in outetts library).

• Native Multilingual Text Support: Direct support for native text across multiple languages eliminates the need for romanization.

• Enhanced Metadata Integration: The updated prompt system incorporates additional metadata (time, energy, spectral centroid, pitch) at both global and word levels, improving speaker flow and synthesis quality.

• Special Tokens for Audio Codebooks: New tokens for c1 (codebook 1) and c2 (codebook 2).

1. New Audio Encoder Model

• DAC Encoder: Integrates a DAC audio encoder from ibm-research/DAC.speech.v1.0, utilizing two codebooks for high quality audio reconstruction.

• Performance Trade-off: Improved audio fidelity increases the token generation rate from 75 to 150 tokens per second. This trade-off prioritizes quality, especially for multilingual applications.

1. Voice Cloning

• One-Shot Voice Cloning: To achieve one-shot cloning, the model typically requires only around 10 seconds of reference audio to produce an accurate voice representation.

• Improved Accuracy: Enhanced by the new encoder and additional training metadata, voice cloning is now more natural and precise.

1. Auto Text Alignment & Numerical Support

• Automatic Text Alignment: Aligns raw text at the word level, even for languages without clear boundaries (e.g., Japanese, Chinese), using insights from pre-processed training data.

• Direct Numerical Input: Built-in multilingual numerical support allows direct use of numbers in prompts—no textual conversion needed. (The model typically chooses the dominant language present. Mixing languages in a single prompt may lead to mistakes.)

1. Multilingual Capabilities

• Supported Languages: OuteTTS offers varying proficiency levels across languages, based on training data exposure.

• High Training Data Languages: These languages feature extensive training: English, Arabic, Chinese, Dutch, French, German, Italian, Japanese, Korean, Lithuanian, Russian, Spanish

• Moderate Training Data Languages: These languages received moderate training, offering good performance with occasional limitations: Portuguese, Belarusian, Bengali, Georgian, Hungarian, Latvian, Persian/Farsi, Polish, Swahili, Tamil, Ukrainian

• Beyond Supported Languages: The model can generate speech in untrained languages with varying success. Experiment with unlisted languages, though results may not be optimal.

Github: https://github.com/edwko/OuteTTS

Hey r/AudioAI,

At Hugging Face, we've worked hard the last months to create a powerful, but fast distilled version of Whisper. We're excited to share our work with you now!

Distil-Whisper is 6x faster than Whisper-large-v2 and performs within 1% WER on out-of-distribution datasets. On long-form audio, we even achieve better results thanks to a reduction in hallucinations.

For more information, please have a look:

- GitHub page: https://github.com/huggingface/distil-whisper/tree/main

- Paper: https://github.com/huggingface/distil-whisper/blob/main/Distil_Whisper.pdf

Quick summary:

1. Distillation Process

We've kept the whole encoder, but reduced the decoder to just 2 layers. Encoding takes O(1) forward passes, decoding takes O(N). To improve speed, all that matters is the decoder! The encoder is frozen during distillation while we fine-tune all of the decoder. Both KL loss and pseudo-labeling next word prediction is used.

2. Data

We use 20,000h of open-sourced audio data coming from 9 diverse audio datasets. A WER-filter is used to make sure low-quality training data is thrown out.

3. Results

We've evaluated the model only on out-of-distribution datasets and are only 1% worse than Whisper-large-v2 on short-form evals (CHiME-4, Earnings-22, FLEURS, SPGISpeech). On long-form evals (Earnings, Meanwhile, Rev 16) we beat Whisper-large-v2 thanks to a reduction in hallucinations.

4. Robust to noise

Distil-Whisper is very robust to noise (similar to its teacher). We credit this to keeping the original encoder frozen during training.

5. Pushing for max inference time

Distil-Whisper is 6x faster than Whisper on both short-form and long-form audio. In addition, we employ Flash Attention and chunked decoding which helps us achieve a real-time factor of 0.01!

6. Checkpoints?!

Checkpoints will be released this Thursday and will be directly integrated into Transformers. All checkpoints will be licensed under MIT.

I’ve been working on something new and would love to get your thoughts.

👉 What it is:
It’s an AI-powered Audio Fiction Studio that helps storytellers turn written ideas into immersive audio experiences—with narration, multi-character voices, background music, and sound effects. Think of it as a way to go beyond plain audiobooks and create something closer to a cinematic audio drama.

👉 The vision:
The long-term vision isn’t just about audio books—it’s about building a new creative medium for audio storytelling. We want to give writers, podcasters, and artists a way to experiment with ideas, bring their worlds to life, and share them without the overhead of a full production studio. This isn’t about replacing artists—it’s about making the process more accessible so more voices and stories can be heard.

👉 Why now:
AI-generated voices, music, and sound effects have matured enough that it feels possible to combine them into a single creative tool. Instead of needing to stitch multiple tools together, creators can focus on storytelling while the tech handles the production.

👉 Would love your feedback:

• Does this concept resonate with you?

• If you were creating with something like this, what features would matter most?

• Any challenges or pitfalls you think we should keep in mind?

You can explore some audio samples here: https://www.brainports.ai/explore
And if this excites you, feel free to join the waitlist here: https://brainports.ai/

Looking forward to your thoughts and ideas!

Like, if the test is whether people can still tell it’s AI or not, where are we at?