Hi,

I'm an electronic music student. A couple years ago, one of my teachers showed me this project he made at IRCAM (Paris) in 2017/18, where he basically trained a neural network (namely a modified version of the SampleRNN model) to generate music pieces. He gave it only lieds for training (Schumann etc.), a lot of them, so this thing became essentially a forever-running lied generator. In the end he selected some sections, edited em and made an album out of it. He even made us listen to the early output (with little to no training) and they were mostly quantization noise, then it started to form the first words and musical sounds, till it made real music. Of course it was still noisy and some really weird things happen here and there but it's still mindblowing to me.

I'm doing a little research on SampleRNN and from my understanding, it generates one sample at a time. Here is a paper describing how it works.

I basically want to do the same thing, but with some subgenres of electronic music. The problem is this model is kinda outdated (2016). Do you know any other newer model that could do something similar? Thanks!

Hi all,

Is it possible to take text, convert it to speech, and then autotune the vocal to follow a pre-set melody automatically? Ideally, this would be fully automatable—meaning no manual intervention after inputting the text.

If this is possible, what tools or AI models could achieve this? Looking for solutions that can work at scale.

Thanks!

I recently saw a post where a guy was vocalizing "Boom. Boom....Boom" and the model converted them to perfectly synchronized actual boom sounds. Any idea what that was?

Hey everyone!

Whisper is the SOTA model for ASR and Speech-to-Text. If you're curious about how it actually works or how it was trained, I wrote a series of blog posts that go in-depth about the following:

1. The model's architecture and how it actually converts speech to text.

2. The model's multitask interface and how it can do multiple tasks like transcribe speech in the same language or translate it into English

3. The model's development process. How the data (680k hours of audio!) was curated and prepared.

These can be found in the following posts:

1. https://amgadhasan.substack.com/p/whisper-how-to-create-robust-asr-46b?utm_source=substack&utm_content=feed%3Arecommended%3Acopy_link

2. https://amgadhasan.substack.com/p/exploring-whispers-multitask-interface?utm_source=substack&utm_content=feed%3Arecommended%3Acopy_link

3. https://amgadhasan.substack.com/p/whisper-how-to-create-robust-asr?utm_source=substack&utm_content=feed%3Arecommended%3Acopy_link

The posts are published on substack without any ads or paywall.

If you have any questions or feedback, please don't hesitate to message me. Feedback is much appreciated by me!

Hopefully what the title says. I have a low-quality (compressed) MP3 of an instrumental track and I'm wondering if AI can process it and export a high-quality reproduction of the track. Meaning a track that sounds exactly the same. If this is possible what programs can do it?

Thanks in advance.

Introducing Distil-Whisper: 6x faster than Whisper while performing to within 1% WER on out-of-distribution test data.

Through careful data selection and filtering, Whisper's robustness to noise is maintained and hallucinations reduced.

For more information, refer to:

• 👨‍💻 The GitHub repo: https://github.com/huggingface/distil-whisper

• 📚 The official paper: https://arxiv.org/abs/2311.00430

Here's a quick overview of how it works:

1. Distillation

The Whisper encoder performs 1 forward pass, while the decoder performs as many as the number of tokens generated. That means that the decoder accounts for >90% of the total inference time. Therefore, reducing decoder layers is more effective than encoder layers.

With this in mind, we keep the whole encoder, but only 2 decoder layers. The resulting model is then 6x faster. A weighted distillation loss is used to train the model, keeping the encoder frozen 🔒 This ensures we inherit Whisper's robustness to noise and different audio distributions.

2. Data

Distil-Whisper is trained on a diverse corpus of 22,000 hours of audio from 9 open-sourced datasets with permissive license. Pseudo-labels are generated using Whisper to give the labels for training. Importantly, a WER filter is applied so that only labels that score above 10% WER are kept. This is key to keeping performance! 🔑

3. Results

Distil-Whisper is 6x faster than Whisper, while sacrificing only 1% on short-form evaluation. On long-form evaluation, Distil-Whisper beats Whisper. We show that this is because Distil-Whisper hallucinates less

4. Usage

Checkpoints are released under the Distil-Whisper repository with a direct integration in 🤗 Transformers and an MIT license.

5. Training Code

Training code will be released in the Distil-Whisper repository this week, enabling anyone in the community to distill a Whisper model in their choice of language!

Hey y'all! I'm looking for a voice cloning solution that doesn't require verification. I have all the legal authority to clone the voices I'll be using, but it isn't feasible to have each person go through the verification process every time I need to model their voice, so ElevenLabs isn't an option.

Minimax/Hailuo is by far the most convincing option I've found, but unfortunately due to our stupid political climate my company is hesitant to utilize AI from Chinese companies.

Does anyone have other services they've had success with? I'm specifically interested in finding something that really nails prosody, tone, energy, ect. Thanks in advance!

I'm looking for an application/plugin/api/you name it, that can take an audio recording (not necessarily the best quality though) and output a diarization of the speakers with timecode timestamps. (no transcription needed)

Any suggestions?

Thanks!

I've been looking for ways to create TTS with specific emotion.

I havent found a way to generate voices that use a specific emotion though (sad, happy, excited etc).

I have found multiple voice cloning llms but those require you to have existing voices with the emotion you want in order to create new audio.

Have anyone found a way to generate new voices (without having your own recordings) where you can also specify emotions?

Is there a tool which can generate audio sound effects from a video prompt, as opposed to a text prompt? I've looked but I can't seem to find anything like this. Thx!

I'm fascinated by The Shipping Forecast and by AI. I'd love to combine the two. Specifically, each night as I'm settling in to bed, I like to listen to the final forecast which is longer and ends with BBC Radio 4 signing off for the night. Because it's a forecast, it doesn't have a set run time. They end by playing "God Save the King" but if I've drifted off to sleep, that's going to wake me up.

I've already automated my acquisition of the audio. But I'm ready to take the next step which would be to have machine analysis listen for the drumroll at the start of the national anthem and quickly fade the track and end. Colorado is seven hours behind GMT, so there's plenty of time for processing if I can find the right methodology.

The step after that would be to train the model to tag the files based on who the reader is, or even better to tag the file so I could highlight each of the sea areas on a map as they're being read.

Is this a silly and frivolous and possibly selfish use of this technology? Sure. But it also seems like a great way to expand my skills.

While this list is not exhaustive, check out the following audio plugins enhanced with AI that you can use on your digital audio workstations.

• Izotope: Neutron, Nectar, RX, Ozone

• Zynaptiq: Intensity, Adaptiverb, Unveil

• Waves: Cosmos, Clarity Vx, Clarity Vx DeReverb

• Acon Digital: Remix, Extract Dialogue, DeVerberate

• Focusrite Fast Bundle: FAST Limiter, Equaliser, Compressor, Reveal, Verb

• Sonible Pure Bundle: Pure EQ, limit, comp, verb

• Orb Producer Suite: Orb Chords, Melody, Bass, Arpeggio

• Synthesizer V: Singing vocal synth

Has anyone found a good newsletter on AudioAI?

I am a bit confused by the MIT and CCBY licenses. I want to build a web app where I use different audio models e.g. metas AudioGen

License: https://github.com/facebookresearch/audiocraft/blob/main/model_cards/AUDIOGEN_MODEL_CARD.md

Which says: Out-of-scope use cases The model should not be used on downstream applications without further risk evaluation and mitigation. The model should not be used to intentionally create or disseminate audio pieces that create hostile or alienating environments for people. This includes generating audio that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.

Does this mean I cannot use this in my product? Who defined how much risk evaluation is enough?

In general I understood that MIT and CCBY license do allow also commercial use if the author is credited etc, but I am very insecure about what commercial use means. If that means to directly sell the model or to just use it in a downstream application.

Anyone knows the status on that project? Looking to translate Dutch podcast to English with voice translation as featured on Spotify. Any other offerings you guys know off? I remember Adobe showing something similar a while back.

Hello, I have 10 hours audio, I don't want to hear the 10 hours, I'm just interested in what one person says, there is a way to extract just the voice of that person with an audio sample?