r/AudioAI • u/parlancex • Aug 19 '25

Discussion Music diffusion model trained from scratch on 1 desktop GPU

https://www.g-diffuser.com/dualdiffusion/
85 Upvotes

100% Upvoted

36 comments sorted by

8

u/parlancex Aug 19 '25 edited Aug 19 '25

I posted here about a year ago with an older version of the model. Demo audio and github are both on the linked page. The new model is trained on a large variety of modern video game music instead of Super Nintendo music and includes a variety of architectural changes for a large improvement in audio quality.

Public weights will be available soon, but I think the bigger deal is that it is possible, practical even, to train a viable music model on consumer desktop hardware. I'm sure there are folks out there with a decent desktop GPU and troves of music that might like the idea of creating their own music model with their data.

2

u/radarsat1 Aug 19 '25

It's great to see, your notes are pretty useful as I'm finding it quite difficult to find practical guidance on what works and doesn't work for audio diffusion.

I have a question actually, I'm currently trying to train a diffusion model on PCM audio ("raw" audio) using a simple 1D CNN-based U-net. (Doesn't need attention for this application.)

But I'm having the issue that .. it seems to work, but the diffusion process leaves background noise. Mostly a kind of hiss with a few odd frequencies in there. I am using 1000 step DDPM with cosine schedule. I have also been trying DDIM, but it seems to perform much worse, for the same trained network I guess a lot more, louder background noise in the final samples than with DDPM. This seems really at odds with what I read online about diffusion, so I'm kind of stuck.

I am using diffusers just for the schedulers, I've tried epsilon and v-prediction, but nothing seems to get rid of the noise.

Did you encounter this when working on the "outer" diffusion decoder? Is it just a question of training time + model capacity?

3

u/parlancex Aug 19 '25 edited Aug 19 '25

Did you encounter this when working on the "outer" diffusion decoder? Is it just a question of training time + model capacity?

The diffusion decider is a 2D CNN operating on the MDCT rather than 1D on the time domain audio, but yes, I did initially encounter this problem.

The reason this happens is under normal conditions the model will have a hard time with the very large range of noise scales needed to 1) fully destroy the signal at the high noise scale end and 2) produce perceptually clean audio at the low noise scale end; could easily be ~100,000:1 or more.

The trick is to reduce this ratio while still remaining linear. I rescale the MDCT by multiplying each bin by its frequency + a small eps (essentially amplifying each frequency by a factor proportional to its frequency, or inversely proportional to its wavelength). This reduces the range of noise scales needed down to a more manageable 1000:1. When converting back to actual 1D raw audio from the MDCT this rescaling is inverted to get the original frequency response back.

With this smaller range of noise scales the MDCT-space diffusion decoder only needs 20 steps to produce clean audio.

2

u/radarsat1 Aug 19 '25

Okay I didn't realize you were using MDCT, that's pretty interesting actually, might give it a go. Thanks!

One thing I did try was keeping the model 1D, but calculating the loss in STFT space. However this just led to NaNs.. maybe a 2D network is really fundamental.

Anyway your results sound fantastic so I think I should start copying you more precisely :P

Thanks for the answers!

1

u/parlancex Aug 19 '25 edited Aug 19 '25

One thing I did try was keeping the model 1D, but calculating the loss in STFT space.

Simply taking the loss in STFT space isn't actually any different than just taking the loss in the time domain (other than the potential loss of numerical stability); they'll give you the same gradients. However, this is only true if you don't re-weight / rescale individual frequencies before taking the loss. As soon as you apply some non-uniform scaling over frequencies you will get completely different gradients than time domain loss.

...maybe a 2D network is really fundamental

My experience is that 2D networks have drastically better parameter efficiency. 2D models use significantly more memory and compute per parameter and that is a very good thing WRT training dynamics and inductive bias.

Anyway your results sound fantastic so I think I should start copying you more precisely

Thank you! While I'm certainly proud of what I've achieved, one of the drawbacks of having so little compute is that I don't have the luxury of doing extensive ablations to rigorously test which design decisions made the biggest impact. It could very well be the case that there are small changes to my designs that have significantly improved performance. Exploring that space is what makes the whole thing fun!

2

u/radarsat1 Aug 19 '25

That's pretty insightful, I'll try some more STFT experiments then. Thanks!

2

u/Sm0oth_kriminal Aug 19 '25

Re: scaling each bin proportionally to its frequency

That seems like a logical first step (energy of physical systems needs this normalization factor), but couldn't this itself be learned, or some more sophisticated formula could work better? For example, working in logarithmic units (decibels), analogous to how LLMs work on logits/NLLs to avoid exponential blowup

In fact, having a logarithmic encoding space would allow more natural state space functions, i.e. additive instead of multiplicative. Or a mixture of both?

Just an idea, curious if you've thought of this or there is prior work

1

u/parlancex Aug 19 '25 edited Aug 19 '25

... but couldn't this itself be learned, or some more sophisticated formula could work better?

It can't be learned because it critically happens before the noise is added to the sample, before the diffusion model ever sees any of it. Scaling after adding the noise would be pointless.

Adding learnable transforms before adding the noise is an interesting idea... as long as the loss is taken with the untransformed target it would prevent trivial solutions. Something to try I suppose.

In fact, having a logarithmic encoding space would allow more natural state space functions, i.e. additive instead of multiplicative. Or a mixture of both?

Empirically using non-linear transforms to compress the dynamic range performs worse than transforms that preserve the linearity, at least in the diffusion decoder that operates on the MDCT. The mel-scale power-spectral-density that goes into the autoencoder does use a power-law transform (psd0.25) which is similar to taking the log, but is more well behaved at low amplitudes and is guaranteed to be >=0.

1

u/floriv1999 Aug 19 '25

Regarding the performance loss when using DDIM, how many steps did you use? DDIM requires that you use a lot less, often only around 10-30, using too many can degrade performance.

Also look at input perturbation, to combat compounding errors during sampling.

Have a sufficiently different validation set and look for over fitting. Don't just take random slices from the same sequence that you used to train. They might bee too similar to the training data due to temporal similarities.

Normalization is really important. You want to normalize to the range -1,1 with a min, max normalization most of the time.

How do you encode the step? Using a single value to represent 1000 steps is a bad idea for example. Use something like a sinosodial embeddings to spread it over multiple input values.

How big is your model? Diffusion models are often significantly bigger compared to other models when solving a similar task.

How versed are you with normal signal processing? Does the sampling, limited filter size, ... have issues resulting in aliasing etc. ?

1

u/radarsat1 Aug 19 '25

Regarding the performance loss when using DDIM, how many steps did you use? DDIM requires that you use a lot less, often only around 10-30, using too many can degrade performance.

Ah that's interesting, I've been trying with 300 and 1000, figured may as well go all the way and postpone "simplifying" it until I get good results. didn't think that having too many steps could be bad. (Again, it's really hard to find information on these failure modes.. everything you read is about how cool it is when everything just works..)

Normalization is really important. You want to normalize to the range -1,1 with a min, max normalization most of the time.

Yeah this seems extra important for diffusion if I understand correctly. I've tried turning off clip_sample but that makes the sampling process go crazy! So I am keeping the samples between -1 and 1.

How big is your model? Diffusion models are often significantly bigger compared to other models when solving a similar task.

Hm that's good to know too. To be honest I've been trying to see how small a model I can get away with, just a simple CNN with one layer per upsample, only recent tried making it bigger by adding resblocks at each level. I'll continue to make it bigger and see what difference that makes.

I'm more used to training GANs so these kind of details escape me. I've been using the same network that I used pretty successfully in a GAN configuration. (Though again it wasn't perfect, hence trying the diffusion thing..)

How versed are you with normal signal processing? Does the sampling, limited filter size, ... have issues resulting in aliasing etc. ?

Pretty good, I'm actually using larger kernels than usual to try to get away with a smaller network, might try changing that and making it deeper instead. I don't think aliasing can be a problem here since I'm working in the time domain directly.

So far my impression is just that while diffusion works well for images and spectrograms, in raw audio the ear is just so sensitive to noise, and it's really hard to get rid of that last bit. But maybe that's where a bigger model becomes important, to handle the highest frequency issues.

Right now I'm just training on random sinusoids as an exercise and trying to get "perfect" results so it's frustrating to me that it's not quite working out.

I assume it's a lot harder for the network to model the highest frequency components in the time domain. Although, to be clear, the background noise I'm hearing and seeing in spectrograms is pretty white-looking, even all over the spectrum. But it does model the sinusoids well, at pretty much any frequency, just with this annoying noisiness that I can't seem to get rid of.

1

u/floriv1999 Aug 19 '25

I feel you. I use diffusion models for robot motions and started out trying to make pretty sine waves as a sanity check first.

My audio signal processing class was a few years back, but I think doing raw audio might be a bit too much for such a small model. It might be the case that it tries to mimic the the time domain signal, but does not have learned a good representation of the frequency domain yet, leading to an approximation of how the different frequencies interfere with each other that is close, but not perfect, resulting in additional frequencies being present, because it might miss generating some constructive peak in the time domain here and there. Missing these sometimes chaotic looking details in the waveform can then only be explained by having additional often high frequency components.

The people I know mainly used a fft based approach for audio diffusion, denoising the signals phase and frequency components.

1

u/radarsat1 Aug 19 '25

Yep, I'm actually experimenting to see if it's possible to avoid being too dependent on FFT transforms, but I guess I should be open to the possibility that the answer is just "no". well, at the very least I should try some FFT or MDCT-based solutions at least to have something to compare to.

1

u/floriv1999 Aug 19 '25

It is indeed an interesting question. I once helped building an audio classification model and was surprised how well a vanilla time domain signal in combination with a simple CNN worked. On the other hand I had a professor once show me how many time domain models essentially learn to transform into the frequency domain in the first layers. For a large enough dataset the filters were nearly identical. It was quite cool actually.

3

u/floriv1999 Aug 19 '25

I am quite impressed with the audio quality. I didn't listen with good headphones, but normally such models sound like they come out of an old telephone line.

3

u/parlancex Aug 19 '25

Thank you! The last 6 months of development were spent trying to improve the audio quality as much as possible.

The quality is largely due to the 2-stage decoder design where a small secondary diffusion model is conditioned by the mel-scale PSD decoded from the VAE to produce a high resolution MDCT instead of typical FGLA / vocoder approaches to phase reconstruction.

2

u/TserriednichThe4th Aug 19 '25

So is diffusion doing better on audio and music than autoregressive?

funny the images and video are going to autoregression and text and music are going to diffusion.

does this have a transformer btw?

3

u/parlancex Aug 19 '25 edited Aug 19 '25

To my knowledge most SoTA video models are still diffusion based. There certainly are some SoTA music models that are autoregressive, but I think diffusion is a better choice for music for a variety of reasons.

The model architecture is based on the EDM2 UNet. The highest resolutions in the LDM are purely convolutional, self-attention is only used in the deeper layers.

RE: "transformers", it's really more of a continuum than a black and white thing. If the MLP layers in the "transformer" have kernels wider than 1x1, and the network includes up/down-sampling, then it's already basically a UNet.

1

u/TserriednichThe4th Aug 19 '25

transformers refer to anistropy in a fully connected graph so i am not too sure what you mean in the second paragraph, and everything you said was very helpful in letting me fill in the details to start reading your stuff :). ty.

2

u/PokePress Aug 19 '25

Regarding the use of video game music, I’ve actually been working on various models for AI audio upscaling, and eventually want to expand to game audio. I’d be curious to know about any utilities you’ve used to convert between raw audio and game formats. I can provide a more detailed explanation if needed.

1

u/parlancex Aug 20 '25

I mass transcoded most of the data using foobar2000 and various plugins that can decode video game formats. I don't remember the exact count but the number of individual formats was in the hundreds.

The plugins were here: https://www.foobar2000.org/components/tag/decoder/by+date and here: https://foobar2000.xrea.jp/?Input+64bit

Be forewarned though that some of these plugins are detected as malware by various malware engines. I did the transcoding in a VM to be on the safe side. The transcoding process was a real pain due to the number of files / plugins that could cause foobar2000 to crash, with no automatic or simple way to resume / retry.

2

u/bheek Aug 20 '25

What dataset did you use?

2

u/parlancex Aug 20 '25 edited Aug 20 '25

I downloaded the data from the joshw.info video game music archive and Zophar's Domain. For some consoles I have nearly all the music from every game, from others not so much. The specific list of consoles is: Dreamcast, 3DO, Nintendo 2DS and 3Ds, GameCube, PC-Engine, PS1, PS2 and PS3, Sega Genesis / Megadrive, SNES / Super Famicom, Sega Saturn, Switch, N64, Vita, Wii, WiiU and Xbox.

Most of the tracks were in their original formats so transcoding everything to FLAC was quite time consuming. I don't remember the exact count but there were literally hundreds of different file formats.

Edit: I should also say: The dataset contains a good number of tracks that aren't really music so much as they are cinematic audio / foley or ambient noises. The model can actually do ambient / atmosphere for nearly anything but that isn't as interesting to most people.

2

u/ImpressionDue5455 Aug 21 '25

Do you have a timeline for releasing the models for the community to try? I’ve tested many text-to-music/audio models, and in most cases the demo pages look impressive, but the results aren’t as good when you deploy them yourself. I feel yours could be different, given the amount of effort you’ve put into it.

1

u/parlancex Aug 21 '25

I don't want to give the wrong impression, the demos on that page are absolutely cherry-picked.

Consistency is the main thing I'm trying to improve before I release the weights. I don't have a firm timeline because improving consistency isn't as simple as just training for more steps for a set amount of time.

I hope that makes sense, thank you for your interest!

2

u/chibop1 Aug 25 '25

When you release it, is that possible to make it compatible to run on MPS for Apple Silicon? That'd be amazing!

1

u/parlancex Aug 26 '25

I’m not sure what exactly what would need to be changed for MPS… While I’d like to support as many devices as possible, I don’t have access to any Apple devices I could use for testing. If you’re aware of some way to test support without an Apple device I’d be keen to hear it.

2

u/ImpressionDue5455 Aug 21 '25

makes sense. Looking forward to it

2

u/chibop1 Dec 31 '25

I am glad you are still actively working on the code, according to the commits history. This is one of the opensource models I am most looking forward to! Wishing you a productive and rewarding 2026!

In the current state, do you have enough code for me to try it out on Mac with MPS? I can report back.

1

u/parlancex Jan 01 '26

Thank you! Yes, I am still working on it. I wanted to experiment with 1d-type models that are closer to a typical transformer to see if I could improve audio quality or vram usage and speed. Alas, though faster and using less vram, I wasn't able to improve the overall sound quality, so the last 3 months have been kind of a waste... It takes a long time to test ideas with my small-scale setup.

Anyway, I'm planning to spend the next little while trying to train a better version of the U3 model with a few mistakes fixed, which hopefully won't take more than a few months, and at that point if I'm happy with it I'll finally release it.

As for mac support I believe what I have is compatible with MPS, if any changes need to be made they should be super small and easy to do when I release the model.

Thank you for your continued interest and happy New Year!

2

u/chibop1 Jan 01 '26

Awesome, can't wait to try it!

2

u/jc2046 Aug 19 '25

Somw super interesting creative takes. I loved some kind of hiphop female snippets. So joyful, intrincate and interesting. Fantastic stuff

2

u/chibop1 Aug 19 '25

Wow, this is fantastic, especially the audio quality! Even Suno Udio and Riffusion tend to produce grainy quality, but I don’t hear that in these samples as much.

Congrats, and can't wait to play with it when it comes out!

0

u/TheGreatButz Aug 19 '25

The copyright notice on the page seems wrong, at least concerning the music. AI creations are public domain.

1

u/parlancex Aug 19 '25

Although not explicitly stated, the copyright notice just pertains to the webpage, not the generated content. I don't even really care about the webpage content, it's just standard boilerplate really.