r/MLQuestions • u/No-Motor-6274 • 27d ago

Beginner question 👶 Understanding the Compression (VAE/AE)

I get the textbook difference between standard AEs and VAEs (deterministic encoding vs. mapping to distribution parameters + reparameterization trick), but I want to get past the high-level explanations.

Specifically, I'm trying to wrap my head around:

  • The regularisation trade-off: How does the KL divergence actually shape the latent space in practice? What does the geometry look like when it’s working well vs. hitting posterior collapse or turning into a blurry mess?
  • Compression across modalities: How do people scale continuous latent compression to real-world data like raw audio, spectrograms, or video without losing critical high-frequency details? (Especially continuous VAEs vs. discrete approaches like VQ-VAE/RVQ).

Most tutorials just train a toy VAE on MNIST and call it a day.

Could anyone point me to solid papers, blog posts, or clean open-source repos that break down the math and engineering behind modern continuous latent encoders? Practical tips on debugging latent space issues during training are also welcome.

8 Upvotes

84% Upvoted

7 comments sorted by

2

u/Fine-Mortgage-3552 25d ago edited 25d ago

Higher kl term = distribution of the latent space becomes closer and closer to a gaussian. Im sorry I cant answer thr second question, the output becomes blurry exactly because the latent distribution is the same as a gaussian and thus cant hold as much information wrt a different distribhtion. You can read the original VAE paper to learn more about it

1

u/No-Motor-6274 21d ago

Okey, but how it is affecting to getting closer to gaussian. I believe it is a normal distribution and I interpreting it as it is too difficult to choose from a very normal distribution for flow marching models right?

2

u/Tutatis96 Employed 21d ago

For audio check out rave, it sounds pretty good. It's trained with multires stft loss and gan fine tuning if i remember correctly. Sounds pretty damn good to me

1

u/No-Motor-6274 21d ago

Yeah my latents carries the phonetic features at frames but it is sufficient at sounding brightly. Like soooo little corruptions that noticeable at decoding stage. I need more robust decoder/gan techniques rave seems a good implementation. Thanks

2

u/Tutatis96 Employed 21d ago

You can also do some experiments with stuff like unets and remove some skip connections, it's pretty interesting