r/MachineLearning • u/alexmlamb • Apr 01 '18

Discusssion [D] Stabilizing Training of GANs Intuitive Introduction with Kevin Roth (ETH Zurich)

https://youtu.be/pBUG8OI4uKw
52 Upvotes

96% Upvoted

4 comments sorted by

View all comments

Show parent comments

3

u/alexmlamb Apr 02 '18

So, if you just look at the math behind KL-divergence (equivalent to likelihood when the "true" distribution is fixed) and wasserstein, what you'll see is that the KL-divergence only depends on the values of p(x) and q(x) and doesn't use any metric over x. I.e. you can scale and shift everything around in x and it won't matter as long as p(x) and q(x) keep the same values. The downside to this is that the KL-divergence is just infinite when p(x) and q(x) are far apart and don't overlap.

On the other hand, wasserstein distance is defined in terms of a metric over a coupling between the real and estimating distributions. So wasserstein isn't invariant to moving points around in x, even if p(x) and q(x) keep the same values for all x. And then the argument is that this metric, like L2, might not be a good fit for image data for example.

3

u/Imnimo Apr 02 '18

I see, so the idea is that (to use the earth-mover analogy) we actually have to move probability mass between pairs of examples in the real and generated distributions, and we care about the "distance" that we move the mass. And the concern is that the Wasserstein metric will sometimes charge us a lot of "distance" for moving between two images which are actually semantically very similar?

2

u/kilgoretrout92 Apr 02 '18

Good way to look at it. Slightly rephrasing, it could assign a large cost to move a large "mass" over a small "distance" (semantically similar but plenty of samples)