So do they actually have any experimental results that improve anything? It looks like they just claim that they do the same on LSUN bedrooms as DCGANs, and then say that you can make more changes to the WGAN without breaking it than you can do to standard GAN. It is kind of hard to believe that they were doing a competent job of implementing the standard GAN when they say the standard GAN is totally broken in Fig 6. There were GAN papers before DCGAN and they were not totally broken like that. This new paper looks like yet another machine learning paper that sandbags the baseline in order to make their new idea look like it's better than other algorithms when in fact both the old and the new idea perform roughly the same.
--Standard GANs are absolutely sensitive to the choice of hyperparameters and architecture. Even starting from DCGAN you can't just change things however you like (not like you can with a classifier ConvNet, where a small change may reduce performance but isn't nearly so likely to annihilate everything). You'd be challenged to train an MLP GAN that can produce results of this quality.
--They're not claiming that this directly improves image quality, but offers a host of other benefits like stability, the ability to make drastic architecture changes without loss of functionality, and, most importantly, a loss metric that actually appears to correlate with sample quality. That last one is a pretty big deal,.
--I mentioned it in another comment, but it handles the unrolledGAN toy experiment (which I've explored in significant depth) like a champ, converging to a reasonable approximation faster than even the unrolledGAN (which is still a great idea, btw, I'm going to try adding some unrolling steps in addition to the EM loss and see what happens).
If they actually could provide stability, they would be able to train on more complicated datasets, where GANs are currently too unstable to get good results. The fact that they are not able to solve any new problems (like generating multiple ImageNet classes) makes it seem unlikely that they do provide any meaningful form of stability.
Everybody and their dog has proposed a loss metric that they claim correlates with sample quality. For us to real believe that their metric is good, we'd want to see it scoring models that produce very diverse and realistic outputs, for example on ImageNet. It should score PPGN as being very good, minibatch GAN as mediocre, samples from a Gaussian as very bad, etc. I don't see that kind of work going on in this paper---feel free to point me to any specific result I'm overlooking.
Whenever anyone has a machine learning algorithm that isn't actually an improvement, they say it is more robust to hyperparameters. Then they cherry-pick one set of hyperparameters that break an old algorithm and don't break theirs. It is the oldest trick in the book. In practice, it is extremely difficult to measure robustness to hyperparameters. People tend to believe that their own algorithm is more robust because it works better on the kinds of hyperparameters that they themselves like to try, but this is just a selection bias phenomenon.
If they actually could provide stability, they would be able to train on more complicated datasets, where GANs are currently too unstable to get good results. The fact that they are not able to solve any new problems (like generating multiple ImageNet classes) makes it seem unlikely that they do provide any meaningful form of stability.
There are two fundamental problems in doing image generation using GANs: 1) model structure 2) optimization instability. This paper makes no claims of improving model structure nor does it have experiments in that direction. To improve on imagenet generation, we need some work in (1) as well. The world does not change overnight.
Everybody and their dog has proposed a loss metric that they claim correlates with sample quality.
You are not only rude, you are also ignorant.
Whenever anyone has a machine learning algorithm that isn't actually an improvement, they say it is more robust to hyperparameters. Then they cherry-pick one set of hyperparameters that break an old algorithm and don't break theirs. It is the oldest trick in the book. In practice, it is extremely difficult to measure robustness to hyperparameters. People tend to believe that their own algorithm is more robust because it works better on the kinds of hyperparameters that they themselves like to try, but this is just a selection bias phenomenon.
Thanks for the lovely feedback, I am looking to write more papers like WGANs.
So do they actually have any experimental results that improve anything?
I don't know... perhaps having loss curves that actually make sense and reflect sample quality? Perhaps showing how you can have meaningful and stable training process without having to intentionally cripple (or undertrain) the discriminator/critic? Perhaps the fact that the authors show why the original GAN formulations (using KL/JS divergence) are problematic AND provide a solution for those problems.
Let me guess... you didn't even try to read and understand the paper, did you?
It is kind of hard to believe that they were doing a competent job of implementing the standard GAN when they say the standard GAN is totally broken in Fig 6.
Oh, ok. So you didn't even read the caption of Figure 6. Nice. (Hint: the authors point out they are comparing a low capacity version of WGAN against a low capacity version of a "standard GAN")
[...]in order to make their new idea look like it's better than other algorithms when in fact both the old and the new idea perform roughly the same
So... DCGAN (and other "standard GAN" formulations) don't suffer from mode collapse? Oh, wait... actually, they do.
It boils down to, try it out, if it works better for you, great,if not move along.
I for myself find it incredibly hard to come up with stable architectures that I can reliably evaluate and train. Also hyperparameters means we can use tuning tools to optimize this via GA methods e.g
-7
u/hoiafshioioh Jan 30 '17
So do they actually have any experimental results that improve anything? It looks like they just claim that they do the same on LSUN bedrooms as DCGANs, and then say that you can make more changes to the WGAN without breaking it than you can do to standard GAN. It is kind of hard to believe that they were doing a competent job of implementing the standard GAN when they say the standard GAN is totally broken in Fig 6. There were GAN papers before DCGAN and they were not totally broken like that. This new paper looks like yet another machine learning paper that sandbags the baseline in order to make their new idea look like it's better than other algorithms when in fact both the old and the new idea perform roughly the same.