GAN architecture explained

28 Jan 2018

first, an overview of GAN in more technical detail (pretty sure you know how things work by now on a high level overview): https://danieltakeshi.github.io/2017/03/05/understanding-generative-adversarial-networks/

Code version of a standard GAN, in Keras: https://github.com/wayaai/GAN-Sandbox/blob/master/gan.py

then, the different GANs we explore:

Vanilla GAN to DCGAN (Deep Convolution GAN)

http://guimperarnau.com/blog/2017/03/Fantastic-GANs-and-where-to-find-them#dcgans

DCGAN to WGAN (Wasserstein GAN) https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html

and then, let’s go down into each of the individual parts in more detail below:

Batch Normalisation

For a summary, see the extract from paper by Sergey Ioffe, Christian Szegedy https://arxiv.org/abs/1502.03167

Next, for a simpler breakdown, see this awesome explanation by Karl N. here: https://gab41.lab41.org/batch-normalization-what-the-hey-d480039a9e3b

Cross Entropy

Amazing and simple explanation by Rob DiPietro here: https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/

Very helpful to first understand entropy, then cross entropy, then KL divergence.

RMSProp http://ruder.io/optimizing-gradient-descent/index.html#rmsprop

tanh function

Hyperbolic tangent function is essentially rescaling the sigmoid function from its range of 0 to 1, to -1 and 1. There is horizontal scaling as well.

https://brenocon.com/blog/2013/10/tanh-is-a-rescaled-logistic-sigmoid-function/

kernel initializer Initializations define the way to set the initial random weights of Keras layers. https://faroit.github.io/keras-docs/1.2.2/initializations/

he_normal: Gaussian initialization scaled by fan_in (i.e. its number of inputs)