This presentation proposes a novel prior which is derived using basic theorems from probability theory and off-the-shelf optimizers, to improve fidelity of image generation using GANs by interpolating along any Euclidean straight line without any additional training and architecture modifications

1. Rajhans Singh, Pavan Turaga, Suren Jayasuriya, Ravi Garg, Martin Braun
In collaboration with:

2. 2
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
GAN Overview
• GANs trained on images can generate images by mapping a
point in low-dimensional latent space to a point in high
dimensional image space
• Low dimensional latent spaces are also known as prior
distributions or priors
• GANs has shown applications in super resolution, image-to-
image translation, etc.
Discriminator loss
Generator loss

3. 3
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Why Priors are Important?
• Priors choice can help in generating diverse and realistic output
using interpolation in latent spaces
– Interpolation can help transfer certain semantic feature of image to another
[Radford et al.]
– Successful interpolation also shows that GANs don’t overfit and reproduce
training data
• Distribution Mismatch
– Simple parametric latent space distributions like Normal and Uniform suffer
from distribution mismatch issue in interpolation
– The prior distribution does not match the interpolated point’s distribution
Radford, A., Metz, L., and Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. In International
Conference on Learning Representations (ICLR), 2016

4. 4
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Distribution Mismatch Issue
• The GANs trained on the prior points to generate realistic
images, when used to interpolate between points, often loose
fidelity in image quality due to distribution mismatch
‘Ghosty’ faces near the origin during interpolation from left to right on CelebA dataset using Normal prior distribution
Bad fidelity due to distribution mismatch at the center
• The problem worsens as we use higher dimensional latent
spaces
- mid point and prior distributions distance increases

5. 5
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Addressing Distribution Mismatch Issue
Designing better interpolation schemes that traverse through high
probability mass density regions of the distribution
• Geodesic paths [White et al.]; Traversed paths can be complicated and long
By utilizing latent spaces such that the distribution mismatch is minimum,
e.g., Gamma [Kilcher et al.] and Cauchy distribution[Lesniak et al.]
• Cauchy distribution has undefined moments and heavy tails; may lead to undesirable
outputs, training difficulties
We propose a novel non-parametric prior that generates more realistic
and diversified images using interpolated points in the latent space
White, T., “Sampling generative networks” https://arxiv.org/vc/arxiv/papers/1609/1609.04468, 2016
Kilcher, Y., Lucchi, A., and Hofmann, T., “Semantic interpolation in implicit models”, in International Conference on Learning Representations (ICLR),
2018.
Lesniak, D., Sieradzki, I., and Podolak, I., ”Distribution-interpolation trade off in generative models”, in International Conference on Learning
Representations (ICLR), 2019

6. 6
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Proposed Non-parametric Prior
• If 𝑥𝑥1, 𝑥𝑥2~𝑓𝑓𝑋𝑋(𝑥𝑥), then the pdf of the interpolated point: 1 − 𝜆𝜆 𝑥𝑥1 +
𝜆𝜆𝜆𝜆2, for some 𝜆𝜆 𝜖𝜖 0,1 , is given by
𝑄𝑄 𝑥𝑥; 𝜆𝜆 =
1
𝜆𝜆(1 − 𝜆𝜆)
𝑓𝑓𝑋𝑋
𝑥𝑥
𝜆𝜆
∗ 𝑓𝑓𝑋𝑋
𝑥𝑥
1 − 𝜆𝜆
• The goal is to minimize the KL divergence between 𝑃𝑃 𝑥𝑥 =
𝑓𝑓𝑋𝑋(𝑥𝑥) and 𝑄𝑄 𝑥𝑥; 𝜆𝜆
− 𝑃𝑃(𝑥𝑥) is restricted a compact domain, i.e. [0,1] and discretize into
210
bins to obtain a tractable solution
- Variance constraint is added to avoid delta function as solution
min
𝑃𝑃
𝑓𝑓(𝑃𝑃||𝑄𝑄)
𝑠𝑠. 𝑡𝑡. ∑𝑖𝑖=1
𝑛𝑛
𝑝𝑝𝑖𝑖 = 1,
1
𝑛𝑛
∑𝑖𝑖=1
𝑛𝑛
𝑖𝑖2 𝑝𝑝𝑖𝑖 − ∑𝑖𝑖=1
𝑛𝑛
𝑖𝑖𝑖𝑖𝑖𝑖
2
≥ 𝜀𝜀, 𝑝𝑝𝑖𝑖 ≥ 0

7. 7
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Proposed Non-parametric Prior
• Symmetric
- Did not enforce as a constraint; came out as a solution
• Combine two properties from other priors for interpolation
- Sharp decaying – similar to Gamma distribution
- Side lobes – may represent heavy tail structure of Cauchy
Density of the prior and
its midpoint distribution
similar
Distribution KL
divergence
Uniform .3065
Normal .1544
Proposed .0075

8. 8
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
GAN Generated Images
Interpolation (left to right) through the origin on CelebA dataset
 DC-GAN architecture [Radford et. al.’ 2016]
– CelebA dataset
Origin

9. 9
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Measured using standard metrics used for GAN quality
• Inception Score, FID score
Quantitative Results
Distribution CelebA LSUN Bedroom
Inception Score FID Score Inception Score FID Score
Prior Mid-
Point
Prior Mid-
Point
Prior Mid-
Point
Prior Mid-
Point
Uniform 1.843 1.369 24.055 40.371 2.969 2.649 42.998 76.412
Normal 1.805 1.371 26.173 42.136 2.812 2.591 64.682 108.49
Gamma 1.776 1.618 29.912 28.608 2.930 2.808 162.44 161.37
Cauchy 1.625 1.628 59.601 60.128 3.148 3.149 97.057 97.109
Non-
parametric
1.933 1.681 17.735 19.115 3.028 2.769 27.857 31.472
Best performance on CelebA dataset
among all priors tested
Best or close to the best, while more
consistent results at prior point and mid-
point

10. 10
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Conclusions
•We derived a non-parametric approach to search for a prior
which addresses the distribution mismatch problem
•The proposed prior distribution provides better qualitative
and quantitative results as compared to the standard priors
such as Normal and Uniform distributions
•We showed that our proposed prior yields better quality and
diversity in generated output, without any additional training
data and added architectural complexity

11. 11
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Thank You

12. 12
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Non-Parametric Priors For Generative
Adversarial Networks
Rajhans Singh, Pavan Turaga, Suren Jayasuriya,
Ravi Garg, Martin Braun

13. 13
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Backup

14. 14
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Effect of prior dimensionality

15. 15
© Intel Corporation Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Atom, Intel Core, Iris, Movidius, Myriad, Intel Nervana, OpenVINO, Intel Optane, Stratix, and Xeon are
trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.
Euclidean Norm Distribution
Euclidean norm distribution for normal prior
Euclidean norm distribution for the proposed prior
d  overlap in Norm-
distribution
Preserves overlap at higher
dimensions