2. 01
Agenda
Problems and Architecture
• Pipeline
• Why Classical CV is still awesome
• Why GANs not the best solution
• HSV vs RGB vs YUV
Other steps
• Super Resolution
• Style Transfer
• Noise & aberration reduction. Why JPEG
is a problem
02
03
Color enhancing
• AutoHDR
• Real-Time photo color enhance
with Bilateral Filter
4. Pipeline
Raw to 16-32bit Image Noise & aberration reduction
Color EnhanceStyle Transfer
Super ResolutionSkin editing / Painting
5. Why Classical CV is still awesome
● White box solution
● Many tasks are based on optical physics and clean mathematics so no NN
needed
● Controlled result which is very important for professional photo editing
● Very fast and can be used almost on any device
● No training and datasets needed
- Some very simple examples of color enhancing with polynomial in my article
- Building shape descriptor of image
- Unwarping example
6. Why GANs not the best solution
● To many uncontrolled artifacts
● Require a lot of computational resources
● Hard to train (Mode collapse, dying gradient, etc)
● Need big datasets
So it’s Money x Money x Money = Money3
And bad for data conversion
7. Why GANs not the best solution
BigGan - 1 training cycle: 15 days, 8xGPUs, ~$8k-$12k
8. RGB vs YUV vs HSV
RGB
- Good for NN
- Bad for classic CV
- Not human readable
YUV
- Good for NN
- So-so for classic CV
- Not human readable
- Efficient coding,
reduce bandwidth
HSV
- Bad for NN
- Good for classic CV
- Human readable
- Based on RGB (not
real colorspace)
11. Auto HDR
Raw - Sensors has different physical grid,
Formats has different strategy for saving data,
Sensors data has to be processed differently.
PAIN
Use 3d party software: pip install rawkit
13. Auto HDR
White Balance - Photographers are wrong, classical CV not always right
Train NN to learn objects that should be
white and use this for your WB settings
14. Auto HDR
Camera Response Function (CRF) - relates scene irradiance to image intensities.
It needed for most HDR transformations
God save OpenCV developers
15. Auto HDR
CLAHE - Adaptive histogram equalisation. Useful algorithm to solve histogram
imbalance of the image. Visually similar to what HDR trying to do
OpenCV, scikit-image and many others
16. Auto HDR
HDR - Basically we fuse few images with different parameters to get good
exposure of every object in the image. Tone Mapping technique used to map 32bit
image to 8bit image(for most devices)
OpenCV
36. Super Resolution - ESRGAN
Changes from SRGAN
- Deeper model using Residual-in-Residual Dense Block (RRDB)
- Use Relativistic average GAN instead of the vanilla GAN.
- Improve the perceptual loss by using the features before activation.
- Pixel Shuffle replaced by Nearest Interpolation
- Pretrain Generator model before use in GAN
- Removed Batch Normalization to remove artifacts
https://arxiv.org/abs/1809.00219
https://arxiv.org/abs/1804.02815
https://github.com/xinntao/ESRGAN
https://github.com/xinntao/SFTGAN
37. Super Resolution
What we can see from this improvements?
- Context and Data is what matter
- All things like BN, Max Pooling which destroy data making worse for generator
- Understanding materials behind the image can greatly improve quality
- Skip connection passing context to the end of network also improve results
42. Style Transfer
What matter
- No MaxPool on Encoder network use strides
- Style poisoning works not so good as we want
- No BatchNorm on Transformer network
45. Adaptive Style Transfer
What matter
- GAN, for such tasks when Loss hard to build
- Multiple Artworks per style
- Additional loss - maybe, hard to tell
https://compvis.github.io/adaptive-style-transfer/