ICME2019 Tutorial: Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 4: retinex model based low light enhancement
Semelhante a Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 4: retinex model based low light enhancement
Batch normalization: Accelerating Deep Network Training by Reducing Internal ...ssuser6a46522
Semelhante a Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 4: retinex model based low light enhancement (20)
8. STRUCT Group08
Representative Work
Related Works
Histogram
Equalization
Enhance the contrast
Over-enhancement / under-enhancement
Amplify the noise
Before HE After HE
9. STRUCT Group09
Representative Work
Related Works
Histogram
Equalization
Dehazing Method
Inverted low-light images vs. hazy images
Invert dehaze invert again
Require an additional denoising process
Low-Light Inversion Dehazing Result
10. STRUCT Group010
Representative Work
Related Works
Histogram
Equalization
Dehazing Method Retinex Model
Retinex-based methods
Retinex decomposition
Generate results
S R L
1
enhanceS R L
Gamma
Correction
Low-Light Image
Enhanced Image
Illumination (L)
Reflectance (R)
12. STRUCT GroupRelated Works
Retinex model based methods
Retinex Model
Estimate L, R = I/L, refine R
NPE[TIP13] Bright-pass filter preserve naturalness
12
13. STRUCT GroupRelated Works
Retinex model based methods
Retinex Model
Estimate L, R = I/L, refine R
NPE[TIP13]
Fusion based[SP16]
Bright-pass filter preserve naturalness
13
14. STRUCT GroupRelated Works
Retinex model based methods
Retinex Model
Estimate L, R = I/L, refine R
NPE[TIP13]
Fusion based[SP16]
LIME[TIP17]
Bright-pass filter preserve naturalness
14
15. STRUCT GroupRelated Works
Retinex model based methods
Retinex Model
Estimate L, R = I/L, refine R
NPE[TIP13]
Fusion based[SP16]
LIME[TIP17]
Estimate L and R simultaneously
Bright-pass filter preserve naturalness
15
16. STRUCT GroupRelated Works
Retinex model based methods
Retinex Model
Estimate L, R = I/L, refine R
NPE[TIP13]
Fusion based[SP16]
LIME[TIP17]
Estimate L and R simultaneously
PIE[TIP15]
Bright-pass filter preserve naturalness
16
17. STRUCT GroupRelated Works
Retinex model based methods
Retinex Model
Estimate L, R = I/L, refine R
NPE[TIP13]
Fusion based[SP16]
LIME[TIP17]
Estimate L and R simultaneously
PIE[TIP15]
SRIE[CVPR16]
Bright-pass filter preserve naturalness
18. STRUCT GroupRelated Works
Retinex model based methods
Retinex Model
Estimate L, R = I/L, refine R
NPE[TIP13]
Fusion based[SP16]
LIME[TIP17]
Estimate L and R simultaneously
PIE[TIP15]
SRIE[CVPR16]
CEID[TIP17]
Bright-pass filter preserve naturalness
18
20. STRUCT Group020
Representative Work
Related Works
Histogram
Equalization
Dehazing Method
Retinex Model
Learning-Based
Method
Low-Light Image Dataset
Regression Model OutputInput
Low-Light Image Dataset
…
24. STRUCT Group24 Robust Retinex Model for Low Light Enhancement
Robust Retinex Model for Low Light Enhancement
Structure-Revealing Low-Light Image Enhancement Via Robust
Retinex Model
Mading Li, Jiaying Liu, Wenhan Yang, Xiaoyan Sun, and Zongming Guo
TIP 2018
I R L I R L N
Input image Retinex Model Robust Retinex Model
25. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
An additional noise term
Drawbacks of conventional model
Focus on the estimation of L
Got noisy reflectance for
Calculate both R and L iteratively
Introduce noise to illumination by minimizing
25
' /R R N L
2
|| ||FR L S
26. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
An additional noise term
Priors for low-light images
Illumination map piece-wise smoothed
Reflectance map low contrast
Noise map relatively low intensity
26
27. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
The optimization function
27
28. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
The optimization function
Data fidelity term
28
29. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
The optimization function
Input image Illumination map
Illumination constraint
29
30. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
The optimization function
Reflectance constraint
30
31. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
The optimization function
Without constraint With constraint
Reflectance constraint
31
32. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
The optimization function
Noise constraint
Input image w/o constraint w/ constraint Noise map
32
33. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
The solution
Importing an auxiliary variable T
Augmented Lagrange equation
33
34. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
The solution
Sub-problem R
Sub-problem L
Sub-problem N
34
35. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Robust Retinex model
The solution
Sub-problem T
Updating auxiliary variables
35
36. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
PIE[TIP15] SRIE[CVPR16] Proposed
Input image LIME[TIP17]
NPE[TIP13]
36
37. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
PIE[TIP15] SRIE[CVPR16] Proposed
Input image LIME[TIP17]
NPE[TIP13]
37
38. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
Input image LIME[TIP17]
NPE[TIP13] PIE[TIP15] SRIE[CVPR16] ProposedHE
38
39. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
PIE[TIP15]
SRIE[CVPR16] Proposed
Input image LIME[TIP17]
NPE[TIP13]
39
40. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
PIE[TIP15]
SRIE[CVPR16] Proposed
Input image LIME[TIP17]
NPE[TIP13]
40
41. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
Input image LIME[TIP17]
NPE[TIP13] PIE[TIP15] SRIE[CVPR16] ProposedHE
41
42. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
Objective criteria
The lower, the better quality The higher, the better quality
42
51. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
ProposedInput image Fu[ICIP14]
Underwater image enhancement
51
52. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
Smoky/hazy image enhancement
ProposedInput image He[CVPR09]
52
53. STRUCT GroupRobust Retinex Model for Low Light Enhancement
Experimental Results
Enhancement of images taken under dusty weather
ProposedInput image Fu[MMSP14]
53
57. STRUCT GroupSequential Decomposition for Low Light Enhancement
Architecture
In RGB SpaceS
Illumination Estimation
Reflectance Estimation
L
R
S’
𝐿
W G
Restrict Matrices
L’
57
58. STRUCT GroupSequential Decomposition for Low Light Enhancement
Illumination Estimation
2
1
ˆarg min || || || ||F
L
L L L
Low-Light Image Initial Illumination Estimated Illumination
Estimate illumination independent from reflectance
58
59. STRUCT GroupSequential Decomposition for Low Light Enhancement
Reflectance Estimation
Estimate reflectance based on refined illumination
and original image
2 2 2
arg min || / || || || || ||F F F
R
R S L W R R G
Low-Light Image S / L Estimated Reflectance
59
60. STRUCT GroupSequential Decomposition for Low Light Enhancement
Reflectance Estimation
Use weighted matrices to restrict noise
ˆ| |/ ˆ(1 )
0, if | |ˆ
, otherwise
S
G e S
S
S
S
1
| |
W
S eps
G W
60
61. STRUCT GroupSequential Decomposition for Low Light Enhancement
Solution
Estimate the illumination map
Approximate:
Rewrite the original problem:
Simplify:
2
1
x {h,v}
( ( ))
|| || .
ˆ| L( ) |
d
d d
L x
L
x eps
2
2
x {h,v}
( ( ))ˆarg min || || .
ˆ| L( ) |
d
F
L d d
L x
L L
x eps
2 2
x {h,v}
ˆarg min || || ( ) ( ( )) .F d d
L d
L L A x L x
61
62. STRUCT GroupSequential Decomposition for Low Light Enhancement
Solution
Estimate the illumination map
Estimate the reflectance map
{h,v}
ˆDiag( )T
d d d
d
I D a D l l
{h,v} {h,v}
{h,v}
Diag( )
/
T T
d d d d d
d d
T
d d
d
I D w D D D r
s l D g
62
63. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
Experimental settings
All experiments are performed in MATLAB R2017a with
4G RAM and Intel Core i5-4210H CPU @2.90GHz.
In our experiment the parameters α, β and γ are
empirically set as 0.007, 0.001 and 0.016.
In our experiment the parameters ε and σ are set to 10
and λ is set to 6.
63
64. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
LIME (top panel) and ours (bottom panel)
Input images Illumination Reflectance Result images Details
64
65. STRUCT GroupSequential Decomposition for Low Light Enhancement
Experimental Results
LIME (top panel) and ours (bottom panel)
Input images Illumination Reflectance Result images Details
65
74. STRUCT GroupSequential Decomposition for Low Light Enhancement
Summary
Based on a refined Retinex model
Noise-removal and Enhancing
Sequential decomposition
74
78. STRUCT GroupDeep Retinex Decomposition
Hand-Crafted Retinex
Not easy to be adaptive to complex and varying low-light
conditions
Under-enhancementOver-enhancement Boundary artifacts
78
79. STRUCT GroupDeep Retinex Decomposition
Direct End-to-End Learning
Difficulties in directly recovering normal-light images
Inherent ambiguity
Low-Light Image Dataset
Regression Model OutputInput
Low-Light Image Dataset
…
79
80. STRUCT GroupDeep Retinex Decomposition
Direct End-to-End Learning
Regression to mean
Over-smoothed results with degraded contrast
Over-smoothness Degraded contrast
80
81. STRUCT GroupDeep Retinex Decomposition
Our Solution: Retinex-Net
Retinex Theory + Deep Learning
Low-Light Image Dataset
OutputInput
Retinex
Decomposition
Adjusted
Decomposition
Regression
Model
…
81
83. STRUCT GroupDeep Retinex Decomposition
Architecture of Retinex-Net
, ,
recon ij i j j
i low normal j low normal
L R I S
ir low normalL R R
, ,
exp( )is j i g j i
i low normal j h v
L I R
Reconstruction Loss
Constant Reflectance Loss
Illumination Smoothness Loss
83
84. STRUCT GroupDeep Retinex Decomposition
Architecture of Retinex-Net
, ,
exp( )is j i g j i
i low normal j h v
L I R
,
is i
i low normal
L I
Illumination Smoothness Loss
84
89. STRUCT GroupDeep Retinex Decomposition
Real Photography Pairs
LOw Light paired dataset (LOL)
1000 low/normal-light image pairs
500 are collected by changing only exposure time and ISO
Various scenes, e.g., houses, clubs, streets, etc.
89
90. STRUCT GroupDeep Retinex Decomposition
Dataset
Synthetic Pairs from Raw Images
1000 raw images from RAISE[Dang-Nguyen 2015]
Fitting the histogram of Y channel in YCbCr to real low-light images
Online available: https://daooshee.github.io/BMVC2018website/
90
92. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by LIME I by LIME
Normal-Light Image R by LIME I by LIME
92
93. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by NPE I by NPE
Normal-Light Image R by NPE I by NPE
93
94. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by Retinex-Net I by Retinex-Net
Normal-Light Image R by Retinex-Net I by Retinex-Net
94
95. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by LIME I by LIME
Normal-Light Image R by LIME I by LIME
95
96. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by NPE I by NPE
Normal-Light Image R by NPE I by NPE
96
97. STRUCT GroupDeep Retinex Decomposition
Decomposition
Low-Light Image R by NPE I by NPE
Normal-Light Image R by NPE I by NPE
97