My master thesis work extends the problem formulation of learnable compressive subsampling [1] that focuses on the learning of the best sampling operator in the Fourier domain adapted to spectral properties of a training set of images. I formulated the problem as a reconstruction from a finite number of sparse samples with a prior learned from the external dataset or learned on-fly from the images to be reconstructed. More in
details, I developed two very different methods, one using multiband coding in the spectral domain and the second using a neural network.
The new methods can be applied to many different fields of spectroscopy and Fourier optics, for example in medical (computerized tomography, magnetic resonance spectroscopy) and astronomy (the Square Kilometre Array) imaging, where the capability to reconstruct high-quality images, in the pixel domain, from a limited number of samples, in the frequency domain, is a key issue.
The proposed methods have been tested on diverse datasets covering facial images, medical and multi-band astronomical data, using the mean square error and SSIM as a perceptual measure of the quality of the reconstruction.
Finally, I explored the possible application in data acquisition systems such as computer tomography and radio astronomy. The obtained results demostrate that the properties of the proposed methods have a very promising potential for future research and extensions.
For such reason, the work was both presented at the poster session of the EUSIPCO 2018 conference in Rome and submitted for a EU patent.
[1] L. Baldassarre, Y.-H. Li, J. Scarlett, B. Gözcü, I. Bogunovic, and V.
Cevher, “Learning-based compressive subsampling,” IEEE Journal of Selected
Topics in Signal Processing, vol. 10, no. 4, pp. 809–822, 2016
Injecting image priors into Learnable Compressive Subsampling
1. Injecting image priors into Learnable
Compressive Subsampling
Martino G. Ferrari
April 30, 2018
Supervisors: Prof. S. Voloshynovskiy
O. Taran
University of Geneva
Faculty of Science
1/21
2. Table of contents
1. Problem formulation
2. My approach
3. Results
4. Applicability
5. Conclusion
2/21
13. Compressive Subsampling
Classical acquisition
Encoder Decoder
x hlp × hlp ˆx
Encoder
b =↓ M(hlp ∗ x)
Compressive Subsampling (CS)
Encoder Decoder
x × ∆ ˆx
Encoder
b = Ax
Decoder
ˆx = hlp ∗ (↑ Rb)
Decoder
ˆx = ∆(b)
where both x and ˆx have dimension n while b has dimension m
4/21
14. CS Decoder
The typical Compressive Subsampling decoding is performed by solving
an optimisation problem, as for example the LASSO [1] minimisation:
ˆx = arg min
x
Ax − b 2
2+α x 1
5/21
15. CS Decoder
The typical Compressive Subsampling decoding is performed by solving
an optimisation problem, as for example the LASSO [1] minimisation:
ˆx = arg min
x
Ax − b 2
2+α x 1
Limitations:
• computationally hard
• noise over-fit
5/21
30. LSC I - Learning
−1 0 1
·104
0.0
0.2
0.4
0.6
0.8
1.0
frequency
energy
non-cumulated
cumulated
1. transform dataset: ΨX = {Ψx1, . . . , ΨxN }
2. cumulative magnitude: c(k) = 1
N
k
j=−J
N
i=0|Ψxi(j)|2
9/21
31. LSC I - Learning
−1 0 1
·104
0.0
0.2
0.4
0.6
0.8
1.0
frequency
energy
non-cumulated
cumulated
1. transform dataset: ΨX = {Ψx1, . . . , ΨxN }
2. cumulative magnitude: c(k) = 1
N
k
j=−J
N
i=0|Ψxi(j)|2
3. sub-band splitting: ΨX = {ΨX1
, . . . , ΨXL
}
9/21
32. LSC I - Learning
−1 0 1
·104
0.0
0.2
0.4
0.6
0.8
1.0
frequency
energy
non-cumulated
cumulated
1. transform dataset: ΨX = {Ψx1, . . . , ΨxN }
2. cumulative magnitude: c(k) = 1
N
k
j=−J
N
i=0|Ψxi(j)|2
3. sub-band splitting: ΨX = {ΨX1
, . . . , ΨXL
}
4. real Cl
re and imaginary Cl
im codebook generation
9/21
33. LSC I - Learning
−1 0 1
·104
0.0
0.2
0.4
0.6
0.8
1.0
frequency
energy
non-cumulated
cumulated
1. transform dataset: ΨX = {Ψx1, . . . , ΨxN }
2. cumulative magnitude: c(k) = 1
N
k
j=−J
N
i=0|Ψxi(j)|2
3. sub-band splitting: ΨX = {ΨX1
, . . . , ΨXL
}
4. real Cl
re and imaginary Cl
im codebook generation
5. sampling-pattern computation:
ˆΩl = arg max
Ωl
(P Ωl
σCl
re
+ P Ωl
σCl
im
)
9/21
34. LSC II - Sampling and Encoding
The sub-band sampling is expressed as:
bl
= P Ωl
(Ψx)l
The code identification is computed in a single step for both imaginary
and real part:
ˆcl
re, ˆcl
im = arg min cl
re,cl
im
bl
re − P Ωi
cl
re
2
2 +
bl
im − PΩi
cl
im
2
2+
|bl
|−P Ωi
|cl
re + jcl
im| 2
2+
arg(bl
) − P Ωi arg(cl
re + jcl
im) 2π
10/21
35. LSC III - Decoding
The decoder is linear and can be expressed as:
ˆx = Ψ∗
(PT
Ω1
b1
+ PT
ΩC
1
PΩC
1
(ˆc1
re + jˆc1
im)),
. . . ,
(PT
ΩL
bL
+ PT
ΩC
L
PΩC
L
(ˆcL
re + jˆcL
im))
where ΩC
l is the complementary set to Ωl
11/21
36. DIP - Overview
Implement a hourglass network [4] using the Deep Image Prior [5]
framework:
Minimisation problem is as follow:
ˆθ = arg min
θ
b − PΩΨfθ(z)
2
2 + βΩθ(θ)
12/21
37. DIP - Prior Injection
With prior injection, the minimisation problem becomes:
ˆθ = arg min
θ
b − PΩΨfθ(z)
2
2 + α fθ(z) − c
2
2 + βΩθ(θ)
while the final reconstruction is obtained through the following linear
operation:
ˆx = Ψ∗
(PT
Ωb + PT
ΩC PΩC Ψfˆθ(z))
13/21
60. Summary
Strengths
• low-sampling-rate recovery
• few prior data needed
• fast encoder/decoder (LSC)
• robust to noise (LSC)
• no prior-training required (DIP)
Weaknesses
• some dependency on signal
alignment
• complex decoder (DIP)
• prior-training required (LSC)
20/21
61. Summary
Strengths
• low-sampling-rate recovery
• few prior data needed
• fast encoder/decoder (LSC)
• robust to noise (LSC)
• no prior-training required (DIP)
Weaknesses
• some dependency on signal
alignment
• complex decoder (DIP)
• prior-training required (LSC)
Future development
• investigate better sub-band splitting (LSC)
• improve coding models (LSC)
• investigate new prior model and cost function (DIP)
• combine deep models within LSC (DIP + LSC)
* This work has been submitted to EUSIPCO 2018 20/21
70. References i
Robert Tibshirani.
Regression shrinkage and selection via the lasso.
Journal of the Royal Statistical Society. Series B (Methodological),
58(1):267–288, 1996.
Bubacarr Bah, Ali Sadeghian, and Volkan Cevher.
Energy-aware adaptive bi-lipschitz embeddings.
CoRR, abs/1307.3457, 2013.
Luca Baldassarre, Yen-Huan Li, Jonathan Scarlett, Baran Gzc, Ilija
Bogunovic, and Volkan Cevher.
Learning-based Compressive Subsampling.
IEEE Journal of Selected Topics in Signal Processing,
10(4):809–822, June 2016.
arXiv: 1510.06188.
71. References ii
Alejandro Newell, Kaiyu Yang, and Jia Deng.
Stacked hourglass networks for human pose estimation.
CoRR, abs/1603.06937, 2016.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.
Deep image prior.
arXiv preprint arXiv:1711.10925, 2017.