Talk by Dr. Nikita Morikiakov on inverse problems in medical imaging with deep learning. Inverse problem is the type of problems in natural sciences when one has to infer from a set of observations the causal factors that produced them. In medical imaging, important examples of inverse problems would be recontruction in CT and MRI, where the volumetric representation of an object is computed from the projection and Fourier space data respectively. In a classical approach, one relies on domain specific knowledge contained in physical-analytical models to develop a reconstruction algorithm, which is often given by a certain iterative refinement procedure. Recent research in inverse problems seeks to develop a mathematically coherent foundation for combining data driven models, based on deep learning, with the analytical knowledge contained in the classical reconstruction procedures. In this talk we will give a brief overview of these developments and then focus on particular applications in Digital Breast Tomosynthesis and MRI reconstruction.

1. Inverse problems in medical
imaging
Nikita Moriakov
Diagnostic Image Analysis Group
Department of Radiology and Nuclear Medicine
Radboud University Medical Center, Nijmegen

2. Plan
• Inverse problems
• Deep Learning in Inverse Problems: Fully
learned inversion vs. Learned iterative
schemes
• Results and future work

3. Inverse problems

4. What is inverse problem?
• 𝑓𝑡𝑟𝑢𝑒 ∈ 𝑋 is an unknown true model parameter, 𝑔 ∈ 𝑌 is data, 𝑒 is sample
from measurement noise and 𝐴: 𝑋 → 𝑌 is a continuous operator mapping
model parameter to data (in the absence of noise)
• Examples: CT & MR reconstruction, denoising, inpainting.

5. Examples: Radon Transform
• Let 𝑓 ∈ 𝐿1 ℝ2 be a function such
that 𝑓(𝑥, 𝑦) is the attenuation
coefficient of scanned object at
point (𝑥, 𝑦)
• If X-ray going along the line 𝐿 has
incident intensity 𝐼0, the outcoming
intensity equals 𝐼0 𝑒
− 𝑥,𝑦 ∈𝐿
𝑓 𝑥,𝑦 𝑑𝐿

6. Examples: Radon Transform
• Parametrize all lines by the angle 𝜃 and
the distance 𝑠 to the origin
• The Radon transform of 𝑓 is a function of
𝑠 ∈ ℝ, 𝜃 ∈ [0, 2 𝜋) given by

7. Radon, 1907

8. Back projection (key for inversion)

9. Examples: Radon Transform
• This direct analytic inversion is called
Filtered Back Projection today.
• Radon transform is invertible (when we
know projections from all directions and
all rays intersecting the object).

10. Examples: DBT
• Available since late 2000s.
• Rapidly replacing Digital
Mammography
• Can have resolution as high as DM
standards.

11. Examples: DBT
• The x-ray tube moves in an arc
over the compressed breast
capturing multiple images of each
breast from different angles in a
continuous or step-and-shoot
fashion.

12. Examples: MR
• Fourier transform ℱ maps images
to frequency domain.
• Some measurements in this domain
are taken, giving subsampling mask
𝑃.
• Inverse Fourier transform maps
back to the image domain.
• Reconstructed image can deviate
from target if k-space is
undersampled.

13. Problems with inverse problems?
• 𝐴 can have non-trivial kernel. E.g., for CT with limited number of view
angles the following holds:
• 𝐴 can have discontinuous inverse, thus variations in noise can have strong
effect on reconstruction.

14. Bayesian view and regularization,
MAP
• Finding most likely reconstruction given measurements amounts to
finding 𝑎𝑟𝑔𝑚𝑎𝑥𝑖𝑚𝑎𝑔𝑒 𝑃 𝐼𝑚𝑎𝑔𝑒 𝑀𝑒𝑎𝑠).
• 𝑎𝑟𝑔𝑚𝑎𝑥𝑖𝑚𝑎𝑔𝑒 𝑃 𝐼𝑚𝑎𝑔𝑒 𝑀𝑒𝑎𝑠) = 𝑎𝑟𝑔𝑚𝑎𝑥𝑖𝑚𝑎𝑔𝑒
𝑃 𝑀𝑒𝑎𝑠 𝐼𝑚𝑎𝑔𝑒)⋅𝑃 𝐼𝑚𝑎𝑔𝑒
𝑃(𝑀𝑒𝑎𝑠)
=
𝑎𝑟𝑔𝑚𝑎𝑥𝑖𝑚𝑎𝑔𝑒 (log 𝑃 𝑀𝑒𝑎𝑠 𝐼𝑚𝑎𝑔𝑒 + log 𝑃(𝐼𝑚𝑎𝑔𝑒))
• log 𝑃(𝑀𝑒𝑎𝑠|𝐼𝑚𝑎𝑔𝑒) is the log-likelihood of observed data, log 𝑃(𝐼𝑚𝑎𝑔𝑒)
is the prior term.
• Prior term is used for regularization such as TV regularization.

15. Deep Learning for Inverse
Problems

16. Inverse problems with DL
Fully learned data driven
reconstruction
• Generic parametrization by a neural
network to find an inverse mapping
𝑅 𝜃: 𝑌 → 𝑋 with 𝜃 being the neural
network weights
• No need to have explicit forward
operator or data likelihood
• Need to use fully connected layers
and hence requires a lot of
parameters.
Learned iterative schemes
• Contains explicit knowledge of
forward operator built in the
architecture of 𝑅 𝜃: 𝑌 → 𝑋
• Architecture motivated by existing
optimization algorithms

17. Inverse problems with DL
Learned regularizers
• Train a regularizer 𝑆 𝜃: 𝑋 → ℝ
separately, which is parametrized
by a neural network, and is ideally
proportional to the image prior
(Bayesian view).
• Can be trained adversarily.

18. Fully learned inference
• Zhu, Liu, Rosen, Rosen “Image reconstruction by domain transform
manifold learning” 2018

19. Learned Iterative Schemes: LPD
• Learned Primal-Dual (LPD) is an example of learned iterative schemes.
• Architecture motivated by Primal-Dual Hybrid Gradient Method.

20. Learned Primal-Dual
• A deep neural network
• Iterative procedure inspired by
Primal-Dual Hybrid Gradient
algorithm
• Consists of a primal/dual
reconstruction blocks which
performs small “steps” in
image and projection space
respectively.

21. Network architecture

22. Results (Adler & Öktem)

23. Learned Iterative Schemes: RIM
• The goal in MAP estimate is finding
• This is often done in via an iterative scheme
• To avoid the need to learn prior, we can “generalize” this to the form
• So can optimize this as a recurrent neural network.

24. Recurrent Inference Machines for MR
Lonning, Putzky, Caan, Welling Recurrent Inference Machines for Accelerated MRI Reconstruction 2018

25. RIM for MRI Reconstruction
Lonning, Putzky, Caan, Welling Recurrent Inference Machines for Accelerated MRI Reconstruction 2018

26. Results and future work

27. SPIE 2019 Medical Imaging

28. Data for the experiment
• Synthetic breast images generated by realistic digital breast
phantom (from I. Sechopoulos et al.)
• 25 view angles: [-24, -22, … , 22, 24] deg

29. Results (LPD on DBT)

30. Network architecture (LPD)

31. LPD for DBT architecture (DBToR)
𝑚
𝑇(𝑚)

32. Results (beginning of training)

33. Results (end of training)

34. Results

35. Future work

36. Compressed breast phantom
Classification
Image Segmentation:
Skin – Adipose – Glandular
Caballo M. et al . 2018 “An Unsupervised Automatic Segmentation algorithm for breast tissue” – Med. Phys.
Finite Element
Compression
BCT images acquired
from patients at Radboudumc

37. Finite Element Compression
Breast Density Map
• Create a mesh of the breast
• Simulate the compression of soft tissue (adipose,
glandular & skin) using the high-performance explicit
finite element solver, developed for medical application

38. Compression
Compression
The breast support is move up by 20
mm (to make the bottom flat);
the compression is performed by
moving down the compression paddle
Voxel Resolution of (0.273 mm)3

39. Results

40. Sources
• Arridge, Maass, Öktem, Schönlieb “Solving Inverse Problems Using Data
Driven Models”, Acta Numerica 2019.
• Adler, Öktem “Learned Primal-dual Reconstruction”
• Markoe, “Analytic Tomography”
• Zhu, Liu, Rosen, Rosen “Image reconstruction by domain transform
manifold learning”, Nature 2018
• Lønning, Putzky, Caan, Welling “Recurrent Inference Machines for
Accelerated MRI Reconstruction”, Medical Image Analysis, 2018

41. Thank you!