Whispers of Speckles (Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging) + (Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning)
Whispers of Speckles
(Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging)
(Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning)
Presented at the Workshop on Machine Learning for Medical Image Analysis (WMLMIA), IIT Mandi, 25 June 2015.
Semelhante a Whispers of Speckles (Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging) + (Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning)
The Singularity: Toward a Post-Human RealityLarry Smarr
Semelhante a Whispers of Speckles (Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging) + (Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning) (20)
Whispers of Speckles (Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging) + (Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning)
2. Whispers of Speckle
Part I: Building Computational Imaging
Frameworks for Acoustic and Optical
Speckle Imaging
Dr. Debdoot Sheet
Assistant Professor
Department of Electrical Engineering
Indian Institute of Technology Kharagpur
www.facweb.iitkgp.ernet.in/~debdoot/
3. Inspiration
“A wonderful fact to reflect
upon, that every human
creature is constituted to be
that profound secret and
mystery to every other.”
- Charles Dickens
(A Tale of Two Cities)
“If you want to find the
secrets of the universe, think
in terms of energy,
frequency and vibration.”
- Nikola Tesla
25 June 2015 3Whispers of Speckles [Debdoot Sheet] - WMLMIA
4. Motivation
Whispers of Speckles [Debdoot Sheet] - WMLMIA 4
D. Sheet (2014), PhD Thesis
25 June 2015
Text books
R. K. Das (2012), PhD Thesis
A. Barui (2011), PhD Thesis
5. Introduction
• Human body consists of organs and
systems made up of different tissues.
• Pathological conditions and
abnormalities affect their normal
functioning.
• Critical soft tissue abnormalities include
– Plaque formation in the blood vascular
system.
– Lesions in the breast.
– Degeneration of the retina.
– Wounds in the skin.
• Traditional practice of Histopathological
diagnosis requires invasive Biopsy /
Excision for tissue collection
– Not possible in vessels in living Humans
– Improper sampling from Breast lesion
affects diagnostic outcome
– Not possible in retina in living Humans
– Not possible in healing wounds.
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 5
6. ACHIEVING IN SITU HISTOLOGY OF
VASCULAR PLAQUES
625 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
7. • Atherosclerosis
– Plaque builds up in arteries
– Forms anywhere in the vascular system
• Cardiovascular diseases (CVD)
• In vivo Imaging of Plaques
– CT Angiography (CTA)
– MR Angiography (MRA)
– Intravascular Ultrasound (IVUS)
– Intravascular OCT (IV-OCT)
– Intravascular Near-Infrared Spectroscopy
(NIR)
• Plaque Vulnerability Assessment
– Calcification, fibrosis identification
– Lipid pool and Necrosis burden estimation
Source: NIH – National Heart,
Lung, and Blood Institute
Blood Vascular System
25 June 2015 7Whispers of Speckles [Debdoot Sheet] - WMLMIA
• Spectral analysis of received ultrasonic echo
signal
– Lizzi et al., 1983
– Nair et al., 2001
– Kawasaki et al., 2002
– Virtual Histology (Volcano Corp.)
– iMap (BostonScientific)
• Texture analysis of B-mode image/signal
– Katouzian et al., 2008, 2010, 2012 (Prog. Hist. /
PH)
– Esclara et al., 2009
– Seabra et al., 2011
• Limitations
– Unable to identify heterogeneous tissue
composition
– Cannot discriminate between dense fibrous tissue
and calcification
– Fails to discriminate true necrosis from shadows
8. Backdrop
8
White light source
350nm 750nm
Power
Stained tissue section
Tissue specific spectrum
350nm 750nm
Power
Calcified
Fibrotic
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
9. 9
White light source
350nm 750nm
Power
Stained tissue section
Tissue specific spectrum
350nm 750nm
Power
Calcified
Fibrotic
: Probing energy (Light)
: Physiological property (Tissue type)
f
1
f
Inferring tissue
type based on
color
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
10. 10
Computation Modelling of Tissue Energy
Interaction for In situ Histopathology
Computed histology
: Probing energy (Acoustic)
: Tissue type (Backscatterer density)
f
1
f
Inferring tissue
type based on
backscattering
response
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
11. Limited Resolution Challenge
11
r1
r2
r3
P. M. Shankar, “A general statistical model for ultrasonic backscattering from tissues”,
IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control., vol. 47, no. 3, pp. 727-736,
May 2000.
11 rr f
22 rr f
33 rr f
Ultrasound
signal
backscattered
within a
resolution cell
i
i
r
r
fE
E
Signal sensed by the
transducer
irfEf 1
ˆ
Estimated functional ensemble of
backscatterer density
ˆ Improper estimation of tissue type in
inhomogeneous media
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
12. Statistical Physics in Acoustic Imaging
r1
r2
r3
r1
r2
m=0.5
Ω1
Ω2
r
P(r)
m=1.0
Ω1
Ω2
Ω3
r
P(r)
P. M. Shankar, “A general statistical model for ultrasonic backscattering from
tissues”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control., vol. 47, no. 3,
pp. 727-736, May 2000.
2
12
exp
2
,| r
m
m
rm
mr m
mm
N
25 June 2015 12Whispers of Speckles [Debdoot Sheet] - WMLMIA
13. Statistical physics of ultrasonic backscattering
Lipidic
r
P(r)
Fibrotic
r
P(r)
Calcified
r
P(r)
V. Dumane and P. M. Shankar, “Use of frequency
diversity and Nakagami statistics in ultrasonic
tissue characterization”, IEEE Trans. Ultrasonics,
Ferroelectrics, Freq. Control, vol. 48, no. 4, pp.
1139-1146, Jul. 2001
F. Destrempes, J. Meunier, M. . F. Giroux, G.
Soulez, G. Cloutier, “Segmentation in ultrasonic b-
mode images of healthy carotid arteries using
mixture of Nakagami distributions and stochastic
optimization”, IEEE Trans. Med. Imaging, vol. 28,
no. 2, pp. 215-229, Feb. 2009.
25 June 2015 13Whispers of Speckles [Debdoot Sheet] - WMLMIA
14.
32121
221121
,,, 1
,
1
,,
1
,
111
)(,|,|
,|
;),(,),,,,,(||
L
l
lll
L
l
lll
L
l
lll
mrpmrpp
mrpp
ymprfyrp
NN
N
Mathematical intractability, the problem
)(
)(
)|(
)|( yP
rp
yrp
ryp The probabilistic decision making framework
Scales unknown
Correlation among scales unknown
No. components unknown
Prior probab. of each comp. unknown
25 June 2015 14Whispers of Speckles [Debdoot Sheet] - WMLMIA
15. Proposed Solution
Statistical physics model of ultrasonic backscattering
Set of signal received by the transducer
Training set of annotated examples to be used for supervised learning
Supervised learner for learning tissue specific statistical physics model
train;,|)(
),(
)|,(
),|( RR
yHyP
rp
yrp
ryp
Solution through Transfer
Learning Framework25 June 2015 15Whispers of Speckles [Debdoot Sheet] - WMLMIA
16. HOW TO DEAL WITH THIS AS A
MACHINE LEARNING CHALLENGE?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 16
17. Learning?
A computer program is said to learn from
experience E with respect to some class of
tasks T and performance measure P, if its
performance at tasks in T, as measured by
P, improves with experience E
-Tom Mitchell
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 17
18. Demystifying Learning
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 18
Man 1 Man 2 Man 3Man 4
Great Wall logo
Great Wall tower
Kim Jung
WangDebdoot
Experience (E)
Performance(P)
Debdoot, Kim, Jung and Wang are standing near the
Great Wall logo and the Great Wall tower is behind them.
19. How was it Learning?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 19
Man 1 Man 2 Man 3Man 4
Great Wall logo
Great Wall tower
Kim Jung
WangDebdoot
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Debdoot, Kim, Jung and Wang are standing near the
Great Wall logo and the Great Wall tower is behind them.
Recognize
humans
20. GETTING MACHINES TO LEARN
TISSUE – ENERGY INTERACTION
FOR IN SITU HISTOLOGY
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 20
21. IVUS Tissue Characterization
21
Background
Lipidic
Fibrotic
Calcified
Necrosis
Iterative self-organizing
atherosclerotic tissue labeling in
intravascular ultrasound images and
comparison with virtual histology,
IEEE TBME, 59(11), 2012
Hunting for necrosis in the shadows
of intravascular ultrasound, CMIG,
38(2), 2014
Joint learning of ultrasonic
backscattering statistical physics and
signal confidence primal for
characterizing atherosclerotic plaques
using intravascular ultrasound, Med.
Image Anal,18(1), 2014
Nakagami parameter
and signal
confidence estimate
Random forest
learning
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
22. Ultrasound Signal Confidence
• An ultrasonic pulse as well as backscattered
echo travel along the same path through a
heterogeneous media.
• They are subjected to the same attenuation.
• Confidence of the received signal is a
reflection of fidelity of samples received by
the transducer.
• It can be estimated by treating its
propagation as a random walk along an
ultrasonic scan-line.
• A random walker starting at a point on the
scan-line reaches the virtual transducer
element placed at the origin of each scan-
line.
• This random walk is solved using the electric
network equivalent and solving it in the
paradigm of graph theory.
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 22
A. Karamalis, W. Wein, T. Klein, N. Navab (2012) Ultrasound confidence maps
using random walks, Medical Image Analysis, 16:1101–1112.
23. Transfer Learning
Framework
23
Ultrasound RF data
(i) Signal confidence
(ii) Speckle statistics
Tissue labels
f
Learnt random forest
Learning
phase
(offline) Tissue labels
Prediction
(online)
f
Whispers of Speckles [Debdoot Sheet] - WMLMIA25 June 2015
24. Random Forests for Learning
25 June 2015 24Whispers of Speckles [Debdoot Sheet] - WMLMIA
A. Criminisi and J. Shotton, Decision Forests for Computer
Vision and Medical Image Analysis, Springer, 2013.
25. Experiment Design
• Data Collection:
– Columbia University, New York City, NY, USA
– Interventional Cardiologist: Dr. Stephane G. Carlier
– Cardiovascular Histopathologist: Dr. Renu Virmani,
CV Path Institute, Gaithersburg, USA
– Cases # 13
– Tissue Sections # 53
– Atlantis, 40 MHz IVUS, Boston Scientific,CA, USA
– Sampling freq: 400 MHz
– Sampling geometry: 256 scan lines per rotation,
2048 samples per scan line
• Learning
– Source task: {Ω,m} estimated at 28 scales +
Ultrasonic Confidence (A. Karamalis, et al. (2012))
– Target task: Random forest 50 decision trees
• Cross validation
– 53 fold cross validation
– Learn with 52, test on the remaining
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 25
26. Ultrasonic Histology of
Atherosclerotic Plaques
• Characterization based on ultrasonic statistical physics.
• Superior machine learning algorithm.
• Reliability measure for estimation of tissues.
Probability of Calcified
tissues
Probability of Fibrotic
tissues
Probability of Lipidic
tissues
Probability of Necrotic
tissues
Calcified
Lipidic
Fibrotic
Necrotic
25 June 2015 26Whispers of Speckles [Debdoot Sheet] - WMLMIA
38. Take home message
• Different types of soft tissues have characteristic response
when interacting with acoustic energy.
• Heterogeneous tissues can be identified by learning of
tissue specific energy interaction response using statistical
physics models.
• Transfer Learning can be employed for efficiently solving
tissue characterization problems modeled as tissue-energy
interaction problems.
– CPU/GPU handshaking can be used for fast implementation of such
tasks
• Explore possibility of Functional Histopathology In situ
25 June 2015 38Whispers of Speckles [Debdoot Sheet] - WMLMIA
39. Whispers of Speckle
Part II: Enlightenment from Shallow to
Complex Reasoning with Deep Learning
Dr. Debdoot Sheet
Assistant Professor
Department of Electrical Engineering
Indian Institute of Technology Kharagpur
www.facweb.iitkgp.ernet.in/~debdoot/
40. DOES THIS METHOD OF TRANSFER
LEARNING APPLY ONLY TO
ULTRASONIC IMAGING?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 40
41. Skin• Skin forms the general covering of the
body protecting us from external
influences.
• Functions
– Thermoregulation
– Sweat secretion
– Tactile, pressure, temperature sensing
• Stratified organization
– Epidermis
– Papillary dermis
– Dermis
– Adipose tissue
• Wound
– Major pathological injury
– Skin is torn, cut, punctured
• Clinical challenge in management
– Healing in person specific
– Patient specific intervention
– In situ investigation of healing is challenge
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 41
42. Skin
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 42
• In situ investigation
– Optical Coherence Tomography (OCT)
• Cobb et al., (2006)
• Barui et al., (2011)
– Optical photography
• Cross-sectional information about healing
wound is not available
– NIR imaging
• Cross-section histological information not
present
• In situ Histology with OCT
– G. van Soest et al., (2010) – Cardiovascular
OCT
– A. Barui et al., (2011) – Cutaneous wound
beds.
• Challenges
– Identify co-located tissue heterogeneity
– Identify and discriminate Inter-digitated
structures
43. Tissue Photon Interaction
Whispers of Speckles [Debdoot Sheet] - WMLMIA 43
Incident
radiation
Regular
reflection Diffuse
reflection
Scattering
Absorption Multispectral optical imageOCT
B. Saleh, Introduction to Subsurface Imaging, Cambridge, 2011.
0.5 mm
0.5 mm
25 June 2015
44. Optical Coherence Tomography
Whispers of Speckles [Debdoot Sheet] - WMLMIA 44
Low time-coherence
light source
Depth scan mirror
Sample
Detector
Source beam
Reference beam
Sample beam
Detector beam
x
z
z
OCT Image
Michelson
interferometer
25 June 2015
45. TPI in Swept Source OCT
Whispers of Speckles [Debdoot Sheet] - WMLMIA 45
Source
Ballistic
backscattering
Non-ballistic
backscattering
Reference
Detector
A. F. Fercher, et al, Optical coherence tomography — principles and applications, Rep. Prog. Phys.
66 (2003) 239–303
Epithelium
Papillary dermis
Dermis
Adipose
Speckle intensity
Probability
density
25 June 2015
S
S
S
S
I
Ip
exp
1
47. Framework
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 47
Learn TPI Model
Training Image Ground Truth Labels
Test Image
Learn TPI Model
Characterized tissue
train;,| II, xH
48. Computational Histology of Skin
• Solution through a transfer learning
approach
• Performance benchmark (Accuracy)
– Epithelium = 99%
– Papilary dermis = 95%
– Dermis = 99%
– Adipose = 98%
• D. Sheet, et al, “In situ histology of mice
skin through transfer learning of tissue
energy interaction in optical coherence
tomography”, J. Biomed. Optics, 18(9),
2013.
25 June 2015 48
Multi-scale
modeling of
OCT speckles
Training
image
set Ground
truth
Random forest
learning
Multi-scale
modeling of
OCT speckles
Test image
Labeled
tissue
Whispers of Speckles [Debdoot Sheet] - WMLMIA
49. In situ Histology
of Skin
OCT
Histo
Epithelium
Epithelium
Papillary dermis
Dermis
Adipose tissue
25 June 2015 49
Papillary dermisDermisAdipose tissueAll tissues
In situ histology of mice skin through
transfer learning of tissue energy
interaction in optical coherence
tomography, J. Biomed. Optics,
18(9), 2013
Whispers of Speckles [Debdoot Sheet] - WMLMIA
50. In vitro
validation
towards
In vivo
translation
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 50
Transfer Learning of Tissue
Photon Interaction in Optical
Coherence Tomography towards
In vivo Histology of the Oral
Mucosa, Proc. ISBI, 2014.
51. Computational Histology of Retina
• Transfer learning approach
– Retinal OCT tissue labeling
• Performance benchmark (Accuracy)
– Anterior coat > 98%
– RPE > 92%
– Posterior coat > 99%
• SPK Karri and D. Sheet, et al.,
“Computational Histology of Retina
through Transfer Learning of Tissue
Photon Interaction in Optical
Coherence Tomography”, Proc. Int.
Symp. Biomedical Imaging (ISBI), 2014.
25 June 2015 51
Multi-scale
modeling of
OCT speckles
Training
image
set
Ground
truth
Random forest
learning
Multi-scale
modeling of
OCT speckles
Test image
Labeled
tissue
Whispers of Speckles [Debdoot Sheet] - WMLMIA
53. State of the Art
• In situ Histology with OCT
– G. van Soest et al., (2010), G.
J. Ughi et al., (2013) –
Cardiovascular OCT
– D. Sheet et al., (2013, 2014) –
Cutaneous wounds, oral
• Challenges
– Heuristic features
• Texture
• Intensity statistics
– Heuristic computational
models
• Transfer learning of speckle
occurrence models
– Incomplete representation
dictionary
Whispers of Speckles [Debdoot Sheet] - WMLMIA 53
Multi-scale
modeling of
OCT speckles
Training
image
set Ground
truth
Random forest
learning
Multi-scale
modeling of
OCT speckles
Test image
Labeled
tissue
25 June 2015
54. Heuristics in State of Art
Whispers of Speckles [Debdoot Sheet] - WMLMIA 5425 June 2015
55. (RE)EXPLORING THE CONCEPTS OF
HIERARCHY IN LEARNING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 55
56. How was it Learning?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 56
Man 1 Man 2 Man 3Man 4
Great Wall logo
Great Wall tower
Kim Jung
WangDebdoot
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Debdoot, Kim, Jung and Wang are standing near the
Great Wall logo and the Great Wall tower is behind them.
Recognize
humans
57. Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 57
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Describe Scene
58. Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 58
Salient Segments
Objectify
Recognize
inanimate
Describe Scene
Recognize
humans
LBP
Wavelets
HoG
Body part
recognition
Human
appearance
Chroma
clustering
Posture
realign Silhouette
matching
Recognize
human
Detect
humans
59. FROM SHALLOW TO COMPLEX
REASONING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 59
60. Heuristics in State of Art
Whispers of Speckles [Debdoot Sheet] - WMLMIA 6025 June 2015
61. The Solution
Whispers of Speckles [Debdoot Sheet] - WMLMIA 61
DenoisingAutoEncoder
DenoisingAutoEncoder
LogisticReg.
25 June 2015
62. Using a Deep Network
Whispers of Speckles [Debdoot Sheet] - WMLMIA 6225 June 2015
63. COMPLEX REASONING AND
ITS DEEP LEARNING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 63
64. Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 64
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Describe Scene
65. Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 65
Salient Segments
Objectify
Recognize
inanimate
Describe Scene
Recognize
humans
LBP
Wavelets
HoG
Body part
recognition
Human
appearance
Chroma
clustering
Posture
realign Silhouette
matching
Recognize
human
Detect
humans
66. How to tackle this dilemma?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 66
Great Wall
behind
Great Wall
logo beside
Debdoot, Kim,
Jung, Wang
67. Multilayer Perceptron (MLP)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 67
Hiddenlayers
Hiddenlayers
Hiddenlayers
68. MLP Learning, troubles thereof
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 68
P
T1
T2
69. MLP Learning troubles, why so?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 69
P
T1
T2
LBP
Wavelets
HoG
Body part
recognition
Human
appearance
Chroma
clustering
Posture
realign Silhouette
matching
Recognize
human?
70. HOW TO DEEP LEARN?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 70
71. Deep Learning, origin and growth
• Around 1950 – NN age
– Neural Nets (McCulloch and Pitts,
1943)
– Unsupervised Learn. (Hebb, 1949)
– Supervised Learn. (Rosenblatt, 1958)
– Associative Memory (Palm, 1980;
Hopfield, 1982)
• 1960
– Discovery of visual sensory cells that
respond to Edges (Hubel and Wiesel,
1962)
– Feed Forward Multi Layer Perceptron
(FF-MLP) (Ivakhnenko, 1968)
• 1980 – Neocognition
– Convolution + WeightReplication +
Subsampling (Fukushima, 1980)
– Max Pooling
– Back-propagation (Werbos, 1981;
LeCunn, 1985, 1988)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 71
72. Deep Learning, origin and growth
• 1980-2000 – Search for simple,
low-complexity, problem-solvers
– Recurrent Neural Network (RNN)
(Hochreiter and Schmidhuber, 1996)
– Local learning Feed forward NN
(Dayan and Hinton, 1996)
– Advanced gradient descent
(Schaback and Werner, 1992)
– Sequential Network Construction
(Honavar and Uhr, 1988)
– Unsupervised Pre-training (Ritter
and Kohonen, 1989)
– Auto-Encoder (Hinton et al., 1989)
– Back Propagating Convolutional
Neural Networks (LeCun et al., 1989,
1990a, 1998)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 72
73. Deep Learning, origin and growth
• 2000 – Era of Deep Learning
– NIPS 2003 Feature Selection
Challenge (Neal and Zhang, 2006)
– MNIST digit recognition (LeCun et
al., 1989)
– Deep Belief Network (DBN) /
Restricted Boltzmann Machines
(Hinton et al., 2006)
– Auto Encoders (Bengio, 2009)
• 2006
– GPU based CNN (Chellapilla et al.,
2006)
• 2009
– GPU DBN (Raina et al., 2009)
• 2011
– Max-Pooling CNN on the GPU
(Ciresan et al., 2011)
• 2012
– Image Net (Krizhevsky et al., 2012)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 73
74. DEEP LEARNING OF COMPLEX
REASONING FOR OCT TISSUE
CHARACTERIZATION
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 74
75. Exploring Deep Architecture
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 75
Multi-scale
modeling of
OCT speckles
Training
image
set
Ground
truth
Random forest
learning
Multi-scale
modeling of
OCT speckles
Test image
Labeled
tissue
Stacked Auto-
Encoders,
Logistic
Regression
Random
Forest
Training
image
set
Ground
truth
http://www.facweb.iitkgp.ernet.in/~debdoot/current.html
76. Auto Encoder for Deep Learning
Whispers of Speckles [Debdoot Sheet] - WMLMIA 7625 June 2015
77. Results in Wounds
Whispers of Speckles [Debdoot Sheet] - WMLMIA 77
(a) OCT image of wound (b) Ground truth (c) In situ histology
Epithelium, Papillary
dermis, Dermis, Adipose
Epithelium, Papillary
dermis, Dermis, Adipose
25 June 2015
78. Experiment Design
• Data Collection
– School of Medical Science
and Technology, Indian
Institute of Technology
Kharagpur
– 1300 nm (HPBW 100 nm)
Swept Source OCT System
• OCS 1300 SS, ThorLabs, NJ,
USA
• 8 bit bitmap images
– Histology for ground truth
• HE stained
• Samples
– Mus musculus (small mice)
– 16 healthy skin
– 2 wounds on skin
• DNN architecture
– Patch size – 36 × 36 px
– DAE1 – 400 nodes
– DAE2 – 100 nodes
– Target – Logistic Reg.
• 5 outputs
– Sparsity – 20%
– Mini-batch training
• In situ Histology
Performance
– Epithelium – 96%
– Papillary dermis – 93%
– Dermis – 99%
– Adipose tissue – 98%
Whispers of Speckles [Debdoot Sheet] - WMLMIA 7825 June 2015
79. Learning of Representations
Whispers of Speckles [Debdoot Sheet] - WMLMIA 79
Representation of speckle
appearance models learned by DAE1
Sparsity of representations learned by
DAE2
25 June 2015
80. END NOTE
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 80
81. Messages for Human Learning
• Photons interact characteristically with different tissues.
– Stochastic similarity exists in speckle appearance.
– Such representations are hard to heuristically encode.
• Deep learning and auto-encoders for computational imaging
– Speckle imaging application viz. OCT tissue characterization
– Hierarchical learning
• Locally embedded representations.
• Sparsity is in learned (auto-encoded) representations.
Whispers of Speckles [Debdoot Sheet] - WMLMIA 81
Queries: Debdoot Sheet (debdoot@ee.iitkgp.ernet.in)
25 June 2015
82. About Deep Learning
“It’s like in quantum physics at the beginning of the
20th century” Trishul Chilimbi (MSR, DNN, Adam)
“The experimentalists and practitioners were ahead
of the theoreticians. They couldn’t explain the
results. We appear to be at a similar stage with
DNNs. We’re realizing the power and the
capabilities, but we still don’t understand the
fundamentals of exactly how they work.”
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 82
83. Take home message
“We’ve humanized the scientist;
now we must scientize the
humanist. We didn’t try to cover
physics... we uncovered it.”
- Robert Resnick
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 83
84. Take home message
• Challenges
– Architectures
• Neural Nets vs. Others
– Implementation
• CPU vs. GPU vs. Cloud
– GPU (VLSI) architectures
• Hierarchical Temporal
Memory
• Potential Causal Connection
• Toolboxes
– Theano (Python/SciPy)
– Pylearn2
– Torch
– Caffe
– Matlab (Rasmus Berg Palm)
• More information
– www.deeplearning.net
– Schmidhuber (2014). Deep
Learning in Neural
Networks: An Overview
(arXiv:1404.7828)
– Bengio (2009). Learning
Deep Architectures for AI.
– Deng and Yu (2013). Deep
Learning: Methods and
Applications.
• Conferences
– Int. Conf. Learning
Representations (ICLR)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 84