SlideShare uma empresa Scribd logo
1 de 84
1
WHISPERS
OF
SPECKLES
DEBDOOT SHEET
LAUNCHING
THIS MONSOON
Venue: IIT Mandi
Date: Thursday, 25 June 2015
Time: 11 am – 12:30 pm
Whispers of Speckle
Part I: Building Computational Imaging
Frameworks for Acoustic and Optical
Speckle Imaging
Dr. Debdoot Sheet
Assistant Professor
Department of Electrical Engineering
Indian Institute of Technology Kharagpur
www.facweb.iitkgp.ernet.in/~debdoot/
Inspiration
“A wonderful fact to reflect
upon, that every human
creature is constituted to be
that profound secret and
mystery to every other.”
- Charles Dickens
(A Tale of Two Cities)
“If you want to find the
secrets of the universe, think
in terms of energy,
frequency and vibration.”
- Nikola Tesla
25 June 2015 3Whispers of Speckles [Debdoot Sheet] - WMLMIA
Motivation
Whispers of Speckles [Debdoot Sheet] - WMLMIA 4
D. Sheet (2014), PhD Thesis
25 June 2015
Text books
R. K. Das (2012), PhD Thesis
A. Barui (2011), PhD Thesis
Introduction
• Human body consists of organs and
systems made up of different tissues.
• Pathological conditions and
abnormalities affect their normal
functioning.
• Critical soft tissue abnormalities include
– Plaque formation in the blood vascular
system.
– Lesions in the breast.
– Degeneration of the retina.
– Wounds in the skin.
• Traditional practice of Histopathological
diagnosis requires invasive Biopsy /
Excision for tissue collection
– Not possible in vessels in living Humans
– Improper sampling from Breast lesion
affects diagnostic outcome
– Not possible in retina in living Humans
– Not possible in healing wounds.
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 5
ACHIEVING IN SITU HISTOLOGY OF
VASCULAR PLAQUES
625 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
• Atherosclerosis
– Plaque builds up in arteries
– Forms anywhere in the vascular system
• Cardiovascular diseases (CVD)
• In vivo Imaging of Plaques
– CT Angiography (CTA)
– MR Angiography (MRA)
– Intravascular Ultrasound (IVUS)
– Intravascular OCT (IV-OCT)
– Intravascular Near-Infrared Spectroscopy
(NIR)
• Plaque Vulnerability Assessment
– Calcification, fibrosis identification
– Lipid pool and Necrosis burden estimation
Source: NIH – National Heart,
Lung, and Blood Institute
Blood Vascular System
25 June 2015 7Whispers of Speckles [Debdoot Sheet] - WMLMIA
• Spectral analysis of received ultrasonic echo
signal
– Lizzi et al., 1983
– Nair et al., 2001
– Kawasaki et al., 2002
– Virtual Histology (Volcano Corp.)
– iMap (BostonScientific)
• Texture analysis of B-mode image/signal
– Katouzian et al., 2008, 2010, 2012 (Prog. Hist. /
PH)
– Esclara et al., 2009
– Seabra et al., 2011
• Limitations
– Unable to identify heterogeneous tissue
composition
– Cannot discriminate between dense fibrous tissue
and calcification
– Fails to discriminate true necrosis from shadows
Backdrop
8
White light source
350nm 750nm
Power
Stained tissue section
Tissue specific spectrum
350nm 750nm
Power
Calcified
Fibrotic
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
9
White light source
350nm 750nm
Power
Stained tissue section
Tissue specific spectrum
350nm 750nm
Power
Calcified
Fibrotic
 : Probing energy (Light)
 : Physiological property (Tissue type)
  f
  1
 f
Inferring tissue
type based on
color
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
10
Computation Modelling of Tissue Energy
Interaction for In situ Histopathology
Computed histology
 : Probing energy (Acoustic)
 : Tissue type (Backscatterer density)
  f
  1
 f
Inferring tissue
type based on
backscattering
response
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
Limited Resolution Challenge
11
r1
r2
r3
P. M. Shankar, “A general statistical model for ultrasonic backscattering from tissues”,
IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control., vol. 47, no. 3, pp. 727-736,
May 2000.
 11 rr f  
 22 rr f  
 33 rr f  
Ultrasound
signal
backscattered
within a
resolution cell
 
  i
i
r
r
fE
E




Signal sensed by the
transducer
   irfEf  1
ˆ 

Estimated functional ensemble of
backscatterer density
 ˆ Improper estimation of tissue type in
inhomogeneous media
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
Statistical Physics in Acoustic Imaging
r1
r2
r3
r1
r2
m=0.5
Ω1
Ω2
r
P(r)
m=1.0
Ω1
Ω2
Ω3
r
P(r)
P. M. Shankar, “A general statistical model for ultrasonic backscattering from
tissues”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control., vol. 47, no. 3,
pp. 727-736, May 2000.
 
 











2
12
exp
2
,| r
m
m
rm
mr m
mm
N
25 June 2015 12Whispers of Speckles [Debdoot Sheet] - WMLMIA
Statistical physics of ultrasonic backscattering
Lipidic
r
P(r)
Fibrotic
r
P(r)
Calcified
r
P(r)
V. Dumane and P. M. Shankar, “Use of frequency
diversity and Nakagami statistics in ultrasonic
tissue characterization”, IEEE Trans. Ultrasonics,
Ferroelectrics, Freq. Control, vol. 48, no. 4, pp.
1139-1146, Jul. 2001
F. Destrempes, J. Meunier, M. . F. Giroux, G.
Soulez, G. Cloutier, “Segmentation in ultrasonic b-
mode images of healthy carotid arteries using
mixture of Nakagami distributions and stochastic
optimization”, IEEE Trans. Med. Imaging, vol. 28,
no. 2, pp. 215-229, Feb. 2009.
25 June 2015 13Whispers of Speckles [Debdoot Sheet] - WMLMIA
   
 
     
 






















32121
221121
,,, 1
,
1
,,
1
,
111
)(,|,|
,|
;),(,),,,,,(||







L
l
lll
L
l
lll
L
l
lll
mrpmrpp
mrpp
ymprfyrp
NN
N
Mathematical intractability, the problem
)(
)(
)|(
)|( yP
rp
yrp
ryp  The probabilistic decision making framework
Scales unknown
Correlation among scales unknown
No. components unknown
Prior probab. of each comp. unknown
25 June 2015 14Whispers of Speckles [Debdoot Sheet] - WMLMIA
Proposed Solution
Statistical physics model of ultrasonic backscattering
Set of signal received by the transducer
Training set of annotated examples to be used for supervised learning
Supervised learner for learning tissue specific statistical physics model
  train;,|)(
),(
)|,(
),|( RR


 yHyP
rp
yrp
ryp
Solution through Transfer
Learning Framework25 June 2015 15Whispers of Speckles [Debdoot Sheet] - WMLMIA
HOW TO DEAL WITH THIS AS A
MACHINE LEARNING CHALLENGE?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 16
Learning?
A computer program is said to learn from
experience E with respect to some class of
tasks T and performance measure P, if its
performance at tasks in T, as measured by
P, improves with experience E
-Tom Mitchell
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 17
Demystifying Learning
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 18
Man 1 Man 2 Man 3Man 4
Great Wall logo
Great Wall tower
Kim Jung
WangDebdoot
Experience (E)
Performance(P)
Debdoot, Kim, Jung and Wang are standing near the
Great Wall logo and the Great Wall tower is behind them.
How was it Learning?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 19
Man 1 Man 2 Man 3Man 4
Great Wall logo
Great Wall tower
Kim Jung
WangDebdoot
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Debdoot, Kim, Jung and Wang are standing near the
Great Wall logo and the Great Wall tower is behind them.
Recognize
humans
GETTING MACHINES TO LEARN
TISSUE – ENERGY INTERACTION
FOR IN SITU HISTOLOGY
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 20
IVUS Tissue Characterization
21
Background
Lipidic
Fibrotic
Calcified
Necrosis
Iterative self-organizing
atherosclerotic tissue labeling in
intravascular ultrasound images and
comparison with virtual histology,
IEEE TBME, 59(11), 2012
Hunting for necrosis in the shadows
of intravascular ultrasound, CMIG,
38(2), 2014
Joint learning of ultrasonic
backscattering statistical physics and
signal confidence primal for
characterizing atherosclerotic plaques
using intravascular ultrasound, Med.
Image Anal,18(1), 2014
Nakagami parameter
and signal
confidence estimate
Random forest
learning
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
Ultrasound Signal Confidence
• An ultrasonic pulse as well as backscattered
echo travel along the same path through a
heterogeneous media.
• They are subjected to the same attenuation.
• Confidence of the received signal is a
reflection of fidelity of samples received by
the transducer.
• It can be estimated by treating its
propagation as a random walk along an
ultrasonic scan-line.
• A random walker starting at a point on the
scan-line reaches the virtual transducer
element placed at the origin of each scan-
line.
• This random walk is solved using the electric
network equivalent and solving it in the
paradigm of graph theory.
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 22
A. Karamalis, W. Wein, T. Klein, N. Navab (2012) Ultrasound confidence maps
using random walks, Medical Image Analysis, 16:1101–1112.
Transfer Learning
Framework
23
Ultrasound RF data
(i) Signal confidence
(ii) Speckle statistics
Tissue labels
 f
Learnt random forest
Learning
phase
(offline) Tissue labels
Prediction
(online)
 f
Whispers of Speckles [Debdoot Sheet] - WMLMIA25 June 2015
Random Forests for Learning
25 June 2015 24Whispers of Speckles [Debdoot Sheet] - WMLMIA
A. Criminisi and J. Shotton, Decision Forests for Computer
Vision and Medical Image Analysis, Springer, 2013.
Experiment Design
• Data Collection:
– Columbia University, New York City, NY, USA
– Interventional Cardiologist: Dr. Stephane G. Carlier
– Cardiovascular Histopathologist: Dr. Renu Virmani,
CV Path Institute, Gaithersburg, USA
– Cases # 13
– Tissue Sections # 53
– Atlantis, 40 MHz IVUS, Boston Scientific,CA, USA
– Sampling freq: 400 MHz
– Sampling geometry: 256 scan lines per rotation,
2048 samples per scan line
• Learning
– Source task: {Ω,m} estimated at 28 scales +
Ultrasonic Confidence (A. Karamalis, et al. (2012))
– Target task: Random forest 50 decision trees
• Cross validation
– 53 fold cross validation
– Learn with 52, test on the remaining
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 25
Ultrasonic Histology of
Atherosclerotic Plaques
• Characterization based on ultrasonic statistical physics.
• Superior machine learning algorithm.
• Reliability measure for estimation of tissues.
Probability of Calcified
tissues
Probability of Fibrotic
tissues
Probability of Lipidic
tissues
Probability of Necrotic
tissues
Calcified
Lipidic
Fibrotic
Necrotic
25 June 2015 26Whispers of Speckles [Debdoot Sheet] - WMLMIA
Coronary Plaque Characterization
and Staging
25 June 2015 27Whispers of Speckles [Debdoot Sheet] - WMLMIA
SOME (AWESOME SCORES ON)
METRICS
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 28
Performance Evaluation
25 June 2015 29
Inter-observer variability
Intra-observer variability
Whispers of Speckles [Debdoot Sheet] - WMLMIA
ANALYZING COMPLEXITY OF
MACHINE LEARNING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 30
Computational Complexity
Training Complexity Testing Complexity
25 June 2015 31Whispers of Speckles [Debdoot Sheet] - WMLMIA
Roles of Source Knowledge
(Features)
Feature 1
Feature2
25 June 2015 32Whispers of Speckles [Debdoot Sheet] - WMLMIA
Importance of Source Knowledge
Genuer, R., Poggi, J.-M.,
Tuleau-Malot, C., (2010).
Variable selection using
random forests. Pat. Recog.
Letters. 31(14):2225-2236
25 June 2015 33Whispers of Speckles [Debdoot Sheet] - WMLMIA
Learning from Approximately
Labeled Minimum Samples
25 June 2015 34
Donomez, P., Lebanon, G., Balasubramanian, K., (2010). Unsupervised supervised learning I: Estimating Classification and
Regression Errors without Labels, J. Mach. Learn. Res. 11:1323-1351
Whispers of Speckles [Debdoot Sheet] - WMLMIA
Variation of Residual Error in
Learning
25 June 2015 35
Learning example
Test results
Whispers of Speckles [Debdoot Sheet] - WMLMIA
END NOTE
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 36
Collaboration and Network
25 June 2015 37Whispers of Speckles [Debdoot Sheet] - WMLMIA
Take home message
• Different types of soft tissues have characteristic response
when interacting with acoustic energy.
• Heterogeneous tissues can be identified by learning of
tissue specific energy interaction response using statistical
physics models.
• Transfer Learning can be employed for efficiently solving
tissue characterization problems modeled as tissue-energy
interaction problems.
– CPU/GPU handshaking can be used for fast implementation of such
tasks
• Explore possibility of Functional Histopathology In situ
25 June 2015 38Whispers of Speckles [Debdoot Sheet] - WMLMIA
Whispers of Speckle
Part II: Enlightenment from Shallow to
Complex Reasoning with Deep Learning
Dr. Debdoot Sheet
Assistant Professor
Department of Electrical Engineering
Indian Institute of Technology Kharagpur
www.facweb.iitkgp.ernet.in/~debdoot/
DOES THIS METHOD OF TRANSFER
LEARNING APPLY ONLY TO
ULTRASONIC IMAGING?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 40
Skin• Skin forms the general covering of the
body protecting us from external
influences.
• Functions
– Thermoregulation
– Sweat secretion
– Tactile, pressure, temperature sensing
• Stratified organization
– Epidermis
– Papillary dermis
– Dermis
– Adipose tissue
• Wound
– Major pathological injury
– Skin is torn, cut, punctured
• Clinical challenge in management
– Healing in person specific
– Patient specific intervention
– In situ investigation of healing is challenge
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 41
Skin
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 42
• In situ investigation
– Optical Coherence Tomography (OCT)
• Cobb et al., (2006)
• Barui et al., (2011)
– Optical photography
• Cross-sectional information about healing
wound is not available
– NIR imaging
• Cross-section histological information not
present
• In situ Histology with OCT
– G. van Soest et al., (2010) – Cardiovascular
OCT
– A. Barui et al., (2011) – Cutaneous wound
beds.
• Challenges
– Identify co-located tissue heterogeneity
– Identify and discriminate Inter-digitated
structures
Tissue Photon Interaction
Whispers of Speckles [Debdoot Sheet] - WMLMIA 43
Incident
radiation
Regular
reflection Diffuse
reflection
Scattering
Absorption Multispectral optical imageOCT
B. Saleh, Introduction to Subsurface Imaging, Cambridge, 2011.
0.5 mm
0.5 mm
25 June 2015
Optical Coherence Tomography
Whispers of Speckles [Debdoot Sheet] - WMLMIA 44
Low time-coherence
light source
Depth scan mirror
Sample
Detector
Source beam
Reference beam
Sample beam
Detector beam
x
z
z
OCT Image
Michelson
interferometer
25 June 2015
TPI in Swept Source OCT
Whispers of Speckles [Debdoot Sheet] - WMLMIA 45
Source
Ballistic
backscattering
Non-ballistic
backscattering
Reference
Detector
A. F. Fercher, et al, Optical coherence tomography — principles and applications, Rep. Prog. Phys.
66 (2003) 239–303
Epithelium
Papillary dermis
Dermis
Adipose
Speckle intensity
Probability
density
25 June 2015
  






S
S
S
S
I
Ip

exp
1
COMPUTATIONAL OPTICAL
COHERENCE HISTOLOGY
THROUGH TRANSFER LEARNING
4625 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
Framework
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 47
Learn TPI Model
Training Image Ground Truth Labels
Test Image
Learn TPI Model
Characterized tissue
  train;,| II, xH
Computational Histology of Skin
• Solution through a transfer learning
approach
• Performance benchmark (Accuracy)
– Epithelium = 99%
– Papilary dermis = 95%
– Dermis = 99%
– Adipose = 98%
• D. Sheet, et al, “In situ histology of mice
skin through transfer learning of tissue
energy interaction in optical coherence
tomography”, J. Biomed. Optics, 18(9),
2013.
25 June 2015 48
Multi-scale
modeling of
OCT speckles
Training
image
set Ground
truth
Random forest
learning
Multi-scale
modeling of
OCT speckles
Test image
Labeled
tissue
Whispers of Speckles [Debdoot Sheet] - WMLMIA
In situ Histology
of Skin
OCT
Histo
Epithelium
Epithelium
Papillary dermis
Dermis
Adipose tissue
25 June 2015 49
Papillary dermisDermisAdipose tissueAll tissues
In situ histology of mice skin through
transfer learning of tissue energy
interaction in optical coherence
tomography, J. Biomed. Optics,
18(9), 2013
Whispers of Speckles [Debdoot Sheet] - WMLMIA
In vitro
validation
towards
In vivo
translation
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 50
Transfer Learning of Tissue
Photon Interaction in Optical
Coherence Tomography towards
In vivo Histology of the Oral
Mucosa, Proc. ISBI, 2014.
Computational Histology of Retina
• Transfer learning approach
– Retinal OCT tissue labeling
• Performance benchmark (Accuracy)
– Anterior coat > 98%
– RPE > 92%
– Posterior coat > 99%
• SPK Karri and D. Sheet, et al.,
“Computational Histology of Retina
through Transfer Learning of Tissue
Photon Interaction in Optical
Coherence Tomography”, Proc. Int.
Symp. Biomedical Imaging (ISBI), 2014.
25 June 2015 51
Multi-scale
modeling of
OCT speckles
Training
image
set
Ground
truth
Random forest
learning
Multi-scale
modeling of
OCT speckles
Test image
Labeled
tissue
Whispers of Speckles [Debdoot Sheet] - WMLMIA
DOES SOMETHING LOOK
FISHY?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 52
State of the Art
• In situ Histology with OCT
– G. van Soest et al., (2010), G.
J. Ughi et al., (2013) –
Cardiovascular OCT
– D. Sheet et al., (2013, 2014) –
Cutaneous wounds, oral
• Challenges
– Heuristic features
• Texture
• Intensity statistics
– Heuristic computational
models
• Transfer learning of speckle
occurrence models
– Incomplete representation
dictionary
Whispers of Speckles [Debdoot Sheet] - WMLMIA 53
Multi-scale
modeling of
OCT speckles
Training
image
set Ground
truth
Random forest
learning
Multi-scale
modeling of
OCT speckles
Test image
Labeled
tissue
25 June 2015
Heuristics in State of Art
Whispers of Speckles [Debdoot Sheet] - WMLMIA 5425 June 2015
(RE)EXPLORING THE CONCEPTS OF
HIERARCHY IN LEARNING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 55
How was it Learning?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 56
Man 1 Man 2 Man 3Man 4
Great Wall logo
Great Wall tower
Kim Jung
WangDebdoot
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Debdoot, Kim, Jung and Wang are standing near the
Great Wall logo and the Great Wall tower is behind them.
Recognize
humans
Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 57
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Describe Scene
Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 58
Salient Segments
Objectify
Recognize
inanimate
Describe Scene
Recognize
humans
LBP
Wavelets
HoG
Body part
recognition
Human
appearance
Chroma
clustering
Posture
realign Silhouette
matching
Recognize
human
Detect
humans
FROM SHALLOW TO COMPLEX
REASONING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 59
Heuristics in State of Art
Whispers of Speckles [Debdoot Sheet] - WMLMIA 6025 June 2015
The Solution
Whispers of Speckles [Debdoot Sheet] - WMLMIA 61
DenoisingAutoEncoder
DenoisingAutoEncoder
LogisticReg.
25 June 2015
Using a Deep Network
Whispers of Speckles [Debdoot Sheet] - WMLMIA 6225 June 2015
COMPLEX REASONING AND
ITS DEEP LEARNING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 63
Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 64
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Objectify
Detect
humans
Recognize
inanimate
Describe Scene
Recognize
humans
Salient Segments
Describe Scene
Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 65
Salient Segments
Objectify
Recognize
inanimate
Describe Scene
Recognize
humans
LBP
Wavelets
HoG
Body part
recognition
Human
appearance
Chroma
clustering
Posture
realign Silhouette
matching
Recognize
human
Detect
humans
How to tackle this dilemma?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 66
Great Wall
behind
Great Wall
logo beside
Debdoot, Kim,
Jung, Wang
Multilayer Perceptron (MLP)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 67
Hiddenlayers
Hiddenlayers
Hiddenlayers
MLP Learning, troubles thereof
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 68
P
T1
T2
MLP Learning troubles, why so?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 69
P
T1
T2
LBP
Wavelets
HoG
Body part
recognition
Human
appearance
Chroma
clustering
Posture
realign Silhouette
matching
Recognize
human?
HOW TO DEEP LEARN?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 70
Deep Learning, origin and growth
• Around 1950 – NN age
– Neural Nets (McCulloch and Pitts,
1943)
– Unsupervised Learn. (Hebb, 1949)
– Supervised Learn. (Rosenblatt, 1958)
– Associative Memory (Palm, 1980;
Hopfield, 1982)
• 1960
– Discovery of visual sensory cells that
respond to Edges (Hubel and Wiesel,
1962)
– Feed Forward Multi Layer Perceptron
(FF-MLP) (Ivakhnenko, 1968)
• 1980 – Neocognition
– Convolution + WeightReplication +
Subsampling (Fukushima, 1980)
– Max Pooling
– Back-propagation (Werbos, 1981;
LeCunn, 1985, 1988)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 71
Deep Learning, origin and growth
• 1980-2000 – Search for simple,
low-complexity, problem-solvers
– Recurrent Neural Network (RNN)
(Hochreiter and Schmidhuber, 1996)
– Local learning Feed forward NN
(Dayan and Hinton, 1996)
– Advanced gradient descent
(Schaback and Werner, 1992)
– Sequential Network Construction
(Honavar and Uhr, 1988)
– Unsupervised Pre-training (Ritter
and Kohonen, 1989)
– Auto-Encoder (Hinton et al., 1989)
– Back Propagating Convolutional
Neural Networks (LeCun et al., 1989,
1990a, 1998)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 72
Deep Learning, origin and growth
• 2000 – Era of Deep Learning
– NIPS 2003 Feature Selection
Challenge (Neal and Zhang, 2006)
– MNIST digit recognition (LeCun et
al., 1989)
– Deep Belief Network (DBN) /
Restricted Boltzmann Machines
(Hinton et al., 2006)
– Auto Encoders (Bengio, 2009)
• 2006
– GPU based CNN (Chellapilla et al.,
2006)
• 2009
– GPU DBN (Raina et al., 2009)
• 2011
– Max-Pooling CNN on the GPU
(Ciresan et al., 2011)
• 2012
– Image Net (Krizhevsky et al., 2012)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 73
DEEP LEARNING OF COMPLEX
REASONING FOR OCT TISSUE
CHARACTERIZATION
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 74
Exploring Deep Architecture
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 75
Multi-scale
modeling of
OCT speckles
Training
image
set
Ground
truth
Random forest
learning
Multi-scale
modeling of
OCT speckles
Test image
Labeled
tissue
Stacked Auto-
Encoders,
Logistic
Regression
Random
Forest
Training
image
set
Ground
truth
http://www.facweb.iitkgp.ernet.in/~debdoot/current.html
Auto Encoder for Deep Learning
Whispers of Speckles [Debdoot Sheet] - WMLMIA 7625 June 2015
Results in Wounds
Whispers of Speckles [Debdoot Sheet] - WMLMIA 77
(a) OCT image of wound (b) Ground truth (c) In situ histology
Epithelium, Papillary
dermis, Dermis, Adipose
Epithelium, Papillary
dermis, Dermis, Adipose
25 June 2015
Experiment Design
• Data Collection
– School of Medical Science
and Technology, Indian
Institute of Technology
Kharagpur
– 1300 nm (HPBW 100 nm)
Swept Source OCT System
• OCS 1300 SS, ThorLabs, NJ,
USA
• 8 bit bitmap images
– Histology for ground truth
• HE stained
• Samples
– Mus musculus (small mice)
– 16 healthy skin
– 2 wounds on skin
• DNN architecture
– Patch size – 36 × 36 px
– DAE1 – 400 nodes
– DAE2 – 100 nodes
– Target – Logistic Reg.
• 5 outputs
– Sparsity – 20%
– Mini-batch training
• In situ Histology
Performance
– Epithelium – 96%
– Papillary dermis – 93%
– Dermis – 99%
– Adipose tissue – 98%
Whispers of Speckles [Debdoot Sheet] - WMLMIA 7825 June 2015
Learning of Representations
Whispers of Speckles [Debdoot Sheet] - WMLMIA 79
Representation of speckle
appearance models learned by DAE1
Sparsity of representations learned by
DAE2
25 June 2015
END NOTE
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 80
Messages for Human Learning
• Photons interact characteristically with different tissues.
– Stochastic similarity exists in speckle appearance.
– Such representations are hard to heuristically encode.
• Deep learning and auto-encoders for computational imaging
– Speckle imaging application viz. OCT tissue characterization
– Hierarchical learning
• Locally embedded representations.
• Sparsity is in learned (auto-encoded) representations.
Whispers of Speckles [Debdoot Sheet] - WMLMIA 81
Queries: Debdoot Sheet (debdoot@ee.iitkgp.ernet.in)
25 June 2015
About Deep Learning
“It’s like in quantum physics at the beginning of the
20th century” Trishul Chilimbi (MSR, DNN, Adam)
“The experimentalists and practitioners were ahead
of the theoreticians. They couldn’t explain the
results. We appear to be at a similar stage with
DNNs. We’re realizing the power and the
capabilities, but we still don’t understand the
fundamentals of exactly how they work.”
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 82
Take home message
“We’ve humanized the scientist;
now we must scientize the
humanist. We didn’t try to cover
physics... we uncovered it.”
- Robert Resnick
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 83
Take home message
• Challenges
– Architectures
• Neural Nets vs. Others
– Implementation
• CPU vs. GPU vs. Cloud
– GPU (VLSI) architectures
• Hierarchical Temporal
Memory
• Potential Causal Connection
• Toolboxes
– Theano (Python/SciPy)
– Pylearn2
– Torch
– Caffe
– Matlab (Rasmus Berg Palm)
• More information
– www.deeplearning.net
– Schmidhuber (2014). Deep
Learning in Neural
Networks: An Overview
(arXiv:1404.7828)
– Bengio (2009). Learning
Deep Architectures for AI.
– Deng and Yu (2013). Deep
Learning: Methods and
Applications.
• Conferences
– Int. Conf. Learning
Representations (ICLR)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 84

Mais conteúdo relacionado

Mais procurados

Variational formulation of unsupervised deep learning for ultrasound image ar...
Variational formulation of unsupervised deep learning for ultrasound image ar...Variational formulation of unsupervised deep learning for ultrasound image ar...
Variational formulation of unsupervised deep learning for ultrasound image ar...Shujaat Khan
 
Darwin’s Magic: Evolutionary Computation in Nanoscience, Bioinformatics and S...
Darwin’s Magic: Evolutionary Computation in Nanoscience, Bioinformatics and S...Darwin’s Magic: Evolutionary Computation in Nanoscience, Bioinformatics and S...
Darwin’s Magic: Evolutionary Computation in Nanoscience, Bioinformatics and S...Natalio Krasnogor
 
A BRIEF OVERVIEW ON DIFFERENT ANIMAL DETECTION METHODS
A BRIEF OVERVIEW ON DIFFERENT ANIMAL DETECTION METHODSA BRIEF OVERVIEW ON DIFFERENT ANIMAL DETECTION METHODS
A BRIEF OVERVIEW ON DIFFERENT ANIMAL DETECTION METHODSsipij
 
Unsupervised Deconvolution Neural Network for High Quality Ultrasound Imaging
Unsupervised Deconvolution Neural Network for High Quality Ultrasound ImagingUnsupervised Deconvolution Neural Network for High Quality Ultrasound Imaging
Unsupervised Deconvolution Neural Network for High Quality Ultrasound ImagingShujaat Khan
 
International Journal of Image Processing (IJIP) Volume (3) Issue (6)
International Journal of Image Processing (IJIP) Volume (3) Issue (6)International Journal of Image Processing (IJIP) Volume (3) Issue (6)
International Journal of Image Processing (IJIP) Volume (3) Issue (6)CSCJournals
 
Deep Learning for Biomedical Unstructured Time Series
Deep Learning for Biomedical  Unstructured Time SeriesDeep Learning for Biomedical  Unstructured Time Series
Deep Learning for Biomedical Unstructured Time SeriesPetteriTeikariPhD
 
Trends in deep learning in 2020 - International Journal of Artificial Intelli...
Trends in deep learning in 2020 - International Journal of Artificial Intelli...Trends in deep learning in 2020 - International Journal of Artificial Intelli...
Trends in deep learning in 2020 - International Journal of Artificial Intelli...gerogepatton
 
Beyond Broken Stick Modeling: R Tutorial for interpretable multivariate analysis
Beyond Broken Stick Modeling: R Tutorial for interpretable multivariate analysisBeyond Broken Stick Modeling: R Tutorial for interpretable multivariate analysis
Beyond Broken Stick Modeling: R Tutorial for interpretable multivariate analysisPetteriTeikariPhD
 
OCT Monte Carlo & Deep Learning
OCT Monte Carlo & Deep LearningOCT Monte Carlo & Deep Learning
OCT Monte Carlo & Deep LearningPetteriTeikariPhD
 
From street photos to fashion trends leveraging user provided noisy labels fo...
From street photos to fashion trends leveraging user provided noisy labels fo...From street photos to fashion trends leveraging user provided noisy labels fo...
From street photos to fashion trends leveraging user provided noisy labels fo...harmonylab
 
Hyperspectral Retinal Imaging
Hyperspectral Retinal ImagingHyperspectral Retinal Imaging
Hyperspectral Retinal ImagingPetteriTeikariPhD
 
HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Est...
HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Est...HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Est...
HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Est...harmonylab
 
Morphing Aircraft Workshop 26/03/13 - Presentation on Auxetics
Morphing Aircraft Workshop 26/03/13 - Presentation on AuxeticsMorphing Aircraft Workshop 26/03/13 - Presentation on Auxetics
Morphing Aircraft Workshop 26/03/13 - Presentation on AuxeticsFabrizio Scarpa
 
IRJET - Automating the Identification of Forest Animals and Alerting in Case ...
IRJET - Automating the Identification of Forest Animals and Alerting in Case ...IRJET - Automating the Identification of Forest Animals and Alerting in Case ...
IRJET - Automating the Identification of Forest Animals and Alerting in Case ...IRJET Journal
 
Introduction to resting state fMRI preprocessing and analysis
Introduction to resting state fMRI preprocessing and analysisIntroduction to resting state fMRI preprocessing and analysis
Introduction to resting state fMRI preprocessing and analysisCameron Craddock
 
Practical Considerations in the design of Embedded Ophthalmic Devices
Practical Considerations in the design of Embedded Ophthalmic DevicesPractical Considerations in the design of Embedded Ophthalmic Devices
Practical Considerations in the design of Embedded Ophthalmic DevicesPetteriTeikariPhD
 
Pupillometry Through the Eyelids
Pupillometry Through the EyelidsPupillometry Through the Eyelids
Pupillometry Through the EyelidsPetteriTeikariPhD
 

Mais procurados (19)

Variational formulation of unsupervised deep learning for ultrasound image ar...
Variational formulation of unsupervised deep learning for ultrasound image ar...Variational formulation of unsupervised deep learning for ultrasound image ar...
Variational formulation of unsupervised deep learning for ultrasound image ar...
 
Advanced Retinal Imaging
Advanced Retinal ImagingAdvanced Retinal Imaging
Advanced Retinal Imaging
 
Darwin’s Magic: Evolutionary Computation in Nanoscience, Bioinformatics and S...
Darwin’s Magic: Evolutionary Computation in Nanoscience, Bioinformatics and S...Darwin’s Magic: Evolutionary Computation in Nanoscience, Bioinformatics and S...
Darwin’s Magic: Evolutionary Computation in Nanoscience, Bioinformatics and S...
 
A BRIEF OVERVIEW ON DIFFERENT ANIMAL DETECTION METHODS
A BRIEF OVERVIEW ON DIFFERENT ANIMAL DETECTION METHODSA BRIEF OVERVIEW ON DIFFERENT ANIMAL DETECTION METHODS
A BRIEF OVERVIEW ON DIFFERENT ANIMAL DETECTION METHODS
 
Unsupervised Deconvolution Neural Network for High Quality Ultrasound Imaging
Unsupervised Deconvolution Neural Network for High Quality Ultrasound ImagingUnsupervised Deconvolution Neural Network for High Quality Ultrasound Imaging
Unsupervised Deconvolution Neural Network for High Quality Ultrasound Imaging
 
International Journal of Image Processing (IJIP) Volume (3) Issue (6)
International Journal of Image Processing (IJIP) Volume (3) Issue (6)International Journal of Image Processing (IJIP) Volume (3) Issue (6)
International Journal of Image Processing (IJIP) Volume (3) Issue (6)
 
Deep Learning for Biomedical Unstructured Time Series
Deep Learning for Biomedical  Unstructured Time SeriesDeep Learning for Biomedical  Unstructured Time Series
Deep Learning for Biomedical Unstructured Time Series
 
Trends in deep learning in 2020 - International Journal of Artificial Intelli...
Trends in deep learning in 2020 - International Journal of Artificial Intelli...Trends in deep learning in 2020 - International Journal of Artificial Intelli...
Trends in deep learning in 2020 - International Journal of Artificial Intelli...
 
Beyond Broken Stick Modeling: R Tutorial for interpretable multivariate analysis
Beyond Broken Stick Modeling: R Tutorial for interpretable multivariate analysisBeyond Broken Stick Modeling: R Tutorial for interpretable multivariate analysis
Beyond Broken Stick Modeling: R Tutorial for interpretable multivariate analysis
 
OCT Monte Carlo & Deep Learning
OCT Monte Carlo & Deep LearningOCT Monte Carlo & Deep Learning
OCT Monte Carlo & Deep Learning
 
From street photos to fashion trends leveraging user provided noisy labels fo...
From street photos to fashion trends leveraging user provided noisy labels fo...From street photos to fashion trends leveraging user provided noisy labels fo...
From street photos to fashion trends leveraging user provided noisy labels fo...
 
Hyperspectral Retinal Imaging
Hyperspectral Retinal ImagingHyperspectral Retinal Imaging
Hyperspectral Retinal Imaging
 
HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Est...
HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Est...HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Est...
HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Est...
 
Morphing Aircraft Workshop 26/03/13 - Presentation on Auxetics
Morphing Aircraft Workshop 26/03/13 - Presentation on AuxeticsMorphing Aircraft Workshop 26/03/13 - Presentation on Auxetics
Morphing Aircraft Workshop 26/03/13 - Presentation on Auxetics
 
IRJET - Automating the Identification of Forest Animals and Alerting in Case ...
IRJET - Automating the Identification of Forest Animals and Alerting in Case ...IRJET - Automating the Identification of Forest Animals and Alerting in Case ...
IRJET - Automating the Identification of Forest Animals and Alerting in Case ...
 
Introduction to resting state fMRI preprocessing and analysis
Introduction to resting state fMRI preprocessing and analysisIntroduction to resting state fMRI preprocessing and analysis
Introduction to resting state fMRI preprocessing and analysis
 
Ai in healthcare
Ai in healthcareAi in healthcare
Ai in healthcare
 
Practical Considerations in the design of Embedded Ophthalmic Devices
Practical Considerations in the design of Embedded Ophthalmic DevicesPractical Considerations in the design of Embedded Ophthalmic Devices
Practical Considerations in the design of Embedded Ophthalmic Devices
 
Pupillometry Through the Eyelids
Pupillometry Through the EyelidsPupillometry Through the Eyelids
Pupillometry Through the Eyelids
 

Semelhante a Whispers of Speckles ( Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging) + (Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning)

tSAXS量測於半導體產業之應用
tSAXS量測於半導體產業之應用tSAXS量測於半導體產業之應用
tSAXS量測於半導體產業之應用CHENHuiMei
 
Global Cyberinfrastructure to Support e-Research
Global Cyberinfrastructure to Support e-ResearchGlobal Cyberinfrastructure to Support e-Research
Global Cyberinfrastructure to Support e-ResearchLarry Smarr
 
Education in a Globally Connected World
Education in a Globally Connected WorldEducation in a Globally Connected World
Education in a Globally Connected WorldLarry Smarr
 
Laboratory Raman spectroscopy ISP NASU
Laboratory Raman spectroscopy ISP NASULaboratory Raman spectroscopy ISP NASU
Laboratory Raman spectroscopy ISP NASUЮлия Деева
 
UG thesis presentation
UG thesis presentationUG thesis presentation
UG thesis presentationLuke Zhao
 
Introduction to Nanotechnology: Part 1
Introduction to Nanotechnology: Part 1Introduction to Nanotechnology: Part 1
Introduction to Nanotechnology: Part 1glennfish
 
Nanotechnology parasitology 20111112
Nanotechnology parasitology 20111112Nanotechnology parasitology 20111112
Nanotechnology parasitology 20111112Rajesh Karyakarte
 
Machine Learning at the (sub)Atomic Scale (or Are The Nanobots Nigh?)
Machine Learning at the (sub)Atomic Scale (or Are The Nanobots Nigh?)Machine Learning at the (sub)Atomic Scale (or Are The Nanobots Nigh?)
Machine Learning at the (sub)Atomic Scale (or Are The Nanobots Nigh?)Philip Moriarty
 
Biomimetics : Compound eyes
Biomimetics : Compound eyesBiomimetics : Compound eyes
Biomimetics : Compound eyesYoung Min Song
 
ASME JOURNAL PAPER - med_008_03_030934
ASME JOURNAL PAPER - med_008_03_030934ASME JOURNAL PAPER - med_008_03_030934
ASME JOURNAL PAPER - med_008_03_030934Sachin Bijadi
 
Missori_THz.pptx
Missori_THz.pptxMissori_THz.pptx
Missori_THz.pptxamirjodeiry
 
The Future of the Internet Enabling New Science
The Future of the Internet Enabling New ScienceThe Future of the Internet Enabling New Science
The Future of the Internet Enabling New ScienceLarry Smarr
 
Analytical methods and instrumentation syllabus
Analytical methods and instrumentation  syllabusAnalytical methods and instrumentation  syllabus
Analytical methods and instrumentation syllabusThivya Prasad
 
CZECH NANO SHOW - Marketa Borovcova - CEITEC
CZECH NANO SHOW - Marketa Borovcova - CEITEC CZECH NANO SHOW - Marketa Borovcova - CEITEC
CZECH NANO SHOW - Marketa Borovcova - CEITEC Jan Fried
 
How Fiber Optics are Transforming our World
How Fiber Optics are Transforming our WorldHow Fiber Optics are Transforming our World
How Fiber Optics are Transforming our WorldLarry Smarr
 
Calit2--Helping the University of California Drive Innovation in California
Calit2--Helping the University of California Drive Innovation in CaliforniaCalit2--Helping the University of California Drive Innovation in California
Calit2--Helping the University of California Drive Innovation in CaliforniaLarry Smarr
 
Lights, Nano, Action! - Assistant Professor Jennifer Dionne
Lights, Nano, Action! - Assistant Professor Jennifer DionneLights, Nano, Action! - Assistant Professor Jennifer Dionne
Lights, Nano, Action! - Assistant Professor Jennifer DionneStanford School of Engineering
 
Active metamaterials
Active metamaterialsActive metamaterials
Active metamaterialsMichael Singh
 
The Singularity: Toward a Post-Human Reality
The Singularity: Toward a Post-Human RealityThe Singularity: Toward a Post-Human Reality
The Singularity: Toward a Post-Human RealityLarry Smarr
 

Semelhante a Whispers of Speckles ( Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging) + (Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning) (20)

tSAXS量測於半導體產業之應用
tSAXS量測於半導體產業之應用tSAXS量測於半導體產業之應用
tSAXS量測於半導體產業之應用
 
Teleportation
TeleportationTeleportation
Teleportation
 
Global Cyberinfrastructure to Support e-Research
Global Cyberinfrastructure to Support e-ResearchGlobal Cyberinfrastructure to Support e-Research
Global Cyberinfrastructure to Support e-Research
 
Education in a Globally Connected World
Education in a Globally Connected WorldEducation in a Globally Connected World
Education in a Globally Connected World
 
Laboratory Raman spectroscopy ISP NASU
Laboratory Raman spectroscopy ISP NASULaboratory Raman spectroscopy ISP NASU
Laboratory Raman spectroscopy ISP NASU
 
UG thesis presentation
UG thesis presentationUG thesis presentation
UG thesis presentation
 
Introduction to Nanotechnology: Part 1
Introduction to Nanotechnology: Part 1Introduction to Nanotechnology: Part 1
Introduction to Nanotechnology: Part 1
 
Nanotechnology parasitology 20111112
Nanotechnology parasitology 20111112Nanotechnology parasitology 20111112
Nanotechnology parasitology 20111112
 
Machine Learning at the (sub)Atomic Scale (or Are The Nanobots Nigh?)
Machine Learning at the (sub)Atomic Scale (or Are The Nanobots Nigh?)Machine Learning at the (sub)Atomic Scale (or Are The Nanobots Nigh?)
Machine Learning at the (sub)Atomic Scale (or Are The Nanobots Nigh?)
 
Biomimetics : Compound eyes
Biomimetics : Compound eyesBiomimetics : Compound eyes
Biomimetics : Compound eyes
 
ASME JOURNAL PAPER - med_008_03_030934
ASME JOURNAL PAPER - med_008_03_030934ASME JOURNAL PAPER - med_008_03_030934
ASME JOURNAL PAPER - med_008_03_030934
 
Missori_THz.pptx
Missori_THz.pptxMissori_THz.pptx
Missori_THz.pptx
 
The Future of the Internet Enabling New Science
The Future of the Internet Enabling New ScienceThe Future of the Internet Enabling New Science
The Future of the Internet Enabling New Science
 
Analytical methods and instrumentation syllabus
Analytical methods and instrumentation  syllabusAnalytical methods and instrumentation  syllabus
Analytical methods and instrumentation syllabus
 
CZECH NANO SHOW - Marketa Borovcova - CEITEC
CZECH NANO SHOW - Marketa Borovcova - CEITEC CZECH NANO SHOW - Marketa Borovcova - CEITEC
CZECH NANO SHOW - Marketa Borovcova - CEITEC
 
How Fiber Optics are Transforming our World
How Fiber Optics are Transforming our WorldHow Fiber Optics are Transforming our World
How Fiber Optics are Transforming our World
 
Calit2--Helping the University of California Drive Innovation in California
Calit2--Helping the University of California Drive Innovation in CaliforniaCalit2--Helping the University of California Drive Innovation in California
Calit2--Helping the University of California Drive Innovation in California
 
Lights, Nano, Action! - Assistant Professor Jennifer Dionne
Lights, Nano, Action! - Assistant Professor Jennifer DionneLights, Nano, Action! - Assistant Professor Jennifer Dionne
Lights, Nano, Action! - Assistant Professor Jennifer Dionne
 
Active metamaterials
Active metamaterialsActive metamaterials
Active metamaterials
 
The Singularity: Toward a Post-Human Reality
The Singularity: Toward a Post-Human RealityThe Singularity: Toward a Post-Human Reality
The Singularity: Toward a Post-Human Reality
 

Último

Keynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designKeynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designMIPLM
 
Choosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for ParentsChoosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for Parentsnavabharathschool99
 
Global Lehigh Strategic Initiatives (without descriptions)
Global Lehigh Strategic Initiatives (without descriptions)Global Lehigh Strategic Initiatives (without descriptions)
Global Lehigh Strategic Initiatives (without descriptions)cama23
 
Concurrency Control in Database Management system
Concurrency Control in Database Management systemConcurrency Control in Database Management system
Concurrency Control in Database Management systemChristalin Nelson
 
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdfInclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdfTechSoup
 
4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptx4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptxmary850239
 
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...Nguyen Thanh Tu Collection
 
Integumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.pptIntegumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.pptshraddhaparab530
 
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITY
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITYISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITY
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITYKayeClaireEstoconing
 
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATIONTHEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATIONHumphrey A Beña
 
Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Celine George
 
ANG SEKTOR NG agrikultura.pptx QUARTER 4
ANG SEKTOR NG agrikultura.pptx QUARTER 4ANG SEKTOR NG agrikultura.pptx QUARTER 4
ANG SEKTOR NG agrikultura.pptx QUARTER 4MiaBumagat1
 
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...Postal Advocate Inc.
 
Food processing presentation for bsc agriculture hons
Food processing presentation for bsc agriculture honsFood processing presentation for bsc agriculture hons
Food processing presentation for bsc agriculture honsManeerUddin
 
4.18.24 Movement Legacies, Reflection, and Review.pptx
4.18.24 Movement Legacies, Reflection, and Review.pptx4.18.24 Movement Legacies, Reflection, and Review.pptx
4.18.24 Movement Legacies, Reflection, and Review.pptxmary850239
 
Earth Day Presentation wow hello nice great
Earth Day Presentation wow hello nice greatEarth Day Presentation wow hello nice great
Earth Day Presentation wow hello nice greatYousafMalik24
 

Último (20)

YOUVE GOT EMAIL_FINALS_EL_DORADO_2024.pptx
YOUVE GOT EMAIL_FINALS_EL_DORADO_2024.pptxYOUVE GOT EMAIL_FINALS_EL_DORADO_2024.pptx
YOUVE GOT EMAIL_FINALS_EL_DORADO_2024.pptx
 
Keynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designKeynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-design
 
Choosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for ParentsChoosing the Right CBSE School A Comprehensive Guide for Parents
Choosing the Right CBSE School A Comprehensive Guide for Parents
 
Global Lehigh Strategic Initiatives (without descriptions)
Global Lehigh Strategic Initiatives (without descriptions)Global Lehigh Strategic Initiatives (without descriptions)
Global Lehigh Strategic Initiatives (without descriptions)
 
Concurrency Control in Database Management system
Concurrency Control in Database Management systemConcurrency Control in Database Management system
Concurrency Control in Database Management system
 
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdfInclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
Inclusivity Essentials_ Creating Accessible Websites for Nonprofits .pdf
 
LEFT_ON_C'N_ PRELIMS_EL_DORADO_2024.pptx
LEFT_ON_C'N_ PRELIMS_EL_DORADO_2024.pptxLEFT_ON_C'N_ PRELIMS_EL_DORADO_2024.pptx
LEFT_ON_C'N_ PRELIMS_EL_DORADO_2024.pptx
 
FINALS_OF_LEFT_ON_C'N_EL_DORADO_2024.pptx
FINALS_OF_LEFT_ON_C'N_EL_DORADO_2024.pptxFINALS_OF_LEFT_ON_C'N_EL_DORADO_2024.pptx
FINALS_OF_LEFT_ON_C'N_EL_DORADO_2024.pptx
 
4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptx4.16.24 21st Century Movements for Black Lives.pptx
4.16.24 21st Century Movements for Black Lives.pptx
 
Raw materials used in Herbal Cosmetics.pptx
Raw materials used in Herbal Cosmetics.pptxRaw materials used in Herbal Cosmetics.pptx
Raw materials used in Herbal Cosmetics.pptx
 
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
 
Integumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.pptIntegumentary System SMP B. Pharm Sem I.ppt
Integumentary System SMP B. Pharm Sem I.ppt
 
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITY
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITYISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITY
ISYU TUNGKOL SA SEKSWLADIDA (ISSUE ABOUT SEXUALITY
 
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATIONTHEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
THEORIES OF ORGANIZATION-PUBLIC ADMINISTRATION
 
Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17
 
ANG SEKTOR NG agrikultura.pptx QUARTER 4
ANG SEKTOR NG agrikultura.pptx QUARTER 4ANG SEKTOR NG agrikultura.pptx QUARTER 4
ANG SEKTOR NG agrikultura.pptx QUARTER 4
 
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...
USPS® Forced Meter Migration - How to Know if Your Postage Meter Will Soon be...
 
Food processing presentation for bsc agriculture hons
Food processing presentation for bsc agriculture honsFood processing presentation for bsc agriculture hons
Food processing presentation for bsc agriculture hons
 
4.18.24 Movement Legacies, Reflection, and Review.pptx
4.18.24 Movement Legacies, Reflection, and Review.pptx4.18.24 Movement Legacies, Reflection, and Review.pptx
4.18.24 Movement Legacies, Reflection, and Review.pptx
 
Earth Day Presentation wow hello nice great
Earth Day Presentation wow hello nice greatEarth Day Presentation wow hello nice great
Earth Day Presentation wow hello nice great
 

Whispers of Speckles ( Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging) + (Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning)

  • 1. 1 WHISPERS OF SPECKLES DEBDOOT SHEET LAUNCHING THIS MONSOON Venue: IIT Mandi Date: Thursday, 25 June 2015 Time: 11 am – 12:30 pm
  • 2. Whispers of Speckle Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging Dr. Debdoot Sheet Assistant Professor Department of Electrical Engineering Indian Institute of Technology Kharagpur www.facweb.iitkgp.ernet.in/~debdoot/
  • 3. Inspiration “A wonderful fact to reflect upon, that every human creature is constituted to be that profound secret and mystery to every other.” - Charles Dickens (A Tale of Two Cities) “If you want to find the secrets of the universe, think in terms of energy, frequency and vibration.” - Nikola Tesla 25 June 2015 3Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 4. Motivation Whispers of Speckles [Debdoot Sheet] - WMLMIA 4 D. Sheet (2014), PhD Thesis 25 June 2015 Text books R. K. Das (2012), PhD Thesis A. Barui (2011), PhD Thesis
  • 5. Introduction • Human body consists of organs and systems made up of different tissues. • Pathological conditions and abnormalities affect their normal functioning. • Critical soft tissue abnormalities include – Plaque formation in the blood vascular system. – Lesions in the breast. – Degeneration of the retina. – Wounds in the skin. • Traditional practice of Histopathological diagnosis requires invasive Biopsy / Excision for tissue collection – Not possible in vessels in living Humans – Improper sampling from Breast lesion affects diagnostic outcome – Not possible in retina in living Humans – Not possible in healing wounds. 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 5
  • 6. ACHIEVING IN SITU HISTOLOGY OF VASCULAR PLAQUES 625 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 7. • Atherosclerosis – Plaque builds up in arteries – Forms anywhere in the vascular system • Cardiovascular diseases (CVD) • In vivo Imaging of Plaques – CT Angiography (CTA) – MR Angiography (MRA) – Intravascular Ultrasound (IVUS) – Intravascular OCT (IV-OCT) – Intravascular Near-Infrared Spectroscopy (NIR) • Plaque Vulnerability Assessment – Calcification, fibrosis identification – Lipid pool and Necrosis burden estimation Source: NIH – National Heart, Lung, and Blood Institute Blood Vascular System 25 June 2015 7Whispers of Speckles [Debdoot Sheet] - WMLMIA • Spectral analysis of received ultrasonic echo signal – Lizzi et al., 1983 – Nair et al., 2001 – Kawasaki et al., 2002 – Virtual Histology (Volcano Corp.) – iMap (BostonScientific) • Texture analysis of B-mode image/signal – Katouzian et al., 2008, 2010, 2012 (Prog. Hist. / PH) – Esclara et al., 2009 – Seabra et al., 2011 • Limitations – Unable to identify heterogeneous tissue composition – Cannot discriminate between dense fibrous tissue and calcification – Fails to discriminate true necrosis from shadows
  • 8. Backdrop 8 White light source 350nm 750nm Power Stained tissue section Tissue specific spectrum 350nm 750nm Power Calcified Fibrotic 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 9. 9 White light source 350nm 750nm Power Stained tissue section Tissue specific spectrum 350nm 750nm Power Calcified Fibrotic  : Probing energy (Light)  : Physiological property (Tissue type)   f   1  f Inferring tissue type based on color 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 10. 10 Computation Modelling of Tissue Energy Interaction for In situ Histopathology Computed histology  : Probing energy (Acoustic)  : Tissue type (Backscatterer density)   f   1  f Inferring tissue type based on backscattering response 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 11. Limited Resolution Challenge 11 r1 r2 r3 P. M. Shankar, “A general statistical model for ultrasonic backscattering from tissues”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control., vol. 47, no. 3, pp. 727-736, May 2000.  11 rr f    22 rr f    33 rr f   Ultrasound signal backscattered within a resolution cell     i i r r fE E     Signal sensed by the transducer    irfEf  1 ˆ   Estimated functional ensemble of backscatterer density  ˆ Improper estimation of tissue type in inhomogeneous media 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 12. Statistical Physics in Acoustic Imaging r1 r2 r3 r1 r2 m=0.5 Ω1 Ω2 r P(r) m=1.0 Ω1 Ω2 Ω3 r P(r) P. M. Shankar, “A general statistical model for ultrasonic backscattering from tissues”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control., vol. 47, no. 3, pp. 727-736, May 2000.                2 12 exp 2 ,| r m m rm mr m mm N 25 June 2015 12Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 13. Statistical physics of ultrasonic backscattering Lipidic r P(r) Fibrotic r P(r) Calcified r P(r) V. Dumane and P. M. Shankar, “Use of frequency diversity and Nakagami statistics in ultrasonic tissue characterization”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control, vol. 48, no. 4, pp. 1139-1146, Jul. 2001 F. Destrempes, J. Meunier, M. . F. Giroux, G. Soulez, G. Cloutier, “Segmentation in ultrasonic b- mode images of healthy carotid arteries using mixture of Nakagami distributions and stochastic optimization”, IEEE Trans. Med. Imaging, vol. 28, no. 2, pp. 215-229, Feb. 2009. 25 June 2015 13Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 14.                                     32121 221121 ,,, 1 , 1 ,, 1 , 111 )(,|,| ,| ;),(,),,,,,(||        L l lll L l lll L l lll mrpmrpp mrpp ymprfyrp NN N Mathematical intractability, the problem )( )( )|( )|( yP rp yrp ryp  The probabilistic decision making framework Scales unknown Correlation among scales unknown No. components unknown Prior probab. of each comp. unknown 25 June 2015 14Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 15. Proposed Solution Statistical physics model of ultrasonic backscattering Set of signal received by the transducer Training set of annotated examples to be used for supervised learning Supervised learner for learning tissue specific statistical physics model   train;,|)( ),( )|,( ),|( RR    yHyP rp yrp ryp Solution through Transfer Learning Framework25 June 2015 15Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 16. HOW TO DEAL WITH THIS AS A MACHINE LEARNING CHALLENGE? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 16
  • 17. Learning? A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E -Tom Mitchell 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 17
  • 18. Demystifying Learning 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 18 Man 1 Man 2 Man 3Man 4 Great Wall logo Great Wall tower Kim Jung WangDebdoot Experience (E) Performance(P) Debdoot, Kim, Jung and Wang are standing near the Great Wall logo and the Great Wall tower is behind them.
  • 19. How was it Learning? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 19 Man 1 Man 2 Man 3Man 4 Great Wall logo Great Wall tower Kim Jung WangDebdoot Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Debdoot, Kim, Jung and Wang are standing near the Great Wall logo and the Great Wall tower is behind them. Recognize humans
  • 20. GETTING MACHINES TO LEARN TISSUE – ENERGY INTERACTION FOR IN SITU HISTOLOGY 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 20
  • 21. IVUS Tissue Characterization 21 Background Lipidic Fibrotic Calcified Necrosis Iterative self-organizing atherosclerotic tissue labeling in intravascular ultrasound images and comparison with virtual histology, IEEE TBME, 59(11), 2012 Hunting for necrosis in the shadows of intravascular ultrasound, CMIG, 38(2), 2014 Joint learning of ultrasonic backscattering statistical physics and signal confidence primal for characterizing atherosclerotic plaques using intravascular ultrasound, Med. Image Anal,18(1), 2014 Nakagami parameter and signal confidence estimate Random forest learning 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 22. Ultrasound Signal Confidence • An ultrasonic pulse as well as backscattered echo travel along the same path through a heterogeneous media. • They are subjected to the same attenuation. • Confidence of the received signal is a reflection of fidelity of samples received by the transducer. • It can be estimated by treating its propagation as a random walk along an ultrasonic scan-line. • A random walker starting at a point on the scan-line reaches the virtual transducer element placed at the origin of each scan- line. • This random walk is solved using the electric network equivalent and solving it in the paradigm of graph theory. 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 22 A. Karamalis, W. Wein, T. Klein, N. Navab (2012) Ultrasound confidence maps using random walks, Medical Image Analysis, 16:1101–1112.
  • 23. Transfer Learning Framework 23 Ultrasound RF data (i) Signal confidence (ii) Speckle statistics Tissue labels  f Learnt random forest Learning phase (offline) Tissue labels Prediction (online)  f Whispers of Speckles [Debdoot Sheet] - WMLMIA25 June 2015
  • 24. Random Forests for Learning 25 June 2015 24Whispers of Speckles [Debdoot Sheet] - WMLMIA A. Criminisi and J. Shotton, Decision Forests for Computer Vision and Medical Image Analysis, Springer, 2013.
  • 25. Experiment Design • Data Collection: – Columbia University, New York City, NY, USA – Interventional Cardiologist: Dr. Stephane G. Carlier – Cardiovascular Histopathologist: Dr. Renu Virmani, CV Path Institute, Gaithersburg, USA – Cases # 13 – Tissue Sections # 53 – Atlantis, 40 MHz IVUS, Boston Scientific,CA, USA – Sampling freq: 400 MHz – Sampling geometry: 256 scan lines per rotation, 2048 samples per scan line • Learning – Source task: {Ω,m} estimated at 28 scales + Ultrasonic Confidence (A. Karamalis, et al. (2012)) – Target task: Random forest 50 decision trees • Cross validation – 53 fold cross validation – Learn with 52, test on the remaining 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 25
  • 26. Ultrasonic Histology of Atherosclerotic Plaques • Characterization based on ultrasonic statistical physics. • Superior machine learning algorithm. • Reliability measure for estimation of tissues. Probability of Calcified tissues Probability of Fibrotic tissues Probability of Lipidic tissues Probability of Necrotic tissues Calcified Lipidic Fibrotic Necrotic 25 June 2015 26Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 27. Coronary Plaque Characterization and Staging 25 June 2015 27Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 28. SOME (AWESOME SCORES ON) METRICS 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 28
  • 29. Performance Evaluation 25 June 2015 29 Inter-observer variability Intra-observer variability Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 30. ANALYZING COMPLEXITY OF MACHINE LEARNING 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 30
  • 31. Computational Complexity Training Complexity Testing Complexity 25 June 2015 31Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 32. Roles of Source Knowledge (Features) Feature 1 Feature2 25 June 2015 32Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 33. Importance of Source Knowledge Genuer, R., Poggi, J.-M., Tuleau-Malot, C., (2010). Variable selection using random forests. Pat. Recog. Letters. 31(14):2225-2236 25 June 2015 33Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 34. Learning from Approximately Labeled Minimum Samples 25 June 2015 34 Donomez, P., Lebanon, G., Balasubramanian, K., (2010). Unsupervised supervised learning I: Estimating Classification and Regression Errors without Labels, J. Mach. Learn. Res. 11:1323-1351 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 35. Variation of Residual Error in Learning 25 June 2015 35 Learning example Test results Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 36. END NOTE 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 36
  • 37. Collaboration and Network 25 June 2015 37Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 38. Take home message • Different types of soft tissues have characteristic response when interacting with acoustic energy. • Heterogeneous tissues can be identified by learning of tissue specific energy interaction response using statistical physics models. • Transfer Learning can be employed for efficiently solving tissue characterization problems modeled as tissue-energy interaction problems. – CPU/GPU handshaking can be used for fast implementation of such tasks • Explore possibility of Functional Histopathology In situ 25 June 2015 38Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 39. Whispers of Speckle Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning Dr. Debdoot Sheet Assistant Professor Department of Electrical Engineering Indian Institute of Technology Kharagpur www.facweb.iitkgp.ernet.in/~debdoot/
  • 40. DOES THIS METHOD OF TRANSFER LEARNING APPLY ONLY TO ULTRASONIC IMAGING? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 40
  • 41. Skin• Skin forms the general covering of the body protecting us from external influences. • Functions – Thermoregulation – Sweat secretion – Tactile, pressure, temperature sensing • Stratified organization – Epidermis – Papillary dermis – Dermis – Adipose tissue • Wound – Major pathological injury – Skin is torn, cut, punctured • Clinical challenge in management – Healing in person specific – Patient specific intervention – In situ investigation of healing is challenge 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 41
  • 42. Skin 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 42 • In situ investigation – Optical Coherence Tomography (OCT) • Cobb et al., (2006) • Barui et al., (2011) – Optical photography • Cross-sectional information about healing wound is not available – NIR imaging • Cross-section histological information not present • In situ Histology with OCT – G. van Soest et al., (2010) – Cardiovascular OCT – A. Barui et al., (2011) – Cutaneous wound beds. • Challenges – Identify co-located tissue heterogeneity – Identify and discriminate Inter-digitated structures
  • 43. Tissue Photon Interaction Whispers of Speckles [Debdoot Sheet] - WMLMIA 43 Incident radiation Regular reflection Diffuse reflection Scattering Absorption Multispectral optical imageOCT B. Saleh, Introduction to Subsurface Imaging, Cambridge, 2011. 0.5 mm 0.5 mm 25 June 2015
  • 44. Optical Coherence Tomography Whispers of Speckles [Debdoot Sheet] - WMLMIA 44 Low time-coherence light source Depth scan mirror Sample Detector Source beam Reference beam Sample beam Detector beam x z z OCT Image Michelson interferometer 25 June 2015
  • 45. TPI in Swept Source OCT Whispers of Speckles [Debdoot Sheet] - WMLMIA 45 Source Ballistic backscattering Non-ballistic backscattering Reference Detector A. F. Fercher, et al, Optical coherence tomography — principles and applications, Rep. Prog. Phys. 66 (2003) 239–303 Epithelium Papillary dermis Dermis Adipose Speckle intensity Probability density 25 June 2015          S S S S I Ip  exp 1
  • 46. COMPUTATIONAL OPTICAL COHERENCE HISTOLOGY THROUGH TRANSFER LEARNING 4625 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 47. Framework 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 47 Learn TPI Model Training Image Ground Truth Labels Test Image Learn TPI Model Characterized tissue   train;,| II, xH
  • 48. Computational Histology of Skin • Solution through a transfer learning approach • Performance benchmark (Accuracy) – Epithelium = 99% – Papilary dermis = 95% – Dermis = 99% – Adipose = 98% • D. Sheet, et al, “In situ histology of mice skin through transfer learning of tissue energy interaction in optical coherence tomography”, J. Biomed. Optics, 18(9), 2013. 25 June 2015 48 Multi-scale modeling of OCT speckles Training image set Ground truth Random forest learning Multi-scale modeling of OCT speckles Test image Labeled tissue Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 49. In situ Histology of Skin OCT Histo Epithelium Epithelium Papillary dermis Dermis Adipose tissue 25 June 2015 49 Papillary dermisDermisAdipose tissueAll tissues In situ histology of mice skin through transfer learning of tissue energy interaction in optical coherence tomography, J. Biomed. Optics, 18(9), 2013 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 50. In vitro validation towards In vivo translation 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 50 Transfer Learning of Tissue Photon Interaction in Optical Coherence Tomography towards In vivo Histology of the Oral Mucosa, Proc. ISBI, 2014.
  • 51. Computational Histology of Retina • Transfer learning approach – Retinal OCT tissue labeling • Performance benchmark (Accuracy) – Anterior coat > 98% – RPE > 92% – Posterior coat > 99% • SPK Karri and D. Sheet, et al., “Computational Histology of Retina through Transfer Learning of Tissue Photon Interaction in Optical Coherence Tomography”, Proc. Int. Symp. Biomedical Imaging (ISBI), 2014. 25 June 2015 51 Multi-scale modeling of OCT speckles Training image set Ground truth Random forest learning Multi-scale modeling of OCT speckles Test image Labeled tissue Whispers of Speckles [Debdoot Sheet] - WMLMIA
  • 52. DOES SOMETHING LOOK FISHY? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 52
  • 53. State of the Art • In situ Histology with OCT – G. van Soest et al., (2010), G. J. Ughi et al., (2013) – Cardiovascular OCT – D. Sheet et al., (2013, 2014) – Cutaneous wounds, oral • Challenges – Heuristic features • Texture • Intensity statistics – Heuristic computational models • Transfer learning of speckle occurrence models – Incomplete representation dictionary Whispers of Speckles [Debdoot Sheet] - WMLMIA 53 Multi-scale modeling of OCT speckles Training image set Ground truth Random forest learning Multi-scale modeling of OCT speckles Test image Labeled tissue 25 June 2015
  • 54. Heuristics in State of Art Whispers of Speckles [Debdoot Sheet] - WMLMIA 5425 June 2015
  • 55. (RE)EXPLORING THE CONCEPTS OF HIERARCHY IN LEARNING 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 55
  • 56. How was it Learning? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 56 Man 1 Man 2 Man 3Man 4 Great Wall logo Great Wall tower Kim Jung WangDebdoot Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Debdoot, Kim, Jung and Wang are standing near the Great Wall logo and the Great Wall tower is behind them. Recognize humans
  • 57. Challenges 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 57 Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Describe Scene
  • 58. Challenges 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 58 Salient Segments Objectify Recognize inanimate Describe Scene Recognize humans LBP Wavelets HoG Body part recognition Human appearance Chroma clustering Posture realign Silhouette matching Recognize human Detect humans
  • 59. FROM SHALLOW TO COMPLEX REASONING 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 59
  • 60. Heuristics in State of Art Whispers of Speckles [Debdoot Sheet] - WMLMIA 6025 June 2015
  • 61. The Solution Whispers of Speckles [Debdoot Sheet] - WMLMIA 61 DenoisingAutoEncoder DenoisingAutoEncoder LogisticReg. 25 June 2015
  • 62. Using a Deep Network Whispers of Speckles [Debdoot Sheet] - WMLMIA 6225 June 2015
  • 63. COMPLEX REASONING AND ITS DEEP LEARNING 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 63
  • 64. Challenges 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 64 Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Describe Scene
  • 65. Challenges 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 65 Salient Segments Objectify Recognize inanimate Describe Scene Recognize humans LBP Wavelets HoG Body part recognition Human appearance Chroma clustering Posture realign Silhouette matching Recognize human Detect humans
  • 66. How to tackle this dilemma? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 66 Great Wall behind Great Wall logo beside Debdoot, Kim, Jung, Wang
  • 67. Multilayer Perceptron (MLP) 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 67 Hiddenlayers Hiddenlayers Hiddenlayers
  • 68. MLP Learning, troubles thereof 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 68 P T1 T2
  • 69. MLP Learning troubles, why so? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 69 P T1 T2 LBP Wavelets HoG Body part recognition Human appearance Chroma clustering Posture realign Silhouette matching Recognize human?
  • 70. HOW TO DEEP LEARN? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 70
  • 71. Deep Learning, origin and growth • Around 1950 – NN age – Neural Nets (McCulloch and Pitts, 1943) – Unsupervised Learn. (Hebb, 1949) – Supervised Learn. (Rosenblatt, 1958) – Associative Memory (Palm, 1980; Hopfield, 1982) • 1960 – Discovery of visual sensory cells that respond to Edges (Hubel and Wiesel, 1962) – Feed Forward Multi Layer Perceptron (FF-MLP) (Ivakhnenko, 1968) • 1980 – Neocognition – Convolution + WeightReplication + Subsampling (Fukushima, 1980) – Max Pooling – Back-propagation (Werbos, 1981; LeCunn, 1985, 1988) 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 71
  • 72. Deep Learning, origin and growth • 1980-2000 – Search for simple, low-complexity, problem-solvers – Recurrent Neural Network (RNN) (Hochreiter and Schmidhuber, 1996) – Local learning Feed forward NN (Dayan and Hinton, 1996) – Advanced gradient descent (Schaback and Werner, 1992) – Sequential Network Construction (Honavar and Uhr, 1988) – Unsupervised Pre-training (Ritter and Kohonen, 1989) – Auto-Encoder (Hinton et al., 1989) – Back Propagating Convolutional Neural Networks (LeCun et al., 1989, 1990a, 1998) 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 72
  • 73. Deep Learning, origin and growth • 2000 – Era of Deep Learning – NIPS 2003 Feature Selection Challenge (Neal and Zhang, 2006) – MNIST digit recognition (LeCun et al., 1989) – Deep Belief Network (DBN) / Restricted Boltzmann Machines (Hinton et al., 2006) – Auto Encoders (Bengio, 2009) • 2006 – GPU based CNN (Chellapilla et al., 2006) • 2009 – GPU DBN (Raina et al., 2009) • 2011 – Max-Pooling CNN on the GPU (Ciresan et al., 2011) • 2012 – Image Net (Krizhevsky et al., 2012) 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 73
  • 74. DEEP LEARNING OF COMPLEX REASONING FOR OCT TISSUE CHARACTERIZATION 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 74
  • 75. Exploring Deep Architecture 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 75 Multi-scale modeling of OCT speckles Training image set Ground truth Random forest learning Multi-scale modeling of OCT speckles Test image Labeled tissue Stacked Auto- Encoders, Logistic Regression Random Forest Training image set Ground truth http://www.facweb.iitkgp.ernet.in/~debdoot/current.html
  • 76. Auto Encoder for Deep Learning Whispers of Speckles [Debdoot Sheet] - WMLMIA 7625 June 2015
  • 77. Results in Wounds Whispers of Speckles [Debdoot Sheet] - WMLMIA 77 (a) OCT image of wound (b) Ground truth (c) In situ histology Epithelium, Papillary dermis, Dermis, Adipose Epithelium, Papillary dermis, Dermis, Adipose 25 June 2015
  • 78. Experiment Design • Data Collection – School of Medical Science and Technology, Indian Institute of Technology Kharagpur – 1300 nm (HPBW 100 nm) Swept Source OCT System • OCS 1300 SS, ThorLabs, NJ, USA • 8 bit bitmap images – Histology for ground truth • HE stained • Samples – Mus musculus (small mice) – 16 healthy skin – 2 wounds on skin • DNN architecture – Patch size – 36 × 36 px – DAE1 – 400 nodes – DAE2 – 100 nodes – Target – Logistic Reg. • 5 outputs – Sparsity – 20% – Mini-batch training • In situ Histology Performance – Epithelium – 96% – Papillary dermis – 93% – Dermis – 99% – Adipose tissue – 98% Whispers of Speckles [Debdoot Sheet] - WMLMIA 7825 June 2015
  • 79. Learning of Representations Whispers of Speckles [Debdoot Sheet] - WMLMIA 79 Representation of speckle appearance models learned by DAE1 Sparsity of representations learned by DAE2 25 June 2015
  • 80. END NOTE 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 80
  • 81. Messages for Human Learning • Photons interact characteristically with different tissues. – Stochastic similarity exists in speckle appearance. – Such representations are hard to heuristically encode. • Deep learning and auto-encoders for computational imaging – Speckle imaging application viz. OCT tissue characterization – Hierarchical learning • Locally embedded representations. • Sparsity is in learned (auto-encoded) representations. Whispers of Speckles [Debdoot Sheet] - WMLMIA 81 Queries: Debdoot Sheet (debdoot@ee.iitkgp.ernet.in) 25 June 2015
  • 82. About Deep Learning “It’s like in quantum physics at the beginning of the 20th century” Trishul Chilimbi (MSR, DNN, Adam) “The experimentalists and practitioners were ahead of the theoreticians. They couldn’t explain the results. We appear to be at a similar stage with DNNs. We’re realizing the power and the capabilities, but we still don’t understand the fundamentals of exactly how they work.” 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 82
  • 83. Take home message “We’ve humanized the scientist; now we must scientize the humanist. We didn’t try to cover physics... we uncovered it.” - Robert Resnick 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 83
  • 84. Take home message • Challenges – Architectures • Neural Nets vs. Others – Implementation • CPU vs. GPU vs. Cloud – GPU (VLSI) architectures • Hierarchical Temporal Memory • Potential Causal Connection • Toolboxes – Theano (Python/SciPy) – Pylearn2 – Torch – Caffe – Matlab (Rasmus Berg Palm) • More information – www.deeplearning.net – Schmidhuber (2014). Deep Learning in Neural Networks: An Overview (arXiv:1404.7828) – Bengio (2009). Learning Deep Architectures for AI. – Deng and Yu (2013). Deep Learning: Methods and Applications. • Conferences – Int. Conf. Learning Representations (ICLR) 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 84