Image annotation and segmentation for gastroenterology use-cases.

1. Image: Annotation &
Segmentation
Taposh Dutta Roy
Sr. Manager, Data Science & Innovation
Kaiser Permanente
Oakland, CA USA

2. Agenda
What is Annotation?
Types of Image Annotation
Supervised Strategies
Unsupervised Strategies

3. What is Annotation?
An annotation is extra information associated with a particular point in
a file or document or other piece of information. It can be a note that
includes a comment or explanation.[1]
Annotation can be done for any type of data – text, image, sound or
video
[1] https://www.merriam-webster.com/dictionary/annotation

4. How would we annotate these images?
A B C

5. What are these images of?
A B C
Annotate : Black
and white cow
Annotate : white
dog?
Annotate : white
cow?

6. How about these image?
What do we perceive in a glance of a real-world scene? Fei-Fei, L., Iyer, A., Koch, C., & Perona, P.
(2007). Journal of Vision, 7(1):10,
https://authors.library.caltech.edu/11195/1/LIFjov07.pdf
* Image source google
* Image source google
* Image source google

7. How about this image?
Horse Racing
Game of POLO
Game of POLO
What do we perceive in a glance of a real-world scene? Fei-Fei, L., Iyer, A., Koch, C., & Perona, P.
(2007). Journal of Vision, 7(1):10,
https://authors.library.caltech.edu/11195/1/LIFjov07.pdf
* Image source google
* Image source google
* Image source google

8. Caution: How the Perceptual System Interprets the
Environment?
Hiyoshi-Taniguchi, K., Kawasaki, M., Yokota, T. et al. EEG Correlates of
Voice and Face Emotional Judgments in the Human Brain. Cogn
Comput 7, 11–19 (2015). https://doi.org/10.1007/s12559-013-9225-0
Hearing lips and seeing voices, Harry McGurk and John MacDonald
https://www.nature.com/articles/264746a0
McGurk Effect https://en.wikipedia.org/wiki/McGurk_effect

9. Image Annotation
Bounding boxes applied to identify
vehicle types and pedestrians.
Blue for pedestrians, green for
cars (taxi) and yellow for van.
pedestrians
Car
Van

10. Types of Image Annotation
Whole Image
Classification
Object
Detection
Image
Segmentation
Image
Annotation
The idea is to simply
identify which objects and
other properties exist in
an image.
One step further to whole
image classification, is to
find the position (bounding
boxes) of individual objects
The idea is to recognize and
understand what's in the image
at the pixel level. Every pixel in
an image belongs to at least one
class.

11. Types of Image Annotation
Classification Object Detection Segmentation
Image source: https://labelbox.com/

12. Machine learning paradigms
Supervised Un-Supervised
Machine
Learning
Using past information to
predict future e.g.
Classification, Regression
Learning patterns in un-
tagged data e.g.
clustering, Anomaly
detection

13. Machine learning paradigms
Supervised Un-Supervised
Machine
Learning
Using past information to
predict future e.g.
Classification, Regression
Learning patterns in un-
tagged data e.g.
clustering, Anomaly
detection

14. Supervised Machine Learning
In supervised machine learning a large amount of time is spent in
development of data-sets. This is a very critical and important step as the
models developed will be trained on this data.
One of the aspects in data set development is accurately defining the
variables both independent and outcome or dependent variable. This
becomes even more important when images are considered, and even
more complex when medical images are involved.
Understanding the data and labeling it accurately will lead to development
of a model that will be more accurate.
To achieve this, we need Annotation & Segmentation !!

15. Image Annotation – Gastroenterology (Aug 2020)
The images from this dataset are collected from real gastro and colonoscopy examinations at Bareum
Hospital in Norway. The dataset contains 110,079 images and 374 videos, and represents anatomical
landmarks as well as pathological and normal findings. The total number of images and video frames
together is around 1 million. (https://www.nature.com/articles/s41597-020-00622-y)

16. How is Annotation done ?
Images are labeled into 2 main
categories – Upper GI Tract and
Lower GI Tract.
Within Upper GI tract its
further separated into
Anatomical landmarks and
Pathological findings.
Within Lower GI tract its
separated into 4 categories -
Anatomical landmarks ,
Pathological findings,
Therapeutic interventions and
Quality of mucosal views.
Borgli, H., Thambawita, V., Smedsrud, P.H. et al. HyperKvasir, a comprehensive multi-class image and video
dataset for gastrointestinal endoscopy. Sci Data 7, 283 (2020). https://doi.org/10.1038/s41597-020-00622-y

17. Machine learning paradigms
Supervised Un-Supervised
Machine
Learning
Using past information to
predict future e.g.
Classification, Regression
Learning patterns in un-
tagged data e.g.
clustering, Anomaly
detection

18. Automatic Image Descriptions
Source : Deep Visual-Semantic Alignments for Generating
Image Descriptions" by Andrej Karpathy and Li Fei-Fei (CVPR
2015)
The goal in medicine is to automatically
segment the images and identify what
the image is telling us.
In the paper on the generating image
descriptions by Stanford in 2015, the
authors have developed the concept -
technology to identify variety of items in
the image.

19. Automatic - Image Segmentation
Dr. Coimbra and team in their paper talk about using segmentation
algorithms such as mean shift, normalized cuts, level-sets on the
automatic classification performance of gastric tissue into three classes:
cancerous, pre-cancerous and normal.
They had images from 2 modalities – Chromoendoscopy image and
Narrow-Band Image.
Coimbra M, Riaz F, Areia M, Baldaque Silva F, Dinis-Ribeiro M. Segmentation for classification of gastroenterology images. Annual
Int Conf IEEE Eng Med Biol Soc. 2010;2010:4744-7. doi: 10.1109/IEMBS.2010.5626622. PMID: 21096247.

20. Automatic Image Segmentation
Mean Shift
Normalized
Cuts
Level Sets
Automatic
Image
Segmentation
Coimbra M, Riaz F, Areia M, Baldaque Silva F, Dinis-Ribeiro M. Segmentation for classification of gastroenterology images. Annual
Int Conf IEEE Eng Med Biol Soc. 2010;2010:4744-7. doi: 10.1109/IEMBS.2010.5626622. PMID: 21096247.

21. Mean Shift Method
Mean shift is a well known method with applications in cluster analysis in computer vision &
image processing. The mean shift clustering algorithm is a practical application of the mode
finding.
Mean-Shift clustering algorithm steps −
Step 1 − First, start with the data points assigned to a
cluster of their own.
Step 2 − Next, this algorithm will compute the centroids.
Step 3 − In this step, location of new centroids will be
updated.
Step 4 − Now, the process will be iterated and moved to
the higher density region.
Step 5 − At last, it will be stopped once the centroids
reach at position from where it cannot move further.
[1] D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis.
IEEE Trans. Pattern Anal. Machine Intell., 24:603–619, 2002. Available at
http://www.caip.rutgers.edu/riul/research/papers/pdf/mnshft.pdf.
[2] https://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/TUZEL1/MeanShift.pdf

22. Mean Shift Method
Mean shift is a well known method with applications in cluster analysis in computer vision &
image processing. The mean shift clustering algorithm is a practical application of the mode
finding.
Mean-Shift clustering algorithm steps −
Step 1 − First, start with the data points assigned to a
cluster of their own.
Step 2 − Next, this algorithm will compute the centroids.
Step 3 − In this step, location of new centroids will be
updated.
Step 4 − Now, the process will be iterated and moved to
the higher density region.
Step 5 − At last, it will be stopped once the centroids
reach at position from where it cannot move further.
[1] D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis.
IEEE Trans. Pattern Anal. Machine Intell., 24:603–619, 2002. Available at
http://www.caip.rutgers.edu/riul/research/papers/pdf/mnshft.pdf.
[2] https://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/TUZEL1/MeanShift.pdf

23. Normalized Cuts
J. Shi, J. Malik, “Normalized cuts and Image Segmentation”,
IEEE Trans. PAMI, vol. 22, no. 8, pp. 888-905, Aug. 2000.
Normalized cut method treats
image segmentation as a graph
partitioning problem.
The normalized cut criterion
measures both the total
dissimilarity between the different
groups as well as the total similarity
within the groups. They use an
efficient computational technique
based on a generalized eigenvalue
problem to optimize this criterion.
(a) (b) (c) (d)
(a) A synthetic image showing three image patches forming a
junction. Image intensity varies from 0 to 1, and Gaussian
noise with alpha = 0:1 is added.
(b) , (c) and(d) shows the top three components of the partition.

24. Level Sets
[1] Osher S., Sethian J.A., Fronts Propagating with Curvature-Dependent Speed: Algorithms Based on Hamilton-Jacobi
Formulations (Journal of Computational Physics, 79(1), page 12-49, 1988).
[2] Kass M., Witkin A., Terzopoulos D., Snakes - Active Contour Models (International Journal of Computer Vision, 1(4), page 321-331,
1987)
[3] https://profs.etsmtl.ca/hlombaert/levelset/
[4] Lin Y, Zheng Q, Chen J, Cai Q, Feng Q. A novel adaptive level set segmentation method. Comput Math Methods Med.
2014;2014:914028. doi:10.1155/2014/914028
One of the advantages of level sets is that they
can provide good measurements of curvature.
The basic idea of the level set method is to
implicitly embed the moving contour into a higher
dimensional level set e.g. the contour around the
hand helps segment the hand portion.
This can help identifying the accurate portion of
an image be it a polyp or something else.

25. Advanced Clustering Techniques for image
clustering and segmentation
DBSCAN HDBSCANS
Topological
Data Analysis
(TDA)
Advanced
Clustering
Techniques

26. Thank You