1. Automated Diagnosis and Prognosis of Skin Cancer Using Dermoscopy
Images with Transfer Learning Methods
Presented by
Mehebuba Nasrin Eity
ID: M200305522
Supervised by
Prof. Dr. Uzzal Kumar Acharjee
Chairman
Dept. of CSE, JnU
Dept. of Computer Science and Engineering
Jagannath University
Presentation on Degree of M.Sc in CSE

2. Contents
● Introduction
● Motivations
● Problem Statements
● Background Study
● Research Contributions
● Emerging key technologies, contributions and research gaps.
● System Model & Methodology
● Experimental Research and Analysis
● Conclusion
● References
2

3. ● Skin cancer is growing exponentially nowadays because of sunlight limitations and
sunburn.
● Cancer cells are mutated cells that often grow more than normal; this abnormal growth
is called a tumor.
● There are two types of tumors: benign (stay in one part), and malignant (spread to other
parts of the body).
● Malignant tumors are cancerous because they spread to other parts of the body. On the
other hand, benign tumors stay in one part of the body.
● Benign tumors have distinct, smooth, regular borders, while malignant tumors grow
faster than benign tumors without border restriction. 3
Introduction

4. The motivations behind this work come out as a result of the research on the existing Skin
Cancer detection using digital “Dermoscopy Images with Transfer Learning”. It brings us to
the following conclusion.
● Data Limitation
● Lack of Early Detection Method
● Privacy on Public Health
● To build an automated system
● Environment and API support
● Depend on pre trained computer vision models
● Low accuracy gap
● High Training delay
● High Evaluation delay
4
Motivations

5. After research on published work, we realized a higher need using an automatic
system for dermoscopy image with transfer learning for skin cancer detection.
5
Problem Statement

6. In this section, I discuss the background study of CNN, EfficientNetV2B0, ViT
related to this research.
● Convolutional Neural Networks (CNN)
● EfficientNet V2 B0
● Vision Transformer (ViT)
6
Background Study

7. 7
Background Study Cont…
● Convolutional Neural Networks (CNN)
Figure-1: CNN Architecture

8. 8
Background Study Cont…
Figure-2: EfficientNet V2 B0 Architecture
● EfficientNet V2 B0

9. 9
Background Study Cont…
Figure-3: ViT Architecture
● Vision Transformer (ViT)

10. Research contributions of this work includes:
● To design an efficient computer vision models
● To provide better classification of benign and malignant tumors.
● To mitigate against existing ISIC Archive Dataset uses the evaluate machine
learning model
● To preprocess dataset with TensorFlow's tf.data API
● To model a hierarchical based on CNN, EfficientNet V2 B0 and Vision
Transformer
● Finally, evaluate the performance of the model to solve the problem should
have a high accuracy score.
10
Research Contributions

11. 11
Emerging key technologies, contributions and research gaps.
No
.
Authors Key Tech… Contributions and Research Gap
1 Araújo et al. (2022) CNN
need to compare effectiveness of various
model
2 Karri et al. (2023) dataset decentralized patient data
3 Daneshjou et al. (2022) ML AI supported decision making
4 Hu et al. (2023) metrics evaluate the precision of skin cancer prognosis
5 Jasil et al. (2021) VGG16, VGG19 Multi-Modal Integration
6 Mijwil et al. (2021) InceptionV3 image quality and quantity
7 SM et al. (2023) EfficientNet ethical caution of using AI

12. 12
Emerging key technologies, contributions and research gaps.
No
.
Authors Key Tech… Contributions and Research Gap
9 Balaha (2023) CNN Evaluate the system with available dataset
10 Khan (2021) CNN need to enhance the performance
11 Afza (2022) ResNet50 improving skin lesion segmentation
12 Shorfuzzaman (2022) EfficientNetB0
need automated methods for detecting
melanoma
13
Sevli (2021)
CNN
need for accurate and efficient automated
classification
14 Keerthana (2023) CNN aiming to reduce inter-operator variability
standardization and curation to facilitate

13. System Model & Methodology
13

14. 14
# Install necessary packages using pip
!pip install
pandas # Required for efficient data manipulation and analysis.
numpy # Essential for numerical operations and working with arrays.
matplotlib # Necessary for creating various types of plots and charts.
seaborn # Enhances data visualization with attractive statistical graphics.
tensorflow # Fundamental for building and training machine learning models.
tensorflow-addons # Extends TensorFlow with additional functionalities.
tensorflow_hub # Provides access to pre-trained machine learning models.
scikit-learn # Comprehensive library for machine learning tasks.
scikit-plot # Generates visualizations for model evaluation.
Vit-keras # Likely used for Vision Transformers in Keras for computer vision.
Python environment

15. 15
About Dataset:
This dataset contains a balanced dataset of images of benign skin image and
malignant skin image. The data consists of two folders and pictures size (224x244).
Dataset taken from ISIC Archive.
Dataset
Malignant Benign Total
Train 1197 1440 2637
Test 300 360 660
3297

16. 16
To achieve a faster training time we will splits train Image into train and validation.
Data augmentation layer will have to be constructed manually but to handle the
loading and passing of the image will use TensorFlow's tf.data API
Preprocessing

17. 17
Model: "cnn_sequential_model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 222, 222, 16) 448
_________________________________________________________________
conv2d_1 (Conv2D) (None, 220, 220, 16) 2320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 110, 110, 16) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 108, 108, 8) 1160
_______________________________________________________________
conv2d_3 (Conv2D) (None, 106, 106, 8) 584
_______________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 53, 53, 8) 0
_______________________________________________________________
flatten (Flatten) (None, 22472) 0
_______________________________________________________________
dropout (Dropout) (None, 22472) 0
_______________________________________________________________
dense (Dense) (None, 128) 2876544
_______________________________________________________________
dense_1 (Dense) (None, 2) 258
=================================================================
Total params: 2,881,314
Trainable params: 2,881,314
Non-trainable params: 0
Model: "efficientnet_v2_sequential_model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
efficientnet_v2_b0 (KerasLay (None, 1280) 5919312
_________________________________________________________________
dropout_1 (Dropout) (None, 1280) 0
_________________________________________________________________
dense_2 (Dense) (None, 128) 163968
_________________________________________________________________
dense_3 (Dense) (None, 2) 258
=================================================================
Total params: 6,083,538
Trainable params: 164,226
Non-trainable params: 5,919,312
Model: "vit_b16_sequential_model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
vit-b16 (Functional) (None, 768) 85798656
_________________________________________________________________
dropout_2 (Dropout) (None, 768) 0
_________________________________________________________________
dense_4 (Dense) (None, 128) 98432
_________________________________________________________________
dense_5 (Dense) (None, 2) 258
=================================================================
Total params: 85,897,346
Trainable params: 98,690
Non-trainable params: 85,798,656
Trainable Parameter

18. 18
Experimental Research and Analysis

19. 19
The CNN model was capable of converging to a test loss similar to the observed
with the training and validation sets. We observe a stable convergence to a lower
loss. No sign of overfitting.
Results for Custom CNN
Figure : Evaluation matrix of the proposed system

20. 20
Some overfitting have occurred during the training loss converges,, but the
validation loss is unstable for the first few epochs and converges to a higher loss
than that observed for the training set.
Results for Custom EfficientNet V2 B0
Figure : Evaluation matrics of the proposed system

21. 21
Some slight overfitting might have occured during the last few training epochs for
the Vision Transformer model since the training and validation losses converges at
a stable rate, but towards the last few epochs there is a gap which forms between
the losses.
Results for Custom VIT B16
Figure : Evaluation matrics of the proposed system

22. 22
CNN Confusion Matrix
=============================================
CNN Performance Metrics:
=============================================
accuracy_score: 0.8318
_____________________________________________
precision_score: 0.8372
_____________________________________________
recall_score: 0.8318
_____________________________________________
f1_score: 0.8322
_____________________________________________
matthews_corrcoef: 0.6679
=============================================

23. 23
EfficientNet V2 B0 Confusion Matrix
=============================================
Performance Metrics:
=============================================
accuracy_score: 0.8833
_____________________________________________
precision_score: 0.8836
_____________________________________________
recall_score: 0.8833
_____________________________________________
f1_score: 0.8834
_____________________________________________
matthews_corrcoef: 0.7651
=============================================

24. 24
VIT B16 Confusion Matrix
=============================================
Performance Metrics:
=============================================
accuracy_score: 0.8621
_____________________________________________
precision_score: 0.8642
_____________________________________________
recall_score: 0.8621
_____________________________________________
f1_score: 0.8612
_____________________________________________
matthews_corrcoef: 0.7228
=============================================

25. 25
CNN Classification Report
precision recall f1-score support
malignant 0.78 0.87 0.83 300
benign 0.88 0.80 0.84 360
accuracy 0.83 660
macro avg 0.83 0.84 0.83 660
weighted avg 0.84 0.83 0.83 660

26. 26
EfficientNet V2 Classification Report
precision recall f1-score support
malignant 0.87 0.88 0.87 300
benign 0.90 0.89 0.89 360
accuracy 0.88 660
macro avg 0.88 0.88 0.88 660
weighted avg 0.88 0.88 0.88 660

27. 27
ViT-b16 Classification Report
precision recall f1-score support
malignant 0.89 0.80 0.84 300
benign 0.84 0.92 0.88 360
accuracy 0.86 660
macro avg 0.87 0.86 0.86 660
weighted avg 0.86 0.86 0.86 660

28. 28
Performance DataFrame
Accuracy score Precision score Recall score F1 score
Matthews
corrcoef
Model CNN 0.831818 0.837174 0.831818 0.832156 0.667856
Model Efficientnet v2 0.883333 0.883583 0.883333 0.883408 0.765138
Model Vit b16 0.862121 0.864211 0.862121 0.861211 0.722836

29. 29
Inference rate
Model: cnn_sequential_model
====================================================
Inference Time: 2.411501s ± 0.614918s
Inference Rate: 0.003654s/sample ± 0.000932s/sample
====================================================
Model: efficientnet_v2_sequential_model
====================================================
Inference Time: 3.244478s ± 1.279871s
Inference Rate: 0.004916s/sample ± 0.001939s/sample
====================================================
Model: vit_b16_sequential_model
====================================================
Inference Time: 9.293184s ± 2.437735s
Inference Rate: 0.014081s/sample ± 0.003694s/sample

30. 30
Inference Rate vs. Matthews Correlation

31. 31
The classification of Skin Cancer was explored with CNN, along with Transfer
Learning models such as EffiecientNet V2 and Vision Transformer. The use of
pretrained models were used to create models which were shown to outperform
the baseline CNN model.
We observe that EfficientNet V2 B0 is the best choice as it outperforms both the
baseline CNN and Vision Transformer models on the trade-off between inference
time and performance as it achieves the highest MCC while maintaining a low
inference rate.
However, Tumor is benign or malignant it should be noted that a model trained to
solve this problem should have a high accuracy score and MCC as misclassification
may bare a fatal outcome. If it is to be used in production as it will influence the
decisions made by health professionals who deal with skin cancer patients.
Conclusion

32. 32
1. Araújo, Rafael Luz, Flávio HD de Araújo, and Romuere RV E. Silva. "Automatic segmentation of melanoma skin cancer
using transfer learning and fine-tuning." Multimedia Systems 28.4 (2022): 1239-1250.
2. Karri, Meghana, Chandra Sekhara Rao Annavarapu, and U. Rajendra Acharya. "Skin lesion segmentation using two-phase
cross-domain transfer learning framework." Computer Methods and Programs in Biomedicine 231 (2023): 107408.
3. Daneshjou, Roxana, et al. "Checklist for evaluation of image-based artificial intelligence reports in dermatology: CLEAR
derm consensus guidelines from the international skin imaging collaboration artificial intelligence working group." JAMA
dermatology 158.1 (2022): 90-96.
4. Hu, Mingzhe, Yuheng Li, and Xiaofeng Yang. "Skinsam: Empowering skin cancer segmentation with segment anything
model." arXiv preprint arXiv:2304.13973 (2023).
5. Jasil, SP Godlin, and V. Ulagamuthalvi. "Deep learning architecture using transfer learning for classification of skin
lesions." Journal of Ambient Intelligence and Humanized Computing (2021): 1-8.
6. Mijwil, Maad M. "Skin cancer disease images classification using deep learning solutions." Multimedia Tools and
Applications 80.17 (2021): 26255-26271.
7. SM, Jaisakthi, et al. "Classification of skin cancer from dermoscopic images using deep neural network architectures."
Multimedia Tools and Applications 82.10 (2023): 15763-15778.
References

33. 33
8. Jobson, Dale, Victoria Mar, and Ian Freckelton. "Legal and ethical considerations of artificial intelligence in skin cancer
diagnosis." Australasian Journal of Dermatology 63.1 (2022): e1-e5.
9. Balaha, H.M., Hassan, A.ES. Skin cancer diagnosis based on deep transfer learning and sparrow search algorithm. Neural
Comput & Applic 35, 815–853 (2023).
10. Khan, Muhammad Attique, et al. "Pixels to classes: intelligent learning framework for multiclass skin lesion localization
and classification." Computers & Electrical Engineering 90 (2021): 106956.
11. Afza, Farhat, et al. "A hierarchical three-step superpixels and deep learning framework for skin lesion classification."
Methods 202 (2022): 88-102.
12. Shorfuzzaman, Mohammad. "An explainable stacked ensemble of deep learning models for improved melanoma skin
cancer detection." Multimedia Systems 28.4 (2022): 1309-1323.
13. Sevli, Onur. "A deep convolutional neural network-based pigmented skin lesion classification application and experts
evaluation." Neural Computing and Applications 33.18 (2021): 12039-12050.
14. Keerthana, Duggani, et al. "Hybrid convolutional neural networks with SVM classifier for classification of skin cancer."
Biomedical Engineering Advances 5 (2023): 100069.
15. Goyal, Manu, et al. "Artificial intelligence-based image classification methods for diagnosis of skin cancer: Challenges and
References

34. THANK YOU
34