1. Eye-Deep
Detecting Diabetes
with Convolutional Neural Networks
team o_O
Mathis Antony
sveitser@gmail.com
Stephan Brüggemann
https://github.com/sveitser/kaggle_diabetic
https://www.kaggle.com/c/diabetic-retinopathy-detection
2. Intro
● Supervised Learning
○ Training set (features + labels) and test set (only features)
○ Training
■ learn relationship between features and labels (on training set)
○ Testing
■ predict labels from test data and measure performance
● Deep learning
○ Deep → many layers
○ Concepts not “new”
■ More data (internet)
■ More computational power (GPUs)
■ advancements in the field
■ great open source software
4. ReLU: max(x, 0)
Leaky ReLU: max(x/100, x)
x (sum of inputs)
y (output)
Rectified Linear Unit: ReLU
inputs
output
1. sum inputs
2. activation function
Artificial Neurons
6. Gradient Descent
1. Compute derivative of loss
function with respect to weights
2. Update weights
η: learning rate
7. Training
1. Initialize weights randomly
2. Until happy, repeat
a. Forward pass through network (make prediction)
b. Calculate error
c. Backward propagation of errors (backprop)
d. Update weights
● Done in mini batches
● One batch in memory at a time if necessary
● Libraries provide almost everything
13. Problem
● input data
○ high resolution color retinal images
○ training set: 35126 images
○ test set: 53576 images
● target
○ stage of diabetic retinopathy
■ 0 No DR
■ 1 Mild
■ 2 Moderate
■ 3 Severe
■ 4 Proliferative DR
● Highly unbalanced dataset
14. Metric
● Quadratic (Weighted) Cohen’s kappa (κ)
○ Agreement between rating of two parties
■ 0 agreement by chance
■ 0 - 0.2 poor
■ ...
■ 0.8 - 1.0 very good
■ 1 total agreement
● “Weighted” → Ordinal classification problem
● “Less penalty for classifying a 0 as a 1 than as a 2”
● Our “solution”:
○ Regression with mean squared error
○ thresholding at [0.5, 1.5, 2.5, 3.5]
16. What are we looking for?
Saiprasad Ravishankar, Arpit Jain, Anurag Mittal
IEEE Conf. on Computer Vision Pattern Recognition (CVPR) 2009 http://ieeexplore.ieee.
org/xpl/articleDetails.jsp?arnumber=5206763
17. Preprocessing
● Simple heuristics to isolate and crop foreground
● Resize to 512 pixel squares
● Standardize each channel (RGB) to have zero mean and unit variance
● That’s it!
● But, training large networks requires a lot of data.
18. Augmentation
● Problem: “Small” Dataset
● Artificially increase size of dataset
○ translation
○ rotation
■ can become the bottleneck for large images
○ flipping
○ shearing
○ stretching
○ color augmentation*
19. Layer Types
● ☑Convolutional Layer (find features)
● ☑Max Pooling Layer (find features + reduce size)
● ☑Fully Connected Layer (prediction from features)
● Dropout Layer (model averaging, against overfitting)
○ Zero half the neurons
○ Network becomes different for each mini batch
5 2 8 1 9 2 2 5
5 0 0 1 9 0 2 0
3 5
5
2 1
2
● Maxout Layer
○ Take maximum value over 2 Neurons
20. ● Input image to many tiny “images” (feature maps) a few pixels wide.
● Extract features on the way through the network.
● Layers with stride 2 halve width and height of feature maps.
● Handy “Units”
○ 2 - 4 convolutional layers with small filters (2 x 2 to 5 x 5)
○ followed by max pooling layer with stride 2 and pool size 3
● Add ReLUs (or similar)
● 1 or 2 fully connected layers with dropout at the end
● Weight decay for convolutional layers.
● “If it doesn’t overfit, you should probably it bigger”.
● In competition:
○ Tiny features → larger input images: 64 → 128 → 256 → 512 (→ 768)
○ More and more convolution and pooling layers
Network Architecture
22. Training
● Deep networks (many layers) are sometimes hard to train.
● Initialization strategy is very important.
● learning rate:
a. Find largest value for loss still converges
b. When loss doesn’t decrease, decrease learning rate by factor of 5 or 10
● Use “Adam” optimizer or “Nesterov Momentum”.
● In competition
a. Dynamic resampling to deal with class imbalance.
b. Train smaller network and use learned weights to initialize bigger
networks.
c. 200 - 250 epochs
d. ~ 2 days to train one network
23.
24. What does it “see”?
input (stage 1) 5x5 pixel occluded prediction overlay
Visualizing and Understanding Convolutional Networks
Matthew D Zeiler, Rob Fergus
http://arxiv.org/abs/1311.2901
25. What does it “see”?
input (stage 1)
Visualizing and Understanding Convolutional Networks
Matthew D Zeiler, Rob Fergus
http://arxiv.org/abs/1311.2901
26. Feature Extraction
● Output of any layer can be used as features
● Could use pretrained networks for feature extraction (unless kaggle)
output of last pooling layer → features
● Original score: κ 0.79 (~ rank 13 on final kaggle leaderboard)
● Features of last pooling layer:
○ Blend Network
■ features → FC 32 → maxout → FC 32 → maxout → output
○ κ ~ 0.80 (~ rank 12)
○ fully connected layers in our convolutional network not well trained
27. Test Time Averaging (TTA)
● From winners of kaggle plankton competition early 2015:
https://github.com/benanne/kaggle-ndsb
● Average output of last pooling layer over multiple augmentations for
each eye
● Use mean and standard deviation of each feature
● Same blend network
○ features → FC 32 → maxout → FC 32 → maxout → output
● with TTA mean κ ~ 0.81 (~ rank 11)
● with TTA mean + standard deviation κ ~ 0.815 (~ rank 10)
28. ● use TTA features from left and right eye and blend
○ [features of this eye, features of patient’s other eye, left eye indicator]
■ left: [left eye features, right eye features, 1] → left eye label
■ right: [right eye features, left eye features, 0] → right eye label
○ mean, standard deviation, indicator: 8193 features
● Train Blend Network: κ → ~ 0.84 (~ rank 2 - 3)
“Per Patient” Blend
● both eyes for each patient in dataset
● some images look very “different”
● correlation between labels for left and
right eye is very high: ρ ~ 0.85
right eye label
for patients
with left eye
label 3
29. Final Result
● Ensembling
○ averaged results from 2 similar network architectures and 3 sets of
weights each: κ → ~ 0.845
HK Electric wins too