Self-supervised learning performance

1. Why self-supervised
learning?
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty,
2019, arXIV
How well do self-supervised models transfer? 2021 , CPVR

2. Supervised
learning
• Supervised Pipeline
• 1000s of Hours (Human
annotators)
• Limits No of images

3. Self-supervised
learning
• Automatically generated pseudo-labels from structure of data
• More images More powerful discriminative framework
• After Training Transfer representations to new task
• Reduce need of large volume of data(For downstream tasks)

4. How well do self-supervised
models transfer?
• Significant improvement (Recent years)
• Approach supervised perform, (ImageNet benchmark
dataset Identical architecture)
Despite still being behind on ImageNet (Initial evidences),
Self-supervised models transfers better to new tasks.

5. How do you validate the statement?
• Study Transfer performance of self-supervised
pretrained models

6. Four
concerns
of Transfer
performance
How : Self-supervised transfer
vs Supervised transfer ?
Is there a best self-supervised
method overall ?
Has SSL overfit ? (ImageNet
Benchmark dataset)
Same information represented?
(Sup VS self-sup)

7. Datasets
Wide varieties of
datasets
Similar datasets such as
ImageNet , CIFAR-10
Quite different dataset :
Medical x-ray images

8. Answering the
concerns: Method
• 13 pre-trained self-supervised models +
supervised baseline.
• All models : RESNET50 architecture
(Pretrained on ImageNet) without labels.
• Model differences (Hyperparms such as
epochs , batch size, data augmentation )

9. Transfer Evaluation
(wide range of tasks)
• Linear Evaluation, fine tuning: MSR
• Small/Large domain shift: FSR
• A frozen and Finetuned backbone: OD
• Dense Prediction tasks: SNE | SS
• Surface Normal prediction: Predict the surface orientation
of the object inside a scene

10. Results: Self-supervised methods are approaching supervised performance on ImageNet

11. Transfer performance
• Highly correlated with Many-shot recognition
• Increasingly less correlated with few-shot R. | OD | Dense Predictions

12. 1. How does self-
supervised transfer
compare to
supervised transfer ?
Compare self-supervised models to the supervised base line
represented by the green star.

13. 2. Is there Is there a
best self supervised
method overall ?
Across all but one
setting, self-
supervised models
outperform
supervision, showing
their superior
transferability.
But there is no single
model that dominates
all setting, showing
the community still a
way to go to reach
truly general feature.

14. 3. Has SSL overfit to
imageNet as a
benchmark?
• ImageNet performance is highly correlated with many-shot recognition

15. Correlation is weaker in other cases
In order to achieve more generalizable representations in the future,
the community needs to consider the wider benchmarks for
evaluation.

16. 4. Do self-supervised and supervised features represent the
same information?
• Model features analysis
Analysis
• Image reconstruction from feature vector (Supervised better)
• Why? Self-supervised (lose color information due to heavy data Augm.)
Reconstruction
• SSM: Have wide attentive focus (Attention)
SSM
• Sup: High spatial focus (Location)
Sup

17. Conclusion
Self-supervision tend to produce
better calibrated classifiers for
downstream recognition tasks.