Call for collaboration in applied transfer learning for text classification tasks https://www.kaggle.com/kashnitsky/exploring-transfer-learning-for-nlp

1. Benchmarking approaches to
transfer learning in NLP
Yury Kashnitskiy, Gianluigi Bardelloni

2. Benchmarking approaches to
transfer learning in NLP
General problem:
1. Scarce well-labeled data in NLP tasks
2. Loads of raw texts available
3. How to utilize raw data to improve
performance in supervised NLP tasks?
Simpler: how to use all these bulks of unlabeled texts?
2

3. Business problem:
1. NLP tasks typically require a lot of labeled data
2. Labeling data is expensive
3. A disciplined approach to minimizing the needed training
size is highly desired
Benchmarking approaches to
transfer learning in NLP
3

4. Background: transfer learning in Computer Vision
Benchmarking approaches to
transfer learning in NLP
Pretraining
Fine-tuning
4

5. General idea:
1. A neural net is trained on raw
data to predict a word given its
context
2. Meanwhile it learns a vector
(“embedding”) for each word
3. Embeddings can be
transferred and used in
supervised learning tasks
(ex: part-of-speech-tagging)
This approach leads to SotA
results in many NLP tasks
Benchmarking approaches to
transfer learning in NLP
5

6. Benchmarking approaches to
transfer learning in NLP
ULMFiT:
1. Take a pretrained
language Model (with
ex. Wikipedia)
2. Fine-tune language
model on your domain
(ex. chats with
customers)
3. Fine-tune classifier
6

7. Benchmarking approaches to
transfer learning in NLP
Task 1. Amazon product reviews classification (English)
Validation accuracy:
Logistic Regression + Tf-Idf: 72.5%
ULMFiT: 79.5%
7

8. Benchmarking approaches to
transfer learning in NLP
Interpreting Logistic Regression with eli5
8

9. Examples of generated text (pure LSTM, PyTorch examples):
"This is a product that gets great secrets to the store's difficulty,
though it's compact and well made. Ordered one in the
mornings, for price. It's good though. Lasts 6 months, really need
to ask Amazon.com) as soon as you have the off!"
"I have been using Panasonic for a year and my mom was on
them to replace my Quadra Action and been pleased with the
gel.”
9
Benchmarking approaches to
transfer learning in NLP

10. Benchmarking approaches to
transfer learning in NLP
Task 2. Classifying chats with customers (Dutch)
Validation accuracy:
Logistic Regression + Tf-Idf: 73.5%
ULMFiT: 70.2%
Logit + ELMo: 66%
10

11. Benchmarking approaches to
transfer learning in NLP
Things to try:
1. Training ULMFiT
models with Dutch
texts
2. Fine-tuning BERT
classifier
3. Trying other
models: GPT-2,
OpenAI transformers
etc.
The goal is to develop best practices for
transfer learning in Dutch classification tasks
11

12. Benchmarking approaches to
transfer learning in NLP
What we expect from the collaboration:
1. Trying different transfer learning approaches
2. Both public and private data (English and Dutch)
3. Sharing code & ideas
In 3 months:
share preliminary results – code, models, guides
12

13. Proposed tasks
1. Benchmarking different approaches on several datasets to
see what works best. The main focus is on BERT and ULMFiT,
however, no limitations
Kaggle Dataset:
- Amazon healthcare reviews (English)
- Amazon pet products reviews (English), Kaggle comp.
- Clickbait news detection (English), Kaggle comp.
- Book reviews sentiment prediction (Dutch)
13
Benchmarking approaches to
transfer learning in NLP

14. 2. Training own models for ULMFiT (Dutch)
3. Exploring Byte Pair Encoding as preprocessing for ULMFiT
4. Exploring preprocessing steps to improve BERT classifier
5. Dealing with typos and noise in text in case of BERT
6. Fine-tuning BERT Language models, exploring it's effect on
classification
14
Proposed tasks
Benchmarking approaches to
transfer learning in NLP

15. Trying other approaches:
1. Huggingface transfer learning tutorial + code
2. Fine-tuning classification head over LSTMs (pure Python)
3. GPT-2 transformers
4. Other OpenAI transformers
Other tasks:
1. Investigating text augmentations and their effect on
classification accuracy
2. Active learning in NLP
3. Hierarchical text classification
4. Few-shot learning (ex. Unsupervised Data Augmentation)
15
Benchmarking approaches to
transfer learning in NLP

16. Yury Kashnitskiy, Gianluigi Bardelloni
yury.kashnitskiy@kpn.com
gianluigi.bardelloni@kpn.com
16