Provides a basic introduction to Natural Language Processing (NLP), its properties, and some common techniques such as stemming, tokenization, bag-of-words, stripping, and n-grams

1. an introduction to
Natural Language Processing
(NLP)

2. Who am I?
• David Rostcheck
• I’m a consulting data
scientist
• Follow my articles on
LinkedIn

3. Who am I?
• Jennifer Barry
• I do research and project
management

4. what is
Natural Language
Processing
and why do we need it?

5. people communicate in
language

6. so our applications are much more
useful
when they understand language

7. and can help us.

8. but the problem is…
computers don’t really
understand language

9. What can we do?

10. solution 1: Deep Learning
idea: teach computers
language (like we teach it to
people)
(these are model-free methods)

11. solution 2: Natural Language
Processing (NLP) via algorithms
idea: fake it! find algorithms
that give approximately correct
results most of the time
(these are heuristics)

12. Today, we learn about faking
it ;)

13. properties of NLP
algorithms

14. They are language-dependent
(you need a new algorithm for
each language)
algorithm-one
algoritmo-dos
Algorithmus-drei

15. They are
----------   fast!


16. easy to implement
… at first, anyway...


17. They are computationally efficient,
so they
scale
very
well!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|

18. they produce results that are
mostly correct
most of the time

19. buuuuut....

20. NLP relies on
statistics so it is
only statistically
true

21. Without real understanding,
accuracy is limited
and cannot easily be improved
NLP: looks fine to me!

22. NLP
techniques

23. stemming
learn-ed
learn-er  learn
learn-ing
removes suffixes –
discards nuance, emphasizes
root concept

24. tokenization
Turns text into a stream of
sentences or words or
syllables

25. This is simple in some languages
…but not so simple in others

26. bag
of
words
transforms a document
into a collection of
unordered words


27. stripping
removes low-information signals
(common words, punctuation)

28. n-gram: a sequence of n
tokens taken together


29. tf/idf (“term frequency over
inverse document frequency”)
Finds unusual words that appear
often in this document
(used in search engines)

30. examples

32. Flesch-Kincaid Grade Level formula:

33. Topic extraction

34. Personality Analysis
(using IBM Watson’s
Personality Insights
service)

35. social network analysis from text

36. Sentiment analysis (using Turi GraphLab Create)

37. Questions?
Contact: drostcheck@leopardllc.com, twitter: @davidrostcheck
Articles: http://linkedin.com/in/davidrostcheck