A light intro to natural language processing on social media, presented as an invited talk at the University of Sheffield Engineering Symposium 2014 in the AI session. As well as an introduction to the area, this presentation covers powerful real-world applications of social media, and touches on the work we do in the Sheffield NLP group. Video cast: https://www.youtube.com/watch?v=QUbRmUinhHw&feature=youtu.be

1. Leveraging the Power of Social Media
Leon Derczynski
Natural Language Processing Group
Department of Computer Science
Faculty of Engineering
University of Sheffield

2. work in the field of “computational linguistics”
focus on turning text
into “understanding”
and “decision support”

3. the “AI effect”
Pamela McCorduck
artificial intelligence is less impressive when
we know how it works

4. the “AI effect”
Pamela McCorduck
artificial intelligence is less impressive when
we know how it works
..so this talk won't have deep technical detail

5. language

7. (note huge evolutionary advancement)

13. ??

14. social media

15. social media – a poster child for big data

17. big data:
promises new insights
is (was) a cool buzzword
causes headaches
what is it?

18. V: velocity
twitter:
255 000 000 users / month
Facebook:
1 280 000 000 users / month

19. VV: volume
reddit:
34 000 000 posts / month
twitter:
650 000 000 messages / month

20. VVV: variety

26. there are many online social networks
we need one of these

27. there are many online social networks
we need one of these

28. artificial intelligence

29. “Human knowledge is expressed in language. So
computational linguistics is very important.”
- Mark Steedman

30. Start: sequence of bytes
[natural
language
processing
goes
here]
End: actionable knowledge

31. why bother programming at all?

32. why bother programming at all?
… let the computer program itself!

33. machine learning:
make decisions about tasks based on things
you've seen before
a little bit like human learning

34. give text and examples of what we want done
machine learns to from these examples

37. understanding language

38. social media text is surprisingly formal

39. they see me rollin
- a typo?

40. they see me rollin
they hatin
- perhaps not. G-dropping mapped from speech

41. they see me rollin
they hatin
patrollin
- incidentally,
this linguistic phenomenon is a good predictor of
education level

42. they see me rollin
they hatin
patrollin
tryna catch me ridin dirty
- a new style!
flawless; not a single mistake

43. omb x
- surely they mean “omg”?

44. omb ✔
- the keys are like, right next to each other
X
really? this guy?

45.  Shall we go out for dinner this evening?

 Ey yo wen u gon let me tap dat

46. spelling ability distribution in net slang users

47. with spelling ability distribution in non-slang users

48. Do you feel luccy, punk?

49. Do you feel luccy, punk?

50. challenge 1: what language is this anyway
je bent Jacques cousteau niet die een nieuwe soort heeft
ontdekt, het is duidelijk, ze bedekken hun gezicht. Get over it
RT @TomPIngram: VIVA LAS VEGAS 16 - NEWS
#constantcontact http://t.co/VrFzZaa7

51. challenge 2: pls type better
I wonde rif Tsubasa is okay..
- misplaced space = two new words
no homwork tonight.. suprising??
- maybe there should be!

52. challenge 3: finding names
derek
x is a person
miles
x might be a person
Marie Claire
x should not be a person
Exodus Porter
x probably an OK person, but actually a beer

53. challenge 3: finding names
Spicy Pickle Jr.
x apparently actually a person

54. challenge 3: finding names
Spicy Pickle Jr.
x apparently actually a person
???

55. old news

56. social media defends against earthquakes
2010

57. Japanese and US quake response times:
down from ~20s to ~17.5s

58. social media predicts epidemics
2012
exhibit a: one dead crow

59. social media mentions of dead crows predict WNV in humans
''There's a dead
crow in my garden''

60. social media predicts you getting flu
2012

61. @mari: i think im sick ugh..
great potential for misuse :)

62. this november:
social media dispatches fire engines
2014

64. trust

65. if hospitals and fire stations act based on tweets,
wrong information is extra-harmful

66. rumours
speculation
misinformation
disinformation
who can you trust online?
Imagine a lie detector for politicians / Fox News

67. responsibility

68. 1. Collect tweets
2. ????
3. Profit!

69. how long do we keep them for?
- “15 years is OK, right?” - NSA
what do we store and process?
- “just metadata, it's harmless” - GCHQ

70. (from Kurt Opshal's slides at the Chaos Communication Congress, photo by Marion Marschalek)

71. bias

72. news
style
social
media

73. most of our language AI was trained on news text
the bias is:
- middle class
- white
-working age
- educated
- male
- 1980s/1990s
- from the US
- journalist
- following AP guidelines

74. your phone rewards you if you talk and write like
(ok.. sort of)

75. your phone rewards you if you talk and write like
(ok.. sort of)
.. and punishes you when you don't.
(not cool!)

76. twitter bias is different
- not German or Nordic
- are young(ish)
lower requirements
- you can publish even if you're not a journalist
- still operates beyond the 1990s
some new requirements
- you do need access to the internet...
- ...and twitter ( 对不起，中国人 )

77. the big picture

78. we're racing ahead and improving life quality
there is immense value in “trivia”
understanding social media lets us
help people better

79. understanding social media lets us
help people better
Thank you!
Leon Derczynski