So einfach geht modernes Roaming fuer Notes und Nomad.pdf
Understanding Big Data with Examples
1. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
#bigdata in Communication Science
Some examples from research
by me and my students
Damian Trilling
d.c.trilling@uva.nl
@damian0604
www.damiantrilling.net
Afdeling Communicatiewetenschap
Universiteit van Amsterdam
October 2013
#bigdata
Damian Trilling
2. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
1 What’s big data?
2 Some examples
Rare events
Tone in tweets
Counting words and n-grams
Network analysis
3 Problems
4 A glimpse in the kitchen
5 Questions?
#bigdata
Damian Trilling
3. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
What’s big data?
#bigdata
Damian Trilling
4. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
What’s big data?
No definition, but . . .
• Existing data
• Too big to code manually
• Sometimes also too big to handle with normal tools
• New research questions
• Call to revisit the relationship between theory and empirical
research
#bigdata
Damian Trilling
5. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
What’s big data?
Some sources
• Social Network Sites
• RSS-feeds
• Databases
• Scraping text from the web
• ...
#bigdata
Damian Trilling
6. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
It’s out there!
You only have to collect it.
#bigdata
Damian Trilling
7. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Some examples
#bigdata
Damian Trilling
8. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Rare events
A recent master thesis
Rare events
#bigdata
Damian Trilling
9. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Rare events
A recent master thesis
Rare events
Imagine you want to analyze some very rare content.
#bigdata
Damian Trilling
10. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Rare events
A recent master thesis
Rare events
Imagine you want to analyze some very rare content.
Normal sampling won’t work, that’s for sure.
#bigdata
Damian Trilling
11. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Rare events
So you’d better collect everything first
Getting all news coverage from Dutch news sites
Pöll, B. (2013). Social media: new sources, new profession? A content analysis of the use of social media as a
source for journalists in online news articles. Master Thesis, Universiteit van Amsterdam.
#bigdata
Damian Trilling
12. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Rare events
So you’d better collect everything first
Getting all news coverage from Dutch news sites
We collected all articles from nine news sites during a period of
two months, resulting in a database with 74.000 articles.
Pöll, B. (2013). Social media: new sources, new profession? A content analysis of the use of social media as a
source for journalists in online news articles. Master Thesis, Universiteit van Amsterdam.
#bigdata
Damian Trilling
13. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Rare events
So you’d better collect everything first
Getting all news coverage from Dutch news sites
We collected all articles from nine news sites during a period of
two months, resulting in a database with 74.000 articles.
In a second step, we filtered those articles containing specific
keywords. Those 292 articles where then manually coded.
Pöll, B. (2013). Social media: new sources, new profession? A content analysis of the use of social media as a
source for journalists in online news articles. Master Thesis, Universiteit van Amsterdam.
#bigdata
Damian Trilling
14. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Rare events
#bigdata
Damian Trilling
15. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Rare events
It’s just one line of code!
url.txt
http://www.gmx.at/themen/wissen/mensch/108g5xi-baeuerlich-schiefe-zaehne
http://www.gmx.at/themen/unterhaltung/klatsch-tratsch/408g740-fuermannbittet-um-verzeihung
http://www.gmx.at/themen/nachrichten/aufruhr-arabien/268g70u-regierungwill-zuruecktreten
http://www.gmx.at/themen/nachrichten/panorama/828g54y-neues-zur-klagegegen-republik
http://www.gmx.at/themen/nachrichten/panorama/968g72s-millionstrafewegen-oelpest
http://www.gmx.at/themen/unterhaltung/klatsch-tratsch/368g6yc-keinbabybauch-nur-fast-food
...
...
...
#bigdata
wget-commando
wget -i urls.txt
Damian Trilling
16. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Tone in tweets
A recent bachelor thesis
Tone in tweets
#bigdata
Damian Trilling
17. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Tone in tweets
A recent bachelor thesis
Tone in tweets
Imagine you want to know something about someone’s behavior on
twitter. Or how a specific topic is discussed on Twitter.
#bigdata
Damian Trilling
18. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Tone in tweets
A recent bachelor thesis
Tone in tweets
Imagine you want to know something about someone’s behavior on
twitter. Or how a specific topic is discussed on Twitter.
Do you really want to go through thousands of tweets by hand?
#bigdata
Damian Trilling
19. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Tone in tweets
So you’d better think about automating your coding
Finding out how negative or positive politicians are towards
there opponents
Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende
factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en
politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam.
#bigdata
Damian Trilling
20. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Tone in tweets
So you’d better think about automating your coding
Finding out how negative or positive politicians are towards
there opponents
We took lists with positive and negative words and with a
politician’s opponents.
Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende
factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en
politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam.
#bigdata
Damian Trilling
21. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Tone in tweets
So you’d better think about automating your coding
Finding out how negative or positive politicians are towards
there opponents
We took lists with positive and negative words and with a
politician’s opponents.
We used a Python-script to check which type of words were used
to refer to opponents.
Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende
factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en
politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam.
#bigdata
Damian Trilling
22. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Tone in tweets
So you’d better think about automating your coding
Finding out how negative or positive politicians are towards
there opponents
We took lists with positive and negative words and with a
politician’s opponents.
We used a Python-script to check which type of words were used
to refer to opponents.
For further analysis, the results where imported in SPSS.
Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende
factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en
politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam.
#bigdata
Damian Trilling
23. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Tone in tweets
#bigdata
Damian Trilling
24. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Tone in tweets
#bigdata
Damian Trilling
25. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Counting words and n-grams
How often are specific expressions used?
Counting words and n-grams
#bigdata
Damian Trilling
26. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Counting words and n-grams
How often are specific expressions used?
Counting words and n-grams
Imagine you want to know which words or expressions dominate a
discourse .
#bigdata
Damian Trilling
27. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Counting words and n-grams
How often are specific expressions used?
Counting words and n-grams
Imagine you want to know which words or expressions dominate a
discourse .
There are plenty of possibilities to get an answer within minutes,
here’s one:
#bigdata
Damian Trilling
28. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Counting words and n-grams
Again, just one or two lines of code!
For example with STATA
• Install the package wordscore (net install
http://www.tcd.ie/Political_Science/wordscores/wordscores)
• voor wordcounts: wordfreq /home/dami/texts/lab92.txt
/home/dami/texts/lab97.txt
• voor ngrams (trigrams in dit geval): phrasefreq 3 lab92.txt
lab97.txt
#bigdata
Damian Trilling
29. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Counting words and n-grams
trigrams in Obama-Tweets
#bigdata
Damian Trilling
30. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Network analysis
Another approach
Network analysis
#bigdata
Damian Trilling
31. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Network analysis
Another approach
Network analysis
Imagine you want to know who talks to whom and how networks
are interconnected .
#bigdata
Damian Trilling
32. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Network analysis
Another approach
Network analysis
Imagine you want to know who talks to whom and how networks
are interconnected .
Use a tool like NodeXL or Gephi!
#bigdata
Damian Trilling
33. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Network analysis
#bigdata
Damian Trilling
34. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Problems
#bigdata
Damian Trilling
35. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Problems
You sometimes depend entirely on commercial parties
#bigdata
Damian Trilling
36. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Problems
You sometimes depend entirely on commercial parties
• Services can shut down (GoogleReader) or change their API
(Twitter)
#bigdata
Damian Trilling
37. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Problems
You sometimes depend entirely on commercial parties
• Services can shut down (GoogleReader) or change their API
(Twitter)
• It’s rather easy to get (up to 3200) tweets from a specific user
(e.g., allmytweets.net), but if you want to capture a
#hashtag, you have to record it live
#bigdata
Damian Trilling
38. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Problems
You sometimes depend entirely on commercial parties
• Services can shut down (GoogleReader) or change their API
(Twitter)
• It’s rather easy to get (up to 3200) tweets from a specific user
(e.g., allmytweets.net), but if you want to capture a
#hashtag, you have to record it live
• Twitter doesn’t give you all tweets, but just about 1% (+ a
bunch of other limits)
#bigdata
Damian Trilling
39. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Problems
Not sure if this a problem or a great opportunity. . .
You cannot rely (only) on ready-made software but shout get ready
to use tools like bash-scripts, grep, python, . . . (Which can be fun!)
#bigdata
Damian Trilling
40. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
A glimpse in the kitchen
#bigdata
Damian Trilling
41. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
What I’m doing right now
Analyzing #tvduell
• 570.000 tweets
• Identifyig clusters of nouns, verbs and adjectives
• Assigning positivity and negativity scores to tweets
• See if they can be interpreted as frames
⇒How are Merkel and Steinbrück framed on the Second Secreen
during the debate?
#bigdata
Damian Trilling
43. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Something you can use?
1 What’s big data?
2 Some examples
Rare events
Tone in tweets
Counting words and n-grams
Network analysis
3 Problems
4 A glimpse in the kitchen
5 Questions?
#bigdata
Damian Trilling
44. What’s big data?
Some examples
Problems
A glimpse in the kitchen
Questions?
Vragen of opmerkingen?
Damian Trilling
d.c.trilling@uva.nl
@damian0604
www.damiantrilling.net
#bigdata
Damian Trilling