UN Global Pulse: Big Data for a Better World (Strata Conf NYC)

“BEYOND TARGETED ADS
BIG DATA FOR A BETTER WORLD”
Robert Kirkpatrick
Director, UN Global Pulse
O’Reilly Strata Conference | New York | October 2012

www.unglobalpulse.org @unglobalpulse

25
October
2012
|
www.unglobalpulse.org

Kenyan Farm Workers

Photo Credit: John Oyuke

25
October
2012
|

Hyperconnectivity

Airplane Flights Telephone Calls

Internet Traffic Social Networking

25
October
2012
|

20th-Century tools…

25
October
2012
|

PRIVATE SECTOR
Monitor operations…in real time
Track market trends…in real time
Get customer feedback…in real time

GLOBAL DEVELOPMENT
Unemployment?
Food Security?
Public Health?
Education?
Migration?
Disaster Relief?

BIG DATA IN REAL TIME: 3 OPPORTUNITIES

1.  Better early warning: Earlier detection of
anomalies, trends and events allows earlier
response.

2.  Real-time awareness: A more accurate and up-to-
date picture of population needs supports more
effective planning and implementation

3.  Real-time feedback: Understanding sooner where
needs are changing -- or are not being met --
allows for rapid, adaptive course correction

25
October
2012
|

BIG DATA IS A HUMAN RIGHTS ISSUE

•  Never analyze personally identifiable information
•  Never analyze confidential data
•  Never seek to re-identify individuals

25
October
2012
|

BACKGROUND: A GROWING BODY OF EVIDENCE

RESEARCH

How Mobile Phone Carriers See the World

Call Detail Records (CDRs)
•  Caller ID (hashed phone #)
•  Caller Tower Location
•  Receiver ID (hashed phone #)
•  Receiver Tower Location
•  Call Start Time
•  Call Duration
Airtime Expense Records
•  Caller ID (hashed phone #)
•  Caller Tower Location
•  Amount of Purchase
•  Time of Purchase
•  Balance at Time of Purchase
25
October
2012
|

Modeling Behaviors in Mobile Data
Consumption variables
• Number of calls, call duration, SMS/MMS/voice
• Size, frequency, total number of airtime purchases
• Handset Type and Features

Social variables
• Degree of the social network
• Weight of the contacts, frequency of communication

Mobility variables
• Diameter of mobility and social network
• Radius of gyration
• Mobility Patterns
Source:
Telefonica
Research,
2011

25
October
2012
|

EXAMPLE: AIRTIME CREDIT PURCHASE DATA

25
October
2012
|

SIZE AND FREQUENCY PREDICT HOUSEHOLD INCOME

Higher household
income
Average size of purchase

Lower household
income

Average # of purchases / month
25
October
2012
|

CALLING PATTERNS AND ECONOMIC OPPORTUNITY

Lower Higher
socioeconomic level socioeconomic level

25
October
2012
|

MEN AND WOMEN USE THEIR PHONES DIFFERENTLY
Men: Women:
•  Fewer calls •  More calls
•  Shorter calls •  Longer calls
•  Smaller social network •  Larger social network
•  More work-related calls •  More personal calls

25
October
2012
|

Tracking population movement to predict cholera

Source: Linus Bengtsson et. al., PLoS Medicine, 2011
25
October
2012
|

A mobility index to evaluate H1N1 response in Mexico City

Source: Telefonica Research, 2011
See: http://www.unglobalpulse.org/publicpolicyandcellphonedata

25
October
2012
|

TWITTER PREDICTS SPREAD OF INFLUENZA

r2 = .958

“You Are What You Tweet: Analyzing Twitter for Public Health. M. J. Paul and M. Dredze, 2011.”
http://www.cs.jhu.edu/%7Empaul/files/2011.icwsm.twitter_health.pdf
25
October
2012
|

Rumi Chunara et. al., American Journal of Tropical Medicine and Hygiene, 2012 86:39-45

25
October
2012
|

GOOGLE SEARCHES FOR SYMPTOMS PREDICT DENGUE

25
October
2012
|

2010 VS. 2011: INDONESIAN TWEETS ABOUT HIV

See: http://www.unglobalpulse.org/WorldAIDSDay-Part2
25
October
2012
|

GLOBAL PULSE RESEARCH 2011

PROOF OF CONCEPT
STUDIES
Online at: http://www.unglobalpulse.org/applyingbigdatatodevelopment

http://www.unglobalpulse.org/projects/can-social-media-mining-add-depth-unemployment-statistics

Online Discussions & Unemployment
Ireland

25
October
2012
|

Online Discussions & Unemployment
United States

25
October
2012
|

http://www.unglobalpulse.org/projects/twitter-and-perceptions-crisis-related-stress

Jakarta: nine million tweets per day

Map
of
Twi*er
usage
in
Jakarta
–
by
Eric
Fischer

25
October
2012
|

Tweets per day about food,
during Ramadan in Indonesia
Start of Ramadan

End of Ramadan

25
October
2012
|

Tweets predict food basket inflation
(rice, chilies, fish, sugar, corn, cooking oil)

Tweets about the
price of rice
(per month)

Official
Food Price Inflation
(monthly from 25 cities)

25
October
2012
|

DIGITAL SERVICES AS HUMAN SENSOR NETWORKS:
Observing fluctuations in well-being…in real-time

COPING STRATEGIES DIGITAL “SMOKE SIGNALS”
•  Buy cheaper foods •  Depletion of airtime credit
•  Work longer hours •  Smaller mobile airtime
•  Reduce energy use purchases
•  Draw down savings •  Failure to repay microloans via
•  Sell assets mobile financial services
•  Borrow from relatives •  Changes in calling patterns
•  Inbound money transfers
•  Web searches for jobs, health
•  Sales of livestock via mobile
trading network
•  “Venting” on social media

25
October
2012
|

Photo: Ministry of Foreign Affairs, Iceland

25
October
2012
|

AGILE GLOBAL DEVELOPMENT?

25
October
2012
|

Integrating real-time data into an institution

•  This data may be less
accurate that official
sources.

•  But it’s faster.

•  And it’s cheaper to
collect.

•  How can we leverage USGS Twitter Earthquake Detector
the speed to change (TED)
the outcome?

25
October
2012
|

THE PROBLEM WITH TELESCOPES…

…AND MACROSCOPES

There’s a universe of data that we can’t see.
25
October
2012
|

A
global
real-time
public/private
data commons?

EXAMPLE R&D PROJECT:
Mobile Networks as Drought Sensors in the Sahel

Proposal
•  Obtain 2011-2012 mobile CDRs and airtime purchases.
•  Derive mobility, consumption, and social variables.
•  Correlate variables with precipitation levels, survey
data.
•  Identify signatures of drought impacts in 2011.
•  Identify signatures of aid impact in 2012.
•  Develop and evaluate prototype during next drought.
•  Release open source “appliance” through GSMA.

25
October
2012
|

Joint Research | Rapid Prototyping | Capacity Building

25
October
2012
|

Pulse Lab Network

Pulse Lab Jakarta…………October 2012
Pulse Lab Kampala……….January 2013
Other locations…………...???

25
October
2012
|

PULSE LABS
R&D INNOVATION STRATEGY
1.  Partner with governments to establish Pulse Labs
2.  Build world-class teams of data scientists,
engineers, and policy experts
3.  Partner with private sector for real-time data and
cutting edge technology
4.  Work with UN agencies and academia to conduct
research around challenges in
5.  Build open source prototypes of tools to automatic
real-time monitoring
6.  Support broad adoption of useful innovations
7.  Share everything we learn and build
25
October
2012
|

SO HOW DO I GET INVOLVED?
ARE YOU..
•  A company with powerful data you think
could make the world a better place?

•  A technology provider with screaming fast
computing or killer analytics?

•  A whiz data scientist interested in hard
problems, positive impact, and global scale?

•  A big data privacy expert who understands
that we cannot help unless we also protect?
25
October
2012
|

Research Tool 1
Crimson Hexagon: ForSight

25
October
2012
|

Food Prices: What a real crisis looks like
23
July
-‐
2
Aug

‘tempeh’
and
‘tofu’
hot
debate

14
-‐
21
Aug

Ramadhan
/
Idul
Fitri

25
October
2012
|

Comparing Crises

Tweets
about
food

18
Mar
-‐
7
Apr
23
July
-‐
2
Aug

Fuel
subsidy
cut
plan
and
‘tempeh’
and
‘tofu’
hot
debate

protests
against
it
during
soybean
shortage

Research Tool 2
SAS Social Media Analytics and SAS Text Miner

25
October
2012
|

Analytic Workflow
1)
Over
200,000
new

5)
Explore
results
and
correlate

3)
Capture
senKment
with
official
staKsKcs
to
official

Indonesian
language

and
mood
for
Bahasa

BPS
staKsKcs
:
Consumer
Price

documents
per
day
Index
(CPI)
for
12
common
foods

Global Pulse Sentiment, Topic &
Internet Relevance Mood & Geography Interactive
Conversation Filter Influence Categories
Dashboard

2)
Extract
conversaKons
4)
Detect
locaKon,
price,

about
rice,
cooking
oil,
availability,
specific

fuel,
employment,
etc.
govt.
programs,
etc.

anxious,

confident,

confused,

hosKle,

sad,

happy

(-”-) ;-) ((+_+)) :@ :( :)

What’s the deal with Indonesians and eggs?

For every 5000 more tweets about eggs…

…we see a 2-3% decrease in food CPI?

The Signals Are Getting Stronger

è Big increase in volume of relevant conversations over 18 months
40000

35000

minyak
(oil)

ketahanan
pangan
(food
security)

30000

budidaya
(culKvaKon)

25000

telur
(eggs)

20000

15000

10000

5000

0

Indonesians are increasingly using social media to discuss basic needs

25
October
2012
|

So Are the Temporal Correlations

è Listening to social conversations provides insight on official data

2.5

2

1.5

1

0.5

0

-‐0.5

-‐1

-‐1.5
Social
Media
Food
Index

-‐2

BAPPENAS
Food
Price
Index

-‐2.5

25
October
2012
|

Next up for Pulse Lab Jakarta research:
1 year of anonymized Indonesian CDRs

•  4 largest carriers
•  170 million subscribers
•  200 billion call records
•  80 terabytes of data

25
October
2012
|

ROBERT KIRKPATRICK
Director
UN Global Pulse
kirkpatrick@un.org
+1 (650) 796-5709

Image credit: Aaron Koblin
24 hours of AT&T phone calls and Internet
traffic flowing through New York City

UN Global Pulse: Big Data for a Better World (Strata Conf NYC)

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (9)

Semelhante a UN Global Pulse: Big Data for a Better World (Strata Conf NYC)

Semelhante a UN Global Pulse: Big Data for a Better World (Strata Conf NYC) (20)

Mais de UN Global Pulse

Mais de UN Global Pulse (20)

UN Global Pulse: Big Data for a Better World (Strata Conf NYC)