Conference talk at WWW 2012, , April 2012, Lyon, France

1. ZenCrowd
Leveraging
Probabilis3c
Reasoning
and
Crowdsourcing
Techniques
for
Large-­‐Scale
En3ty
Linking
Gianluca
Demar3ni,
Djellel
Eddine
Difallah,
and
Philippe
Cudré-­‐Mauroux
eXascale
Infolab,
University
of
Fribourg
Switzerland

2. MoFvaFon
• Linked
Open
Data
(LOD)
• Linking
enFty
from
text
to
LOD
– Rich
snippets
– EnFty-­‐centric
search
• Linking
– Algorithmic
– Manual
(NYT
arFcles)
HTML+ RDFa
Pages
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
2

3. Gianluca
DemarFni,
eXascale
Infolab
3
hVp://dbpedia.org/resource/Facebook
hVp://dbpedia.org/resource/Instagram
Zase:Instagram
owl:sameAs
Google
Android
<p>Facebook
is
not
waiFng
for
its
iniFal
public
offering
to
make
its
first
big
purchase.</p><p>In
its
largest
acquisiFon
to
date,
the
social
network
has
purchased
Instagram,
the
popular
photo-­‐sharing
applicaFon,
for
about
$1
billion
in
cash
and
stock,
the
company
said
Monday.</p>
<p><span
about="hVp://dbpedia.org/resource/
Facebook"><cite
property=”rdfs:label">Facebook</
cite>
is
not
waiFng
for
its
iniFal
public
offering
to
make
its
first
big
purchase.</span></p><p><span
about="hVp://dbpedia.org/resource/Instagram">In
its
largest
acquisiFon
to
date,
the
social
network
has
purchased
<cite
property=”rdfs:label">Instagram</
cite>
,
the
popular
photo-­‐sharing
applicaFon,
for
about
$1
billion
in
cash
and
stock,
the
company
said
Monday.</span></p>
RDFa
enrichment
HTML:

4. Crowdsourcing
• Exploit
human
intelligence
to
solve
– Tasks
simple
for
humans,
complex
for
machines
– With
a
large
number
of
humans
(the
Crowd)
– Small
problems:
micro-­‐tasks
(Amazon
MTurk)
• Examples
– Wikipedia,
Image
tagging
• IncenFves
– Financial,
fun,
visibility
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
4

5. ZenCrowd
• Combine
both
algorithmic
and
manual
linking
• Automate
manual
linking
via
crowdsourcing
• Dynamically
assess
human
workers
with
a
probabilisFc
reasoning
framework
22-­‐Apr-­‐12
5
Crowd
Algorithms
Machines

6. Outline
• Related
approaches
• ZenCrowd:
System
architecture
• ProbabilisFc
model
to
combine
automaFc
matching
and
crowdsourcing
results
• Experimental
evaluaFon
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
6

7. Related
Approaches
• EnFty
Linking
• Ad-­‐hoc
Object
Retrieval
– Keyword
queries
– Looking
for
a
specific
enFty
– IR
indexing
and
ranking
over
RDF
data
• Crowdsourcing
– Training
set,
tagging,
annotaFon
– IR
evaluaFon
– CrowdDB
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
7

8. ZenCrowd
Architecture
Micro
Matching
Tasks
HTML
Pages
HTML+ RDFa
Pages
LOD Open Data Cloud
Crowdsourcing
Platform
ZenCrowd
Entity
Extractors
LOD Index Get Entity
Input Output
Probabilistic
Network
Decision Engine
Micro-
TaskManager
Workers Decisions
Algorithmic
Matchers
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
8

9. Algorithmic
Matching
• Inverted
index
over
LOD
enFFes
– DBPedia,
Freebase,
Geonames,
NYT
• TF-­‐IDF
(IR
ranking
funcFon)
• Top
ranked
URIs
linked
to
enFFes
in
docs
• Threshold
on
the
ranking
funcFon
or
top
N
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
9

10. EnFty
Factor
Graphs
• Graph
components
– Workers,
links,
clicks
– Prior
probabiliFes
– Link
Factors
– Constraints
• ProbabilisFc
Inference
– Select
all
links
with
posterior
prob
>τ
w1
w2
l1
l2
pw1( ) pw2( )
lf1( ) lf2( )
pl1( ) pl2( )
l3
lf3( )
pl3( )
c11
c22
c12
c21
c13
c23
u2-3( )sa1-2( )
2
workers,
6
clicks,
3
candidate
links
Link
priors
Worker
priors
Observed
variables
Link
factors
SameAs
constraints
Dataset
Unicity
constraints
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
10

11. EnFty
Factor
Graphs
• Training
phase
– IniFalize
worker
priors
– with
k
matches
on
known
answers
• UpdaFng
worker
Priors
– Use
link
decision
as
new
observaFons
– Compute
new
worker
probabiliFes
• IdenFfy
(and
discard)
unreliable
workers
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
11

12. Experimental
EvaluaFon
• Datasets
– 25
news
arFcles
from
• CNN.com
(Global
news)
• NYTimes.com
(Global
news)
• Washington-­‐post.com
(US
local
news)
• Timesofindia.indiaFmes.com
(India
local
news)
• Swissinfo.com
(Switzerland
local
news)
– 40M
enFFes
(Freebase,
DBPedia,
Geonames,
NYT)
• 80
workers
on
Amazon
MTurk,
$0.01
per
enFty
• Standard
evaluaFon
measures:
Prec,
Recall,
Acc
• Gold
Standard:
editorial
selecFon
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
12

13. The
micro-­‐task
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
13

14. Experimental
EvaluaFon
• AutomaFc
Approach:
P/R
trade-­‐off
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
1" 2" 3" 4" 5"
Precision)/)Recall)
Top)N)Results)
P"
R"
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
14

15. Experimental
EvaluaFon
• EnFty
Linking
with
Crowdsourcing
and
agreement
vote
(at
least
2
out
of
5
workers
select
the
same
URI)
Top-­‐1
precision:
0.70
the URIs with at least 2 votes are selected as valid links
(we tried various thresholds and manually picked 2 in the
end since it leads to the highest precision scores while keep-
ing good recall values for our experiments). We report on
the performance of this crowdsourcing technique in Table 2.
The values are averaged over all linkable entities for di↵erent
document types and worker communities.
Table 2: Performance results for crowdsourcing with
agreement vote over linkable entities.
US Workers Indian Workers
P R A P R A
GL News 0.79 0.85 0.77 0.60 0.80 0.60
US News 0.52 0.61 0.54 0.50 0.74 0.47
IN News 0.62 0.76 0.65 0.64 0.86 0.63
SW News 0.69 0.82 0.69 0.50 0.69 0.56
All News 0.74 0.82 0.73 0.57 0.78 0.59
The first question we examine is whether there is a di↵er-
ence in reliability between the various populations of work-
Figure
textua
Entity L
We n
ference
method
consisti
phase, c
the wor
is know
In ord
ence in
of bad
ered as
clicks o
In our e
swers in
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
15

16. Experimental
EvaluaFon
• Worker
community
and
textual
context
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
0" 0.2" 0.4" 0.6" 0.8" 1"
Recall&
Precision&
US"
India"
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
1" 2" 3" 4" 5" 6" 7" 8" 9" 10"
Precision)
Document)
Simple"
Snippet"
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
16

17. Experimental
EvaluaFon
• Worker
SelecFon
Top$US$
Worker$
0$
0.5$
1$
0$ 250$ 500$
Worker&Precision&
Number&of&Tasks&
US$Workers$
IN$Workers$
0.6$
0.62$
0.64$
0.66$
0.68$
0.7$
0.72$
0.74$
0.76$
0.78$
0.8$
1$ 2$ 3$ 4$ 5$ 6$ 7$ 8$ 9$
Precision)
Top)K)workers)
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
17

18. Experimental
EvaluaFon
• EnFty
Linking
with
ZenCrowd
– Training
with
first
5
enFFes
+
5%
aserwards
– 3
consecuFve
bad
answers
lead
to
blacklisFng
di↵er-
work-
s per-
point
all en-
tasks
higher
orkers
ws as
rms of
ed on
clicks on the links, hence generating noise in our system.
In our experiments, we consider that 3 consecutive bad an-
swers in the training phase is enough to identify the worker
as a spammer and to blacklist him/her. We report the aver-
age results of ZenCrowd when exploiting the training phase,
constraints, and blacklisting in Table 3. As we can observe,
precision and accuracy values are higher in all cases when
compared to the agreement vote approach.
Table 3: Performance results for crowdsourcing with
ZenCrowd over linkable entities.
US Workers Indian Workers
P R A P R A
GL News 0.84 0.87 0.90 0.67 0.64 0.78
US News 0.64 0.68 0.78 0.55 0.63 0.71
IN News 0.84 0.82 0.89 0.75 0.77 0.80
SW News 0.72 0.80 0.85 0.61 0.62 0.73
All News 0.80 0.81 0.88 0.64 0.62 0.76
Finally, we compare ZenCrowd to the state of the art22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
18

19. Comparing
3
matching
techniques
• ZenCrowd
best
for
75%
of
documents
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
19
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
1" 2" 3" 4" 5" 6" 7" 8" 9" 10"11"12"13"14"15"16"17"18"19"20"21"22"23"24"25"
Precision)
)
Document)
Agr."Vote"
ZenCrowd"
Top"1"
Simple
Crowdsourcing
ZenCrowd
AutomaFc

20. Lessons
Learnt
• Crowdsourcing
+
Prob
reasoning
works!
• But
– Different
worker
communiFes
perform
differently
– Many
low
quality
workers
– No
differences
w/
different
contexts
– CompleFon
Fme
may
vary
(based
on
reward)
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
20

21. Conclusions
• ZenCrowd:
ProbabilisFc
reasoning
over
automaFc
and
crowdsourcing
methods
for
enFty
linking
• Standard
crowdsourcing
improves
6%
over
automaFc
• 4%
-­‐
35%
improvement
over
standard
crowdsourcing
• 14%
average
improvement
over
automaFc
approaches
• Next
steps
– Long-­‐term
worker
behavior
analysis
– More
efficient
and
effecFve
linking
by
LOD
dataset
pre-­‐
selecFon
hVp://diuf.unifr.ch/xi/zencrowd/
22-­‐Apr-­‐12
Gianluca
DemarFni,
eXascale
Infolab
21