Crowdsourcing for NLP Ground Truth Data

From Crowd Knowledge
to Machine Knowledge
gather annotation of types, events, relations, coref

Lora Aroyo and Chris Welty

T
e
x
t Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo Text
Wednesday, October 17, 12 1

Position
are par t of the
t & vag ue ne s s
disag reemen mantics
The human t & relations se
even

Flickr: elkabong Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo

Position
Artiﬁcially
restricting
humans d
Machines oes not h
will learn elp mach
from dive ines to le
arn.
rsity

Flickr: elkabong Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo

Disagreement Framework

• ontology: disagreements on the basic status of events
themselves as referents of linguistic utterances, e.g. are
people events or do events exist at all.

• granularity: disagreements that result from issues of
granularity, e.g. the location being a country, region, or city, the
time being a day, week, month, etc.

• interpretation: disagreements that result from (non-
granular) ambiguity, diﬀerences in perspective, or error in
interpreting an expression, e.g. classifying a person as a
terrorist/hero, ”October Revolution” took place in September.

Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo

Approach Principles
1. tolerate, capture & exploit disagreement
2. understand the disagreement by creating a space of possibilities (frequencies & similarities)
3. score the machine output based on where it falls in this space
4. adaptable to new annotation tasks

Flickr: auroille Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo

Event Extraction
crowdsourcing ground truth data

Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo Lora Aroyo

Event Participants
Disagreement
Israeli
Prime minister 10%
50% Government
Benjamin
Netanyahu Israeli Cabinet 15%
his Cabinet 15%
35%
Benjamin {TOLD}
Netanyahu Benjamin
Israeli Prime Netanyahu’s 5%
15% minister Cabinet
Cabinet 45%

Temporal Disagreement

Prime minister
50% Benjamin 50%
Sunday
Netanyahu
March 1, 1998 25%
35%
Benjamin {TOLD} March 1998 15%
Netanyahu
Spring 1998 5%
Israeli Prime
15% minister


Spatial Disagreement

Southern
35%
30% Israel Lebanon
{WILLING TO
WITHDRAW} Lebanon 45%
65% Israel's Northern
Frontier
Middle East 10%


it
seems
to
refer
to
an

Top
Israeli
officials
SENT
strong

does not inference
or

new
SIGNALS
Sunday
that
Israel

refer to communicated
feeling

wants
to
withdraw
from
southern

an event more
than
specific

Lebanon,
...
event.

a
group
of
people
did

refers to
something
specific
at
a

an event
specific
point
in
6me.

the
actors
in
ques6on

(top
Israeli
officials)

refers to
performed
an
ac6on

an event
during
a
specified
6me

(Sunday).

it
refers
to
what
the

israelis
did
on
sunday,

a
specific
6me.

it
is
not
a
par6cular

That
1978
resolu6on
calls
for

movement
that
has
or
is

Israel's
uncondi6onal
does not going
on
but
a
request
that

WITHDRAWAL
from
the
self-‐ refer to the
country
of
Israel

declared
security
zone
it
an event remove
their
forces
from

occupies
in
south
Lebanon,
...
the
zone
they
occupy.
does not
refer to
an event the
sentence
is
speaking
of

a
demand
for
a
withdrawal

that
had
not
yet
occurred.

refers to
an event
Because
it
is
describing
a

historical
issue
concerning

the
resolu6on
of
1978


Relation Extraction
crowdsourcing ground truth data

Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo Lora Aroyo

6 experiments
• 2 professional • 30 CFworkers per
annotators per sentence sentence @20sentences
@20sentences
• 30 CFworkers per
• 10 CFworkers per sentence @10sentences
sentence @20sentences + relations deﬁnitions

• 20 CFworkers per • 10 CFworkers per
sentence @20sentences sentence @20sentence
explanation validation


The Task


The Steps:
Example Sentence (1)


The Steps:
Example Sentence (2)


OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food
and Drug Administration (FDA) for the treatment of SCHIZOPHRENIA
and bipolar disorder.

Is SCHIZOPHRENIA related to OLANZAPINE ?

treated_by treats may_treat

RESPIRATORY ALKALOSIS is a medical condition in which increased
respiration (HYPERVENTILATION) elevates the blood pH (a condition
generally called alkalosis).
RESPIRATORY
Is HYPERVENTILATION related to
ALKALOSIS
?

diagnosed_by cause_of cause may_cause symptom_of


He was the ﬁrst physician to identify the relationship between
HEMOPHILIA and HEMOPHILIC ARTHROPATHY.
HEMOPHILIC
Is HEMOPHILIA related to ?
ARTHROPATHY

other
treated_by cause_of cause may_cause symptom_of
has_manifestation

relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY

It just says there is a relation between the two
but gives no speciﬁcs about what the relation is

There is a relationship between the two disorders but the
sentence does not indicate what that relationship is.
identify the relationship between


Distributions Compared
140"

120"

100"

80"

60"

40"

20"

0"
sTB" sT" sMT" sPB" sP" sMP" sDB" sD" sMD" sCO" sC" sMC" sLO" sHM" sDF" sSS" sOTH" sNO"

10"workers" 20"workers" 30"workers"


Sentence-Relation Distribution
Professional Annotators


10w x 20s


10w x 20s
is
differentiate
d from
is_related

caused_by
caused_by

x caused_by

is_type_of, is_a
has_type, includes

is_related
to
maybe
related to

20w x 20s

is
differentiate
d from
is_related

caused_by
caused_by

x caused_by

is_type_of, is_a
has_type, includes

is_related
to
maybe
related to

30w x 20s

is
differentiate
d from
is_related

caused_by
caused_by
x caused_by

is_type_of, is_a
has_type, includes

is_related
to
maybe
related to

30w x 10s
with relation explanations


The Task


The Steps: Example Sentence


The Dark Side of Crowdsourcing
Disagreement

• disagreement is beautiful, except when it results from spamming
• crowdsourcing has to account for people that want to get paid for
not doing any work
• spammers generate disagreement for the wrong reasons
• most spam detection requires gold standard

• a new way of measuring
Extraction of
ground truth

•
Putative Events
a new set of semantic
input:
putative events
Manual selection of
Gold Questions features for learning in
input:
output of A event extraction
Phase I:
Phase I:
A. Collect event input:
annotations + output of A C. Filtering spam
motivations event annotations

input: input: input:
list of events list of events list of events

Phase III: Phase IV:
Phase II:
A. Collect event A. Collect event
A. Collect event types modalities +
+ motivations role ﬁllers +
motivations motivations

output of A output of A output of A
output of A Manual output of A Manual output of A Manual
selection of selection of selection of
Gold Questions Gold Questions Gold Questions

Phase II: Phase IV:
Phase III:
B. Filtering spam B. Filtering spam B. Filtering spam
event types event modalities event role ﬁllers


Questions?

@laroyo
http://lora-aroyo.org Truth Data
Croudwsourcing for gathering NLP Ground Lora Aroyo

Crowdsourcing for NLP Ground Truth Data

Recomendados

Recomendados

Mais conteúdo relacionado

Mais de Lora Aroyo

Mais de Lora Aroyo (20)

Crowdsourcing for NLP Ground Truth Data