"Video Killed the Radio Star": From MTV to Snapchat

“Video Killed the Radio Star”
the path from MTV to Snapchat
Lora Aroyo
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

The
CNN/YouTube
Republican
Debate
on
2007-‐11-‐28

h;p://www.blogherald.com/2010/10/27/history-‐of-‐online-‐video/

massive
amount
of
digital
content
to
explore
…


but
at
some
point
it
all
looks
the
same
…


Massive Scale:
A lifetime of video content is uploaded to YouTube everyday.
Granularity Mismatch:
Searching for the relevant video fragments is still not possible.
Passive Engagement:
Video is still primarily a linear net-time viewing activity

… people search & browse
with some implicit relevance in mind

snapchat
genera8on
…


audiences
feel
disconnected
&
lost
…


there
is
huge
seman8c
&
cultural
GAP


so=ware
systems
are
ever
more
intelligent

but
they
don’t
actually
understand
people

focus
on
human
knowledge
in
machine-‐readable
form

but
there
are
types
of
human
knowledge

that
can’t
be
captured
by
machines

classical
AI
involves
human
experts
to
manually

provide
training
knowledge
for
machines

human
expert-‐based
ground
truth
does
not
scale

for
current
demand
for
machines
to
deal
with
wide

ranges
of
real-‐world
tasks
and
contexts

we
need
to
be
able
to
….

support
of
mulGple
perspecGves


to
provide
an
approach
to
capturing
human
knowledge

in
a
way
that
is
scalable
&
adequate
to
real-‐world
needs

the
key
scien8ﬁc
challenge
is

Goodbye
Single
Truth
Hello
Multiple
Perspectives

humans
accurately
perform
interpreta8on
tasks


humans
accurately
perform
interpreta8on
tasks

can
their
eﬀort
be
adequately
harnessed
in
a

scien8ﬁcally
reliable
manner
that
scales
across
tasks,

contexts
&
data
modali8es?

Quan8ty
is
the
new
Quality

Human
Computa8on
adopts
human
intelligence
at

scale
to
improve
purely
machine-‐based
systems

diversity
of
opinion

Independent

decentralized

aggregated

James
Surowiecki

“the
wise
crowd”


a
novel
approach
to
gather
diversity
of
perspec8ves
&

opinions
from
the
crowd,
expand
expert
vocabularies
with

these
and
gather
new
type
of
gold
standard
for
machines

L.
Aroyo,
C.
Welty:
Crowd
Truth:
Harnessing
disagreement
in
crowdsourcing
a
rela?on
extrac?on
gold
standard.
ACM
WebSci
2013.

L.
Aroyo,
C.
Welty.
The
Three
Sides
of
CrowdTruth,
Journal
of
Human
Computa?on,
2014

http://CrowdTruth.org
http://data.CrowdTruth.org/
http://game.crowdtruth.org

Visual
Content
Domina8on

•  90%
of
informa8on
transmiSed
to
the
brain
is
visual
(processed
60,000X
faster
in

the
brain
than
text)

•  Videos
increase
average
page
conversion
rates
by
86%

•  Visuals
are
social-‐media-‐ready/friendly
-‐
easily
sharable

•  Posts
with
visuals
receive
94%
more
page
visits

•  Visuals
are
becoming
easier
and
easier
to
create
as
photo
/
video
ediGng
tools

become
more
accessible

any piece of media can be the starting point to
a world of compelling visual experiences.
turning “mute” images into content-aware images.

NEW JERSEY
HUDSON RIVER
CENTRAL PARK
URBANIZATION
VERIZON
METLIFE BUILDING
SUNSET
EAST RIVER
NEW YORK CITY
SKYSCRAPER
UPPER EAST SIDE
turning “mute” images into content-aware images.
any piece of media can be the starting point to
a world of compelling visual experiences.

combining machine processing with
crowdsourcing for enriching, curating &
gathering metadata
quickly & cheaply — at scale.
NEW JERSEY
HUDSON RIVER
CENTRAL PARK
URBANIZATION
VERIZON
METLIFE BUILDING
SUNSET
EAST RIVER
NEW YORK CITY
SKYSCRAPER
UPPER EAST SIDE

NEW JERSEY
HUDSON RIVER
CENTRAL PARK
URBANIZATION
VERIZON
NEW YORK CITY
SKYSCRAPER
METLIFE
BUILDING
UPPER EAST SIDE
EAST RIVER
MIDTOWN
MANHATTAN
PAN-AM BUILDING
PAN-AM AIRLINES HELICOPTER CRASH
AIR TRAVEL
ARCHITECTURE
turning “context-free” images in
relationship-aware images

NEW JERSEY
HUDSON RIVER
CENTRAL PARK
URBANIZATION
VERIZON
NEW YORK CITY
SKYSCRAPER
METLIFE
BUILDING
UPPER EAST SIDE
EAST RIVER
MIDTOWN
MANHATTAN
PAN-AM BUILDING
PAN-AM AIRLINES HELICOPTER CRASH
AIR TRAVEL
ARCHITECTURE
… not only images, but also for videos
YOUTUBE: NYC FROM THE
EMPIRE STATE BUILDING
allowing viewers to explore relationships across themes,
locations, characters, etc. — within a video.

h;p://www.adweek.com/socialGmes/millennials-‐love-‐video-‐on-‐mobile-‐social-‐channels-‐infographic/622313

BRIDGING THE GAP BETWEEN
PEOPLE & THE OVERWHELMING
AMOUNT OF ONLINE MULTIMEDIA CONTENT

HyperVideos:
Link video fragments in non-linear paths
Binging Engagement:
Construct continuous and interactive experiences
Video Snacks:
Break video down into snackable moments
SOLUTIONS

•  Decomposing &
granular description
of images & videos.
•  Constructing
mediaGraph with
rich media semantics.
•  Continuously
enriching &
consolidating
machine, expert, &
user content
descriptions.

Machines
&
Crowds

http://waisda.nl
Crowdsourcing
Video
Tags

@Sound
and
Vision

Results
of
First
Pilot

– The
ﬁrst
6
months:

•  44.362
pageviews

•  12.279
visits
(3+
min
online)

•  555
registered
players
(thousands
anonymous
players!)

– 340.551
tags
added
to
602
items

– 137.421
matches

Results
of
First
Pilot

11

PartcipaGng
Museums

1,782

Works
of
Art
in
the
Research

36,981
Tags
collected

2,017

Users
who
tagged

First
two
years
(2006-‐2008)

Q: Why did you tag?
0% 20% 40% 60% 80% 100%
don't remember
to connect with others
so that I could find works again later
other (please specify)
to learn about art
to improve search for other users
for fun
to help museums document art work
Public
MMA

Tags
by
Documentalists

•  Tags
describe
mainly
short
segments

•  Tags
are
oaen
not
very
speciﬁc

•  Tags
not
describe
programmes
as
a
whole

•  User
tags
were
useful
&
speciﬁc
-‐-‐>
domain
dependent

user vocabulary
8% in professional vocabulary
23% in Dutch lexicon
89% found on Google
locations (7%)
engeland
persons (31%)
objects (57%)
On
the
Role
of
User-‐Generated
Metadata
in
A/V
Collec?ons

Riste
Gligorov
et
al.
KCAP
Int.
Conference
on
Knowledge
Capture
2011

Crowd
vs.
Professionals

System MAP
All user tags 0.219
Consensus user tags only 0.143
NCRV tags 0.138
NCRV catalog 0.077
Captions 0.157
Captions + User tags 0.247
Captions + NCRV catalog 0.183
Captions + NCRV tags 0.201
NCRV tags + User tags 0.263
NCRV tags + NCRV catalog 0.150
All – User tags 0.208
All 0.276
All tags better than consensus only
• Improvement of 53%
• Consensus tags have
• higher precision: 0.59 vs. 0.49
• but lower recall: 0.28 vs. 0.42
WAISDA?
Tags
vs.
Rest

System MAP
All user tags 0.219
NCRV tags 0.138
NCRV catalog 0.077
Captions 0.157
All 0.276
All tags better than rest
• Individually
• beat NCRV tags by 69%
• beat captions by 39%
WAISDA?
Tags
vs.
Rest

System MAP
All user tags 0.219
NCRV tags 0.138
NCRV catalog 0.077
Captions 0.157
All 0.276
• Individually
• Combined
WAISDA?
Tags
vs.
Rest

System MAP
All user tags 0.219
NCRV tags 0.138
NCRV catalog 0.077
Captions 0.157
All 0.276
All data performs best
• largely due to contribution of
user tags – 33%
WAISDA?
Tags
vs.
Rest

System MAP
All user tags 0.219
NCRV tags 0.138
NCRV catalog 0.077
Captions 0.157
All 0.276
All tags better than consensus only
• Consensus tags have
• higher precision: 0.59 vs. 0.49
• but lower recall: 0.28 vs. 0.42
• Individually
All data performs best
• largely due to contribution of
user tags – 33%
• Combined
WAISDA?
Tags
vs.
Rest

Current
Pilot

h;p://spotvogel.vroegevogels.vara.nl/

Accurator
ask the right crowd, enrich your collection
hSp://annotate.accurator.nl

Crowdsourcing
&
Nichesourcing

@Rijksmuseum

Rijksmuseum Amsterdam collection
over 1 million artworks

only a small fraction of about 8000 items
are currently on display

… online collection grows
125.000 artworks already available
another 40.000 are added every year

expertise of museum professionals is in
describing & annotating collection with art-
historical information, e.g. when they were
created, by whom, etc.

detailed information about depicted objects, e.g.
which species the animal or plant belongs to,
is in most cases not available

annotated only with “bird with blue head near
branch with red leaf”
species of the bird and the plant are missing

use crowdsourcing to get more annotations
use nichesourcing, i.e. niches of people with the
right expertise, to add more specific information

use sources like Twitter to find experts or
groups of experts on certain areas, e.g. bird
lovers, ornithologists or people who enjoy bird-
watching in their spare time

platform where users enter tags:
(1) structured vocabulary terms or (2) free text
hSp://annotate.accurator.nl

for tasks that are too difficult:
game in which players can carry out an expert
annotation task with some assistance

BIRDWATCHING RIJKSMUSEUM
Sunday October 4, 10.00 am - 14.00 pm
Cuypers Library Rijksmuseum
On World Animal Day, the Rijksmuseum will host a
birdwatching day in collaboration with Naturalis
Biodiversity Center, Wikimedia Netherlands and the
COMMIT/ SEALINCMedia project.
We are looking for bird watchers to join an expedi-
tion through the digital collections and help the
museums identify bird species in works of art.

dive.beeldengeluid.nl

In
Digital

Hermeneu8cs

Event-‐centric
Explora8on

@Sound
&
Vision
and
Royal
Library

3rd
Price
at
the
SemanGc
Web
Challenge
2014

OPENIMAGES.EU

•  3000
videos

•  NL
InsGtute
for
Sound
&
Vision

•  mostly
news
broadcasts

DELPHER.NL

•  1.5
Million
Scans
of

•  Radio
bulleGns

•  (hand
annotated)

•  1937
–
1984

Simple
Event
Model
(SEM)

OpenAnnota8on
(OA)
&
SKOS

DIVE:MEDIA OBJECT
SEM:EVENT

SEM:PLACE

SEM:TIME

SEM:ACTOR

SKOS:CONCEPT

OA:ANNOTATION

•  LINKS
TO
EUROPEANA
(MULTILINGUAL)

•  LINKS
TO
DBPEDIA

Digital
Submarine
UI

Inﬁnity
of
Explora8on

Events
Linking
Objects

Crowd
Bringing

the
Human
Perspec8ves

Linked
(Open)
Data

En8ty
&
Event
Extra8on
with
CrowdTruth.org

ENTITY EXTRACTION
EVENTS CROWDSOURCING AND LINKING TO
CONCEPTS THROUGH CROWDTRUTH.ORG
SEGMENTATION & KEYFRAMES
LINKING EVENTS AND
CONCEPTS TO KEYFRAMES

Erp,
M.
van;
Oomen,
J.;
Segers,
R.;
Akker,
C.
van
de;
Aroyo,
L.;
Jacobs,
G.;
Legêne,
S;
Meij,
L.
van
der;O
ssenbruggen,
J.R.
van;
Schreiber,
G.

AutomaGc
Heritage
Metadata
Enrichment
with
Historic
Events
Museums
and
the
Web
2011
h;p://www.museumsandtheweb.com/mw2011/
papers/automaGc_heritage_metadata_enrichment_with_hi


engaging
users
through
event
narratives

“Digital
HermeneuGcs:
Agora
and
the
online
understanding
of
cultural

heritage”
In
proc.
of
Web
Science
Conference,
(ACM:
New
York,
2011)

Interpreta8on
Support
for
Online
CollecGons

Explora8ve
Search

Engagement
with
Games

Links
from
the
slides

On
the
Web
•  http://waida.nl
•  http://prestoprime.org
•  http://agora.cs.vu.nl
•  http://sealincmedia.wordpress.com
•  http://dive.beeldengeluid.nl
•  http://diveplu.beeldengeluid.nl
•  http://annotate.accurator.nl
•  http://accurator.nl
•  http://crowdtruth.org
•  http://data.crowdtruth.org
•  http://game.crowdtruth.org
•  http://www.adweek.com/socialtimes/
millennials-love-video-on-mobile-social-
channels-infographic/622313
•  http://www.blogherald.com/2010/10/27/
history-of-online-video/
•  http://wm.cs.vu.nl

On
TwiSer

@waisda

@agora-‐project

@sealincmedia

@prestocenter

@vistatv

#CrowdTruth

#Accurator


Lecture
Reading
Material

h;p://www.aaai.org/ojs/index.php/aimagazine/arGcle/view/2564

Truth
Is
a
Lie:
Crowd
Truth
and
the
Seven
Myths
of
Human
AnnotaGon

h;ps://www.wired.com/2006/06/crowds/

THE
RISE
OF
CROWDSOURCING

h;ps://www.microsoa.com/en-‐us/research/project/algorithmic-‐crowdsourcing/

h;p://cci.mit.edu/publicaGons/CCIwp2011-‐04.pdf

Programming
the
Global
Brain

h;p://www.orchid.ac.uk/eprints/248/1/main.pdf

The
ACTIVECROWDTOOLKIT:
An
Open-‐Source
Tool
for
Benchmarking
AcGve

Learning
Algorithms
for
Crowdsourcing
Research


"Video Killed the Radio Star": From MTV to Snapchat

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Semelhante a "Video Killed the Radio Star": From MTV to Snapchat

Semelhante a "Video Killed the Radio Star": From MTV to Snapchat (20)

Mais de Lora Aroyo

Mais de Lora Aroyo (15)

Último

Último (20)

"Video Killed the Radio Star": From MTV to Snapchat