StorySourcing: Telling Stories with Humans & Machines

http://lora-aroyo.org @laroyo
Lora Aroyo
StorySourcing:
TELLING STORIES WITH HUMANS & MACHINES
User Centric Data Science Group

Information
Heritage Organizations as Inventories of the World
André Malraux, The Imaginary Museum of World Sculpture, 1953

Interpretation
Heritage Organizations as a Place to Engage with the World

CULTURAL HERITAGE
4
Before the Digital Age
Lots of manual effort
Focus on internal collection
management
Focus on art historical
significance
Access targeted to
researchers & professionals
Small curated selection
online for general audiences
onsite

DIGITAL HERITAGE
5
Bringing collections online
Focus on massive
digitization of heritage
collections
Getting large collections
online
Still need significant art
historical understanding to
get access
Metadata not sufficient for
the online presence

Knowledge Representation, Taxonomies, Thesauri
METADATA ENRICHMENT
Shared structured knowledge

Linked Data, Semantic Web, Interoperability, Standards
METADATA ENRICHMENT
Shift from metadata for internal use to metadata for online access

Linked Data, Semantic Web, Interoperability, Standards
METADATA ENRICHMENT
Building community for shared knowledge creation, use & maintenance
http://www.getty.edu/research/tools/vocabularies/lod/index.html

Rijksmuseum
Using Linked Data to Diversify Search Results a Case Study in Cultural Heritage
Chris Dijkshoorn, Lora Aroyo, Guus Schreiber, Jan Wielemaker, and Lizzy Jongma

2005 - 2007
http://multimedian.project.cwi.nl/

http://lora-aroyo.org @laroyo http://multimedian.project.cwi.nl/
2005 - 2007

BIG DATA
Shift from single institutions to connected heritage
https://www.europeana.eu/portal/en

Europeana.eu
sharing cultural heritage for
enjoyment, education and research
In 2008 launched with 4.5 mil
digitised items & 1,000
contributing organisations
In 2018 it collaborates with
thousands of European
archives, libraries & museums
> 50 mil digitised items:
● Books
● Music
● Artworks
Thematic collections on:
● Art
● Fashion
● Music
● Photography
● World War I
https://www.europeana.eu/portal/en

ADDRESSED THE WEB ACCESS & SCALE ISSUES ...
through using automated methods to enrich & curate metadata

BUT THAT WASN’T ENOUGH FOR TRUE ENGAGEMENT
Still there is much more focus on information support
rather than interpretation support for online collections

http://lora-aroyo.org @laroyo Gravity (2013)
LOST IN CULTURAL SPACE MORE THAN EVER
The sense of disconnect was now bigger as there has never
been so much online information and so difficult to find ...

http://lora-aroyo.org @laroyo 23
… BECAUSE THERE WAS NO CONTEXT
Entities were not sufficient to endure engagement with online collections

“THE GALLERY OF CORNELIS VAN DER GEEST”
Willem van Haecht, 1628

WHAT HAPPENS IN THIS PAINTING?
Hunting, dogs, outdoors

Religious, Madonna, Madonna and Child, Quentin Metsys

Archduke Albert and Archduchess Isabella, Cornelis van der Geest

Battle scenes, warriors, soldiers

SO MANY STORIES THAT CAN BE TOLD ...

SO MANY INTERPRETATIONS ...

theory of interpretation of
information bringing people
and technology together to:
● model information
● offer engaging interaction
● support interpretation
DIGITAL HERMENEUTICS
Chiel van den Akker, Susan Legêne, Marieke van Erp, Lora Aroyo, Roxane Segers, Lourens van der Meij, Jacco van
Ossenbruggen, Guus Schreiber, Bob Wielinga, Johan Oomen, and Geertje Jacobs (2011).
Digital hermeneutics: Agora and the online understanding of cultural heritage. In Proceedings of the 3rd International Web
Science Conference (WebSci '11). ACM, New York, NY, USA

LINKING OBJECTS THROUGH EVENTS & ENTITIES
Erp, M. van; Oomen, J.; Segers, R.; Akker, C. van de; Aroyo, L.; Jacobs, G.; Legêne, S; Meij, L. van der; O ssenbruggen, J.R. van;
Schreiber, G. Automatic Heritage Metadata Enrichment with Historic Events Museums and the Web 2011
http://diveproject.beeldengeluid.nl/

Erp, M. van; Oomen, J.; Segers, R.; Akker, C. van de; Aroyo, L.; Jacobs, G.; Legêne, S; Meij, L. van der; O ssenbruggen, J.R. van;
Schreiber, G. Automatic Heritage Metadata Enrichment with Historic Events Museums and the Web 2011
ENGAGING USERS THROUGH EVENT NARRATIVES

AGORA PROJECT
Modeling Historical Events
Segers, R., Erp, M.V., Meij, L.V., Aroyo, L., Schreiber, G., Wielinga, B.F., Ossenbruggen, J.V., Oomen, J., & Jacobs, G. (2011).
Hacking History : Automatic Historical Event Extraction for Enriching Cultural Heritage Multimedia Collections. In Proc. of the
6th International Conference on Knowledge Capture (K-CAP’11)

AGORA PROJECT
Event Properties & Relations

AGORA PROJECT
Proto-narratives with Events

DIVE+
Event-centric Explorative Search
DIVE into the event-based browsing of linked historical media (2015)
V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics

DIVE+
Explorative Search
V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics:

DIVE+
Filters for Events
filter on events

DIVE+
Building Exploration Narratives
narrative

DIVE+ MEDIA SUITE
Explorative Search for Media Collections
de Boer V., Melgar L., Inel O., Ortiz C.M., Aroyo L., Oomen J. (2017)
Enriching Media Collections for Event-Based Exploration. In Proceedings of Metadata and Semantic Research (MTSR 2017),
Communications in Computer and Information Science, vol 755. Springer
http://mediasuite.clariah.nl/

Narratives in animated GIFs
Remixing Archival Stories with millenials
Inel O., Sauer, S., Aroyo L. (2018)
A Study of Narrative Creation by Means of Crowds and Niches

CrowDDriven
Engaging Audiences with Tagging & Curating
Diego Rens, Marco Schreurs, Egemen Uzunali and Youssef Azriouil. (Master Thesis)
Supervised by Lora Aroyo

Tagasauris, Inc.
DIVE Event-based Browser for TV Media Exploration
http://tagasauris.com

M.C. Escher, Day and Night , 1938
BUT THIS ONLY WORKS IF THERE ARE EVENTS ...
Event vocabularies are difficult: too many, not structured, not shared, not
standardized, lots of variations, perspectives, no agreement across communities

CROWDTRUTH.ORG
a spatial representation of meaning that harnesses disagreement
http://crowdtruth.orghttp://data.crowdtruth.org

CROWDTRUTH.ORG
a spatial representation of meaning
that harnesses disagreement
a human computation
(crowdsourcing) approach to:
● gather diversity of
perspectives & opinions
from crowds & niches
● expand expert
vocabularies with these
● gather new type of gold
standard for machines

COMFORT ZONE
50
Defending the single truth, the institutional quality validation
http://crowdtruth.org

One truth: knowledge acquisition and
curation assume one correct interpretation
for every object
All cases are created equal: they are all
either true or false
Disagreement bad: when people disagree,
they don’t understand the problem
Experts rule: knowledge is always
captured from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if cases cause
disagreement - add instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with old
a set of assumptions and rules
that we rarely question
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
COMFORT ZONE DISRUPTED

COMFORT ZONE DISRUPTED
Everything is relative, and life is full of perspectives and opinions
M.C. Escher, Relativity , 1953

On the role of user-generated metadata in audio visual collections (2011).
R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011
VIDEO METADATA ENRICHMENT
The Netherlands Institute for Sound and Vision
http://waisda.nl

On the role of user-generated metadata in audio visual collections (2011).
R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011
VIDEO METADATA ENRICHMENT
The Netherlands Institute for Sound and Vision
http://spotvogel.vroegevogels.vara.nl/

L. Aroyo, C. Welty: CrowdTruth: Harnessing disagreement in crowdsourcing relex gold standard. ACM WebSci 2013.
L. Aroyo, C. Welty. The Three Sides of CrowdTruth, Journal of Human Computation, 2014
VIDEO ENRICHMENT
CrowdTruth with Amazon Mechanical Turk & Figure Eight

L. Aroyo, C. Welty: CrowdTruth: Harnessing disagreement in crowdsourcing relex gold standard. ACM WebSci 2013.
L. Aroyo, C. Welty. The Three Sides of CrowdTruth, Journal of Human Computation, 2014
IMAGE ENRICHMENT
CrowdTruth with Amazon Mechanical Turk & Figure Eight

Nikita Galinkin, Zoltán Szlávik, Lora Aroyo and Benjamin Timmermans (2017).
Catch Them If You Can: A Simulation Study on Malicious Behavior in a Cultural Heritage
Question Answering System. The 29th Benelux Conference on Artificial Intelligence (BNAIC 2017).
IMAGE ENRICHMENT
CrowdTruth with Mauritshuis

Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014).
Accurator: Nichesourcing for Cultural Heritage
NICHESOURCING: FINDING NICHES IN THE CROWD
Accurator tool: SealincMedia Project
http://sealincmedia.wordpress.com

NICHESOURCING IN THE CULTURAL HERITAGE
Accurator tool
http://annotate.accurator.nl

CREATING EXPERTS WITH GAMES
Accurator tool

DigiBird: on the fly collection integration supported by the crowd (2017)
Chris Dijkshoorn, Christina-Lulia Bucur, Maarten Brinkerink, Sander Pieterse and Lora Aroyo
NICHESOURCING EVENTS
Part of the SealincMedia Project

DigiBird: on the fly collection integration supported by the crowd (2017)
Chris Dijkshoorn, Christina-Lulia Bucur, Maarten Brinkerink, Sander Pieterse and Lora Aroyo
NICHESOURCING EVENTS
DigiBird Project

SUCCESS STORIES: NIOD
Linked Data & Crowdsourcing for historical & personal events
https://www.oorlogsbronnen.nl/

ADDING EVENTS TO THE NOB THESAURUS

EVENTS THESAURUS

PERSONAL EVENTS

PEOPLE PORTAL

632.953 artworks - 411.745 Rijksstudios
SUCCESS STORIES: RIJKSMUSEUM
Crowdsourcing with Rijksstudio
https://www.rijksmuseum.nl/en/rijksstudio

Rijksmuseum API
https://www.rijksmuseum.nl/en/api

Creativity with Open Data

LESSONS LEARNED ...
Crowds are large and contribute at scale
Crowds bring natural diversity
Crowds help gathering real human semantics
There are niches of experts in the crowds
Experts and crowds are complimentary
together they encompass a multitude
of opinions and perspectives
Experts and crowds have different semantics
Experts and crowds are
interested in different stories
Experts and crowds use different vocabularies
Crowds are enthusiasts, motivated,
driven by altruism

The world is full
of shades of grey
Capturing and understanding opinions,
perspectives & contexts is in the center
of understanding people
LESSONS LEARNED ...
CrowdTruth defines multi-dimensional
space to measure quality
CrowdTruth defines hyper-dimensional
space to represent ambiguity
Nichesourcing helps expanding expertise
beyond the walls of organizations
Nichesourcing needs active engagement
online and with onsite campaigns

CROWDTRUTH.ORG
Not just a framework for crowdsourcing, it is a state of mind ...

Lora Aroyo
StorySourcing:
TELLING STORIES WITH HUMANS & MACHINES
User Centric Data Science Group

StorySourcing: Telling Stories with Humans & Machines

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (7)

Semelhante a StorySourcing: Telling Stories with Humans & Machines

Semelhante a StorySourcing: Telling Stories with Humans & Machines (20)

Mais de Lora Aroyo

Mais de Lora Aroyo (20)

Último

Último (20)

StorySourcing: Telling Stories with Humans & Machines