SlideShare uma empresa Scribd logo
1 de 66
http://lora-aroyo.org @laroyo
Harnessing Human Semantics at Scale
Measurable, Reproducible, Engaging, Sustainable
Crowdsourcing & Nichesourcing
Lora Aroyo
Join to participate in the CATS4ML Data Challenge
cats4ml.humancomputation.com
http://lora-aroyo.org @laroyo
20071998 2006 2009
the data science journey of an online DVD rental
http://lora-aroyo.org @laroyo
20071998 2006 2009
streaming experiments
http://lora-aroyo.org @laroyo
20071998 2006 2009
Netflix Prize
http://lora-aroyo.org @laroyo
20071998 2006 2009
Team BellKor wins Netflix Prize
http://lora-aroyo.org @laroyo
20071998 2006 2014
data science for personalization
http://lora-aroyo.org @laroyo
20061994 2003 2016 2017
the data science journey of an online bookstore
http://lora-aroyo.org @laroyo
20061994 2003 2016 2017
the data science journey of an online bookstore
http://lora-aroyo.org @laroyo
20061994 2003 2016 2017
the data science journey of an online bookstore
http://lora-aroyo.org @laroyo
20061994 2003 2016 2017
the data science journey of an online bookstore
http://lora-aroyo.org @laroyo
https://data-flair.training/blogs/data-science-use-cases/
http://lora-aroyo.org @laroyo
data is at the centre of every process
http://lora-aroyo.org @laroyo
data is essential to evolve with users
http://lora-aroyo.org @laroyo
Ceci n'est pas … la mona lisa
http://lora-aroyo.org @laroyo
Ceci n'est pas … la mona lisa
Louvre’s Mona Lisa
is only #14
http://lora-aroyo.org @laroyo
the battle of two worlds
9,6 million
Louvre
visitors 2019
14 million
website visitors
2,3 million
social media
http://lora-aroyo.org @laroyo
in the (very near) future
most visitors will be digital-born
not bound by time or location
native to new forms of co-makership
native to new media
Siebe Weide, Max Meijer and Marieke Krabshuis (2012).
Agenda 2026: Study on the Future of the Dutch Museum Sector
http://lora-aroyo.org @laroyo
variety of meanings
multitude of perspectives
abundance of sources
endless contexts
know your data
http://lora-aroyo.org @laroyo
crowdsourcing to know your data at scale
http://lora-aroyo.org @laroyo
variety of types
multitude of platforms
abundance of interactions
endless characteristics
know your crowds
http://lora-aroyo.org @laroyo
https://www.rijksmuseum.nl/en/rijksstudio
Engage with Co-creation
http://lora-aroyo.org @laroyo
Engage with Co-creativity
http://lora-aroyo.org @laroyo
Engage with Co-curation
http://lora-aroyo.org @laroyo
Engage the Expert Niche
http://annotate.accurator.nl
http://lora-aroyo.org @laroyo
expertise of Rijksmuseum professionals is
in annotating their collection
with art-historical information, e.g. when they
were created, by whom, etc.
http://lora-aroyo.org @laroyo
detailed domain-specific information
about depicted objects, e.g. which species the
animal or plant belongs to,
is in most cases not available
http://lora-aroyo.org @laroyo
use nichesourcing, i.e. niches of people with
the right expertise, to add more specific
information
http://lora-aroyo.org @laroyo
Keep Reproducing
http://annotate.accurator.nl
http://lora-aroyo.org @laroyo
Engage with Games
training the general crowd to be a niche:
game in which players can carry out an expert
annotation tasks with some assistance
http://lora-aroyo.org @laroyo
http://waisda.nl
Engage with Games
http://lora-aroyo.org @laroyo
http://waisda.nl
Engage with Games
http://lora-aroyo.org @laroyo
http://spotvogel.vroegevogels.vara.nl
Keep Reproducing
http://lora-aroyo.org @laroyo
CrowdTruth.org
Experiment with Paid Crowds
http://lora-aroyo.org @laroyo
CrowdTruth.org
Experiment with Paid Crowds
http://lora-aroyo.org @laroyo
CrowdTruth.org
Experiment with Paid Crowds
http://lora-aroyo.org @laroyo
http://crowdtruth.org/
http://lora-aroyo.org @laroyo
http://data.crowdtruth.org/
http://lora-aroyo.org @laroyo
Challenges
http://lora-aroyo.org @laroyo
Low reproducibility rates
Difficult to estimate & control the time to complete
Difficult to assess & compare quality
Demands continuous promotional effort
Active learning (human-in-the-loop) needs different expertise
Difficult to incorporate results into existing content infrastructure
Challenges
Crowdsourcing typically undertaken in isolation
http://lora-aroyo.org @laroyo
Assess Impact of Task Design
http://lora-aroyo.org @laroyo
Instructions
Layout
Sequence
Crowds
Payment
Campaign
Assess Impact of Task Design
experiment with different designs
http://lora-aroyo.org @laroyo
for example
mapping music to mood
http://lora-aroyo.org @laroyo
Choose one:
Which is the mood most appropriate
for each song?
Goal:
(Lee and Hu 2012)
1 song - 1 mood???
http://lora-aroyo.org @laroyo
If “One Truth” & “No Disagreement”
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5
W1 1
W2 1
W3 1
W4 1
W5 1
W6 1
W7
W8
W9 1
W10 1
Totals 1 3 1 2 1
http://lora-aroyo.org @laroyo
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other
W1 1 1 1
W2 1 1 1
W3 1 1 1
W4 1 1
W5 1 1
W6 1 1 1
W7 1 1 1
W8 1 1 1
W9 1 1
W10 1 1 1 1 1
Totals 3 5 6 5 2 8
If “Many Truths” & “Disagreement”
Web & Media Group
http://lora-aroyo.org @laroyo
simplification of context
this all results in
Web & Media Group
http://lora-aroyo.org @laroyo
http://lora-aroyo.org @laroyo
● Identify Crowdsourcing Goals through user log analysis
○ # queries, #unique queries, #queries of specific type
○ ranked by popularity
○ ranked by popularity and with error, e.g.
■ # queries entered over 50 times with 0 results
■ # queries of specific type with 0 results
○ which will have biggest impact
○ which has biggest urgency
● … or through other user analysis
○ museum visits, external channels
Assess Impact of Results
http://lora-aroyo.org @laroyo
for example
in video search
http://lora-aroyo.org @laroyo
people search for fragments
experts annotate full videos
35% of search queries result in not found
people search for fragments
experts annotate full videos
35% of search queries result in not found
for example
in video search
http://lora-aroyo.org @laroyo
Measure Quality
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
http://lora-aroyo.org @laroyo
Measure Quality
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
time-based annotation
bernhard
88% of the tags useful
for specific genres
describe short segments
often not very specific
don’t describe program as a whole
http://lora-aroyo.org @laroyo
for example
in video search
video annotation is time-consuming
5 times the video duration
experts use a specific vocabulary
that is unknown to general audiences
video annotation is time-consuming
5 times the video duration
experts use a specific vocabulary
that is unknown to general audiences
http://lora-aroyo.org @laroyo
user vocabulary
8% in professional vocabulary
23% in Dutch lexicon
89% found on Google
locations (7%)
engeland
persons (31%)
objects (57%)
Measure Quality
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
Web & Media Group
http://lora-aroyo.org @laroyo
human subjectivity, ambiguity & uncertainty of expression
natural part of human semantics
http://lora-aroyo.org @laroyo
measure quality
quality is not just about spam
quality is typically multi-dimensional
understand the diversity in crowd answers
do not ignore multitude of interpretations
understand the variety of contexts
identify cases with high ambiguity, similarity, …
experiment with explicit metrics
experiment with different designs
http://lora-aroyo.org @laroyo
Measure Progress
6 months 2 years
340,551 tags 36,981 tags
137.421 matches
602 items 1.782 items
555 registered players 2,017 users (taggers)
thousands of anonymous players
12,279 visits (3+ min online)
44,362 pageviews
Riste Gligorov, Michiel Hildebrand, Jacco van Ossenbruggen, Guus Schreiber, Lora Aroyo (2011).
On the role of user-generated metadata in audio visual collections. International conference
on Knowledge capture K-CAP '11, Pages 145-152
http://lora-aroyo.org @laroyo
campaign, campaign, campaign
http://lora-aroyo.org @laroyo
http://lora-aroyo.org @laroyo
http://lora-aroyo.org @laroyo
http://lora-aroyo.org @laroyo
Measurable quality
Reproducible results
Sustainable settings
Engaging interaction
Goals
http://lora-aroyo.org @laroyo
Crowdsourcing AI Blindspots
CATS4ML Data Challenge
cats4ml.humancomputation.com
ˈl ɪ k ə r t
http://lora-aroyo.org @laroyo
Your AI model is as good
as your evaluation data
… but is your evaluation
data missing relevant
examples?
… and how can we find
such examples, especially
if they are AI blindspots
(i.e. unknown unknowns)?
CATS4ML Challenge
offers a crowdsourced red
team for finding
blindspots of your AI
models
cats4ml.humancomputation.com
http://lora-aroyo.org @laroyo
AI Blindspots
real images with visual patterns that confuse AI models
in ways humans might find meaningful
Lipstick?
Airplane?
Car?
Construction worker? Thanksgiving?
https://opensource.google/projects/open-images-dataset
http://lora-aroyo.org @laroyo
Inspired by Bug Bounty
this is a data challenge to find
the blindspots in our AI models
Challenge running
until mid Jan 2021
Join, and start hunting
for AI Blindspots, and
spread the word to other
teams that might be
interested!
cats4ml.humancomputation.com
cats4ml.humancomputation.com

Mais conteúdo relacionado

Mais procurados

Ethical issues facing Artificial Intelligence
Ethical issues facing Artificial IntelligenceEthical issues facing Artificial Intelligence
Ethical issues facing Artificial IntelligenceRah Abdelhak
 
Wimmics Research Team 2015 Activity Report
Wimmics Research Team 2015 Activity ReportWimmics Research Team 2015 Activity Report
Wimmics Research Team 2015 Activity ReportFabien Gandon
 
Ethical Considerations in the Design of Artificial Intelligence
Ethical Considerations in the Design of Artificial IntelligenceEthical Considerations in the Design of Artificial Intelligence
Ethical Considerations in the Design of Artificial IntelligenceJohn C. Havens
 
Overview of the Research in Wimmics 2018
Overview of the Research in Wimmics 2018Overview of the Research in Wimmics 2018
Overview of the Research in Wimmics 2018Fabien Gandon
 
Wimmics Overview 2021
Wimmics Overview 2021Wimmics Overview 2021
Wimmics Overview 2021Fabien Gandon
 
Dynamic UXR: Ethical Responsibilities and AI. Carol Smith at Strive in Toronto
Dynamic UXR: Ethical Responsibilities and AI. Carol Smith at Strive in TorontoDynamic UXR: Ethical Responsibilities and AI. Carol Smith at Strive in Toronto
Dynamic UXR: Ethical Responsibilities and AI. Carol Smith at Strive in TorontoCarol Smith
 
Y conf talk - Andrej Karpathy
Y conf talk - Andrej KarpathyY conf talk - Andrej Karpathy
Y conf talk - Andrej KarpathySze Siong Teo
 
Demystifying Artificial Intelligence: Solving Difficult Problems at ProductCa...
Demystifying Artificial Intelligence: Solving Difficult Problems at ProductCa...Demystifying Artificial Intelligence: Solving Difficult Problems at ProductCa...
Demystifying Artificial Intelligence: Solving Difficult Problems at ProductCa...Carol Smith
 
2018 Princeton Fintech & Quant Conference: AI, Machine Learning & Deep Learni...
2018 Princeton Fintech & Quant Conference: AI, Machine Learning & Deep Learni...2018 Princeton Fintech & Quant Conference: AI, Machine Learning & Deep Learni...
2018 Princeton Fintech & Quant Conference: AI, Machine Learning & Deep Learni...Yogesh Malhotra, PhD,MSQF, CISSP,CISA,CEH
 
AI - Last Year Progress (2018-2019)
AI - Last Year Progress (2018-2019)AI - Last Year Progress (2018-2019)
AI - Last Year Progress (2018-2019)Grigory Sapunov
 
Codes of Ethics and the Ethics of Code
Codes of Ethics and the Ethics of CodeCodes of Ethics and the Ethics of Code
Codes of Ethics and the Ethics of CodeMark Underwood
 
Semantics, Deep Learning, and the Transformation of Business
Semantics, Deep Learning, and the Transformation of BusinessSemantics, Deep Learning, and the Transformation of Business
Semantics, Deep Learning, and the Transformation of BusinessSteve Omohundro
 
The Web We Mix - benevolent AIs for a resilient web
The Web We Mix - benevolent AIs for a resilient webThe Web We Mix - benevolent AIs for a resilient web
The Web We Mix - benevolent AIs for a resilient webFabien Gandon
 
Machine Learning for Designers
Machine Learning for DesignersMachine Learning for Designers
Machine Learning for DesignersMemi Beltrame
 
Machine Intelligence - Wie Systeme lernen und unseren Alltag verändern
Machine Intelligence - Wie Systeme lernen und unseren Alltag verändernMachine Intelligence - Wie Systeme lernen und unseren Alltag verändern
Machine Intelligence - Wie Systeme lernen und unseren Alltag verändernMark Cieliebak
 
Machine Learning Introduction for Digital Business Leaders
Machine Learning Introduction for Digital Business LeadersMachine Learning Introduction for Digital Business Leaders
Machine Learning Introduction for Digital Business LeadersSudha Jamthe
 
Ai ml-demystified-mwux2017-final-171016011705
Ai ml-demystified-mwux2017-final-171016011705Ai ml-demystified-mwux2017-final-171016011705
Ai ml-demystified-mwux2017-final-171016011705Sayali Surve
 

Mais procurados (20)

Ethical issues facing Artificial Intelligence
Ethical issues facing Artificial IntelligenceEthical issues facing Artificial Intelligence
Ethical issues facing Artificial Intelligence
 
Wimmics Research Team 2015 Activity Report
Wimmics Research Team 2015 Activity ReportWimmics Research Team 2015 Activity Report
Wimmics Research Team 2015 Activity Report
 
Ethical Considerations in the Design of Artificial Intelligence
Ethical Considerations in the Design of Artificial IntelligenceEthical Considerations in the Design of Artificial Intelligence
Ethical Considerations in the Design of Artificial Intelligence
 
Overview of the Research in Wimmics 2018
Overview of the Research in Wimmics 2018Overview of the Research in Wimmics 2018
Overview of the Research in Wimmics 2018
 
The Ethics of AI
The Ethics of AIThe Ethics of AI
The Ethics of AI
 
Wimmics Overview 2021
Wimmics Overview 2021Wimmics Overview 2021
Wimmics Overview 2021
 
Dynamic UXR: Ethical Responsibilities and AI. Carol Smith at Strive in Toronto
Dynamic UXR: Ethical Responsibilities and AI. Carol Smith at Strive in TorontoDynamic UXR: Ethical Responsibilities and AI. Carol Smith at Strive in Toronto
Dynamic UXR: Ethical Responsibilities and AI. Carol Smith at Strive in Toronto
 
Y conf talk - Andrej Karpathy
Y conf talk - Andrej KarpathyY conf talk - Andrej Karpathy
Y conf talk - Andrej Karpathy
 
Demystifying Artificial Intelligence: Solving Difficult Problems at ProductCa...
Demystifying Artificial Intelligence: Solving Difficult Problems at ProductCa...Demystifying Artificial Intelligence: Solving Difficult Problems at ProductCa...
Demystifying Artificial Intelligence: Solving Difficult Problems at ProductCa...
 
2018 Princeton Fintech & Quant Conference: AI, Machine Learning & Deep Learni...
2018 Princeton Fintech & Quant Conference: AI, Machine Learning & Deep Learni...2018 Princeton Fintech & Quant Conference: AI, Machine Learning & Deep Learni...
2018 Princeton Fintech & Quant Conference: AI, Machine Learning & Deep Learni...
 
AI - Last Year Progress (2018-2019)
AI - Last Year Progress (2018-2019)AI - Last Year Progress (2018-2019)
AI - Last Year Progress (2018-2019)
 
Demystifying AI
Demystifying AIDemystifying AI
Demystifying AI
 
Codes of Ethics and the Ethics of Code
Codes of Ethics and the Ethics of CodeCodes of Ethics and the Ethics of Code
Codes of Ethics and the Ethics of Code
 
Semantics, Deep Learning, and the Transformation of Business
Semantics, Deep Learning, and the Transformation of BusinessSemantics, Deep Learning, and the Transformation of Business
Semantics, Deep Learning, and the Transformation of Business
 
The Web We Mix - benevolent AIs for a resilient web
The Web We Mix - benevolent AIs for a resilient webThe Web We Mix - benevolent AIs for a resilient web
The Web We Mix - benevolent AIs for a resilient web
 
Machine Learning for Designers
Machine Learning for DesignersMachine Learning for Designers
Machine Learning for Designers
 
Machine Intelligence - Wie Systeme lernen und unseren Alltag verändern
Machine Intelligence - Wie Systeme lernen und unseren Alltag verändernMachine Intelligence - Wie Systeme lernen und unseren Alltag verändern
Machine Intelligence - Wie Systeme lernen und unseren Alltag verändern
 
From Big Data to AI
From Big Data to AIFrom Big Data to AI
From Big Data to AI
 
Machine Learning Introduction for Digital Business Leaders
Machine Learning Introduction for Digital Business LeadersMachine Learning Introduction for Digital Business Leaders
Machine Learning Introduction for Digital Business Leaders
 
Ai ml-demystified-mwux2017-final-171016011705
Ai ml-demystified-mwux2017-final-171016011705Ai ml-demystified-mwux2017-final-171016011705
Ai ml-demystified-mwux2017-final-171016011705
 

Semelhante a Harnessing Human Semantics at Scale (updated)

Digital Humanities Benelux 2017: Keynote Lora Aroyo
Digital Humanities Benelux 2017: Keynote Lora AroyoDigital Humanities Benelux 2017: Keynote Lora Aroyo
Digital Humanities Benelux 2017: Keynote Lora AroyoLora Aroyo
 
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital AgeEuropeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital AgeLora Aroyo
 
Data Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityData Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityLora Aroyo
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumLora Aroyo
 
Semantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordLora Aroyo
 
CrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsCrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsLora Aroyo
 
CrowdTruth for User-Centric Relevance
CrowdTruth for User-Centric RelevanceCrowdTruth for User-Centric Relevance
CrowdTruth for User-Centric RelevanceLora Aroyo
 
CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015Lora Aroyo
 
Modelling Users’ Profiles and Interests based on Cross-Folksonomy Analysis ...
Modelling Users’ Profiles and  Interests based on  Cross-Folksonomy Analysis ...Modelling Users’ Profiles and  Interests based on  Cross-Folksonomy Analysis ...
Modelling Users’ Profiles and Interests based on Cross-Folksonomy Analysis ...Martin Szomszor
 
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...Lora Aroyo
 
Content and seo for Tulos by Gianluca Fiorelli
Content and seo for Tulos by Gianluca FiorelliContent and seo for Tulos by Gianluca Fiorelli
Content and seo for Tulos by Gianluca FiorelliTulos Helsinki Oy
 
Closing Event - Watson Innovation Course
Closing Event - Watson Innovation CourseClosing Event - Watson Innovation Course
Closing Event - Watson Innovation CourseLora Aroyo
 
Gaming with Open Source Frameworks
Gaming with Open Source FrameworksGaming with Open Source Frameworks
Gaming with Open Source FrameworksAlmog Koren
 
"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to Snapchat"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to SnapchatLora Aroyo
 
Open, Connected & Smart Heritage: Towards New Cultural Commons
Open, Connected & Smart Heritage: Towards New Cultural CommonsOpen, Connected & Smart Heritage: Towards New Cultural Commons
Open, Connected & Smart Heritage: Towards New Cultural CommonsLora Aroyo
 
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneMy ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneLora Aroyo
 
Gaming: the future’s challenge; Interactive workshop at ISAGA conference; Dor...
Gaming: the future’s challenge; Interactive workshop at ISAGA conference; Dor...Gaming: the future’s challenge; Interactive workshop at ISAGA conference; Dor...
Gaming: the future’s challenge; Interactive workshop at ISAGA conference; Dor...Pieter van der Hijden
 
Interactive workshop: Gaming: the future's challenge; Pieter van der Hijden; ...
Interactive workshop: Gaming: the future's challenge; Pieter van der Hijden; ...Interactive workshop: Gaming: the future's challenge; Pieter van der Hijden; ...
Interactive workshop: Gaming: the future's challenge; Pieter van der Hijden; ...Pieter van der Hijden
 
Tom Critchlow - Data Feed SEO & Advanced Site Architecture
Tom Critchlow - Data Feed SEO & Advanced Site ArchitectureTom Critchlow - Data Feed SEO & Advanced Site Architecture
Tom Critchlow - Data Feed SEO & Advanced Site Architectureauexpo Conference
 
Gamify your classroom!
Gamify your classroom!Gamify your classroom!
Gamify your classroom!Graham Stanley
 

Semelhante a Harnessing Human Semantics at Scale (updated) (20)

Digital Humanities Benelux 2017: Keynote Lora Aroyo
Digital Humanities Benelux 2017: Keynote Lora AroyoDigital Humanities Benelux 2017: Keynote Lora Aroyo
Digital Humanities Benelux 2017: Keynote Lora Aroyo
 
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital AgeEuropeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
 
Data Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityData Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden University
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
 
Semantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @Oxford
 
CrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsCrowdTruth for Digital Hermeneutics
CrowdTruth for Digital Hermeneutics
 
CrowdTruth for User-Centric Relevance
CrowdTruth for User-Centric RelevanceCrowdTruth for User-Centric Relevance
CrowdTruth for User-Centric Relevance
 
CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015
 
Modelling Users’ Profiles and Interests based on Cross-Folksonomy Analysis ...
Modelling Users’ Profiles and  Interests based on  Cross-Folksonomy Analysis ...Modelling Users’ Profiles and  Interests based on  Cross-Folksonomy Analysis ...
Modelling Users’ Profiles and Interests based on Cross-Folksonomy Analysis ...
 
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
 
Content and seo for Tulos by Gianluca Fiorelli
Content and seo for Tulos by Gianluca FiorelliContent and seo for Tulos by Gianluca Fiorelli
Content and seo for Tulos by Gianluca Fiorelli
 
Closing Event - Watson Innovation Course
Closing Event - Watson Innovation CourseClosing Event - Watson Innovation Course
Closing Event - Watson Innovation Course
 
Gaming with Open Source Frameworks
Gaming with Open Source FrameworksGaming with Open Source Frameworks
Gaming with Open Source Frameworks
 
"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to Snapchat"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to Snapchat
 
Open, Connected & Smart Heritage: Towards New Cultural Commons
Open, Connected & Smart Heritage: Towards New Cultural CommonsOpen, Connected & Smart Heritage: Towards New Cultural Commons
Open, Connected & Smart Heritage: Towards New Cultural Commons
 
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneMy ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
 
Gaming: the future’s challenge; Interactive workshop at ISAGA conference; Dor...
Gaming: the future’s challenge; Interactive workshop at ISAGA conference; Dor...Gaming: the future’s challenge; Interactive workshop at ISAGA conference; Dor...
Gaming: the future’s challenge; Interactive workshop at ISAGA conference; Dor...
 
Interactive workshop: Gaming: the future's challenge; Pieter van der Hijden; ...
Interactive workshop: Gaming: the future's challenge; Pieter van der Hijden; ...Interactive workshop: Gaming: the future's challenge; Pieter van der Hijden; ...
Interactive workshop: Gaming: the future's challenge; Pieter van der Hijden; ...
 
Tom Critchlow - Data Feed SEO & Advanced Site Architecture
Tom Critchlow - Data Feed SEO & Advanced Site ArchitectureTom Critchlow - Data Feed SEO & Advanced Site Architecture
Tom Critchlow - Data Feed SEO & Advanced Site Architecture
 
Gamify your classroom!
Gamify your classroom!Gamify your classroom!
Gamify your classroom!
 

Mais de Lora Aroyo

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfLora Aroyo
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumLora Aroyo
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorLora Aroyo
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataLora Aroyo
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18Lora Aroyo
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsLora Aroyo
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesLora Aroyo
 
Data Science with Humans in the Loop
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the LoopLora Aroyo
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...Lora Aroyo
 
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Lora Aroyo
 
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchLora Aroyo
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyLora Aroyo
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumLora Aroyo
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Lora Aroyo
 
Keynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventKeynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventLora Aroyo
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+Lora Aroyo
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...Lora Aroyo
 
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015Lora Aroyo
 

Mais de Lora Aroyo (18)

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH Symposium
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP Demonstrator
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked Data
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithms
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & Machines
 
Data Science with Humans in the Loop
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the Loop
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
 
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
 
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening Ceremony
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
 
Keynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventKeynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program Event
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
 
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
 

Último

IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTxtailishbaloch
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 

Último (20)

IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 

Harnessing Human Semantics at Scale (updated)