SlideShare uma empresa Scribd logo
1 de 22
Baixar para ler offline
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
Focused Crawl of Web Archives
to Build Event Collections
Martin Klein
Lyudmila Balakireva
Herbert Van de Sompel
Research Library
Los Alamos National Laboratory
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
2
• Often orchestrated by subject matter experts, archivists,
special collection librarians, technicians
• Potentially with guidance from institutional collection policy
• Results in a list of seeds (URIs, social media accounts, etc)
• Utilization of crawling services such as Archive-It, Social Feed
Manager
Background – Event Collection Building
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
3
• Temporal: time passed since event is of concern
 Use of web archives
• Selection: seeds often picked manually
 Use of references from Wikipedia pages
• Relevance: seed assessment often done by humans
 Use of focused crawling with content and temporal
relevance assessment
Inspiration from:
“Extracting Event-Centric Document Collections from Large-Scale Web Archives”
Gerhard Gossen, Elena Demidova, Thomas Risse
https://doi.org/10.1007/978-3-319-67008-9_10
Problems and our Approach
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
4
• Web archives are an invaluable resource for researchers,
historians, journalists, etc.
• Often broad in scope, large in scale, covering different
temporal intervals
• Makes discovery, access, and analysis difficult
Background – Archived Web
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
5
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
6
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
7
• Can we create event collections by focused crawling online-
available web archives?
• How do event collections created from the archived web
compare to those created from the live web?
• How does the amount of time passed since the event affect
the collections built from the live and the archived web?
• How do event collections built from the archived web compare
to manually curated collections?
Questions
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
8
• Topics limited to terror attacks and mass shootings in the U.S.
• From different times in the past
• Focused crawl of:
a) 22 archives, simultaneously, via Memento infrastructure
b) the live web
• Take content and temporal relevance into account, equally
weighted
• Use events’ Wikipedia page as input for focused crawler
Experiment
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
9
1. Content of Wikipedia page + random 60% of page’s references
• Generate topic vector (TF-IDF of 1grams + 2grams)
2. Content of remaining 40% of Wikipedia page’s outlinks
• Generate topic vector (TF-IDF of 1grams + 2grams)
• Compute cosine similarity value between vectors 1 and 2
• Run 10 times
• Take average similarity value as content threshold
Content Relevance
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
10
• Define temporal interval for which crawled pages are
considered relevant
• Event date extracted from Wikipedia event page
• Change point determined from graph of proportional
Wikipedia page edits per day
Temporal Relevance
1
Event Date Change Point Today
0 0
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
11
• Extract datetime from pages via:
• URI
http://www.cnn.com/2017/12/09/us/wildfire-fighting-tactics/
• Meta tags
<meta property="article:published" itemprop="datePublished"
content="2017-12-09T10:14:50-05:00" />
• ODU’s Carbondate tool
http://carbondate.cs.odu.edu/
• Memento datetime
• X-Header
Datetime Extraction
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
12
• Use version of Wikipedia page that was live at change point
• Crawl stop conditions:
• No more relevant documents left
• 5 levels deep
• Utilized crawl priority queue
Crawls
Level 2
Level 1
Level 0
Child 1
Seed
Child 2 Child 3
Child 3.2Child 3.1Child 2.1Child 1.1 Child 3.2Child 1.2
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
13
• New York City, October 31st 2017
• San Bernadino, December 2nd 2015
• Tucson, January 8th 2011
• Binghampton, April 3rd 2009
Collections Crawled (in November 2017)
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
14
NYC, 10/31/2017 – URIs per Level
0 1 2 3 4 5
Crawl depth
NumberofURIs
0500100015002000
Web Archive Crawl
0102030405060708090100
All URIs
Relevant URIs
0 1 2 3 4 5
Crawl depth
0500100015002000
Live Web Crawl
0102030405060708090100
Percent
All URIs
Relevant URIs
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
15
TUC, 01/08/2011 – URIs per Level
0 1 2 3 4 5
Crawl depth
NumberofURIs
020000400006000080000
Web Archive Crawl
0102030405060708090100
All URIs
Relevant URIs
0 1 2 3 4 5
Crawl depth
020000400006000080000
Live Web Crawl
0102030405060708090100
Percent
All URIs
Relevant URIs
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
16
NYC, 10/31/2017 – Relevance over…
Crawled Documents Crawl Time
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
17
TUC, 01/08/2011 – Relevance over…
Crawled Documents Crawl Time
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
18
TUC, 01/08/2011 – Comparison to Archive-IT
0 5000 10000 15000
050001000015000
Documents
AccumulatedRelevance
Web Archive Crawl
Archive−It Crawl
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
19
TUC, 01/08/2011 – Web Archive Contributions
web.archive.org 75%
wayback.archive−it.org
14%
webarchive.loc.gov 7%
web.archive.bibalex.org 2%
archive.is 2%
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
20
• Web archives are great resources to build event collections of
web resources
• Crawling web archives is much slower than the live web
• Collections about very recent events benefit more from the
live web than the archived web
but
• Collections about events from the distant past benefit more
from the archived web than the live web
• Utilizing multiple web archives is beneficial for the collection
• Focused crawls have the potential to outperform manual
collection building
Takeaways
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
21
https://web.archive.org/web/20171206181955/https:/twitter.com/TVNewsArchive/status/938466726190096384
Focused Crawl of Web Archives to Build Event Collections
@mart1nkle1n
WebSci 2018, 05/30/2018, Amsterdam, NL
Focused Crawl of Web Archives
to Build Event Collections
Martin Klein
Lyudmila Balakireva
Herbert Van de Sompel
Research Library
Los Alamos National Laboratory

Mais conteúdo relacionado

Mais procurados

Linked Data for Law Libraries: An Introduction
Linked Data for Law Libraries: An IntroductionLinked Data for Law Libraries: An Introduction
Linked Data for Law Libraries: An IntroductionEmily Nimsakont
 
2015-03-18 research seminar part 1
2015-03-18 research seminar part 12015-03-18 research seminar part 1
2015-03-18 research seminar part 1ifi8106tlu
 
EUNIS 2018 - Migration of a web service back-end from a relational to a docum...
EUNIS 2018 - Migration of a web service back-end from a relational to a docum...EUNIS 2018 - Migration of a web service back-end from a relational to a docum...
EUNIS 2018 - Migration of a web service back-end from a relational to a docum...Marius Politze
 
Zeng marcia ifla-subjectaccesssmartdatadh
Zeng marcia ifla-subjectaccesssmartdatadhZeng marcia ifla-subjectaccesssmartdatadh
Zeng marcia ifla-subjectaccesssmartdatadhMarcia Zeng
 
SSHA 2019: Reconstructring a country
SSHA 2019: Reconstructring a countrySSHA 2019: Reconstructring a country
SSHA 2019: Reconstructring a countryRick Mourits
 
Life after MARC: Cataloging Tools of the Future
Life after MARC: Cataloging Tools of the FutureLife after MARC: Cataloging Tools of the Future
Life after MARC: Cataloging Tools of the FutureEmily Nimsakont
 
Doctoral open day_digital_research_session_Social_Sciences_BL
Doctoral open day_digital_research_session_Social_Sciences_BLDoctoral open day_digital_research_session_Social_Sciences_BL
Doctoral open day_digital_research_session_Social_Sciences_BLAquiles Alencar Brayner
 
Walk Before You Run: Prerequisites to Linked Data
Walk Before You Run: Prerequisites to Linked DataWalk Before You Run: Prerequisites to Linked Data
Walk Before You Run: Prerequisites to Linked DataKenning Arlitsch
 
OpenGLAM CH Hackathons
OpenGLAM CH HackathonsOpenGLAM CH Hackathons
OpenGLAM CH HackathonsBeat Estermann
 
Presenting Your Digital Research
Presenting Your Digital ResearchPresenting Your Digital Research
Presenting Your Digital ResearchShawn Day
 
Introducing linked data into BBC News online
Introducing linked data into BBC News onlineIntroducing linked data into BBC News online
Introducing linked data into BBC News onlineJeremy Tarling
 
A Comparative Kalendar - DH2013 Presentation
A Comparative Kalendar - DH2013 PresentationA Comparative Kalendar - DH2013 Presentation
A Comparative Kalendar - DH2013 Presentationblalbritton
 
Free UKSG webinar: Exploring how emerging open science services can enhance i...
Free UKSG webinar: Exploring how emerging open science services can enhance i...Free UKSG webinar: Exploring how emerging open science services can enhance i...
Free UKSG webinar: Exploring how emerging open science services can enhance i...UKSG: connecting the knowledge community
 

Mais procurados (14)

Linked Data for Law Libraries: An Introduction
Linked Data for Law Libraries: An IntroductionLinked Data for Law Libraries: An Introduction
Linked Data for Law Libraries: An Introduction
 
2015-03-18 research seminar part 1
2015-03-18 research seminar part 12015-03-18 research seminar part 1
2015-03-18 research seminar part 1
 
Esshc presentation ashkan
Esshc presentation ashkanEsshc presentation ashkan
Esshc presentation ashkan
 
EUNIS 2018 - Migration of a web service back-end from a relational to a docum...
EUNIS 2018 - Migration of a web service back-end from a relational to a docum...EUNIS 2018 - Migration of a web service back-end from a relational to a docum...
EUNIS 2018 - Migration of a web service back-end from a relational to a docum...
 
Zeng marcia ifla-subjectaccesssmartdatadh
Zeng marcia ifla-subjectaccesssmartdatadhZeng marcia ifla-subjectaccesssmartdatadh
Zeng marcia ifla-subjectaccesssmartdatadh
 
SSHA 2019: Reconstructring a country
SSHA 2019: Reconstructring a countrySSHA 2019: Reconstructring a country
SSHA 2019: Reconstructring a country
 
Life after MARC: Cataloging Tools of the Future
Life after MARC: Cataloging Tools of the FutureLife after MARC: Cataloging Tools of the Future
Life after MARC: Cataloging Tools of the Future
 
Doctoral open day_digital_research_session_Social_Sciences_BL
Doctoral open day_digital_research_session_Social_Sciences_BLDoctoral open day_digital_research_session_Social_Sciences_BL
Doctoral open day_digital_research_session_Social_Sciences_BL
 
Walk Before You Run: Prerequisites to Linked Data
Walk Before You Run: Prerequisites to Linked DataWalk Before You Run: Prerequisites to Linked Data
Walk Before You Run: Prerequisites to Linked Data
 
OpenGLAM CH Hackathons
OpenGLAM CH HackathonsOpenGLAM CH Hackathons
OpenGLAM CH Hackathons
 
Presenting Your Digital Research
Presenting Your Digital ResearchPresenting Your Digital Research
Presenting Your Digital Research
 
Introducing linked data into BBC News online
Introducing linked data into BBC News onlineIntroducing linked data into BBC News online
Introducing linked data into BBC News online
 
A Comparative Kalendar - DH2013 Presentation
A Comparative Kalendar - DH2013 PresentationA Comparative Kalendar - DH2013 Presentation
A Comparative Kalendar - DH2013 Presentation
 
Free UKSG webinar: Exploring how emerging open science services can enhance i...
Free UKSG webinar: Exploring how emerging open science services can enhance i...Free UKSG webinar: Exploring how emerging open science services can enhance i...
Free UKSG webinar: Exploring how emerging open science services can enhance i...
 

Semelhante a Focused Crawl of Web Archives to Build Event Collections

Creating Topical Collections: Web Archives vs. Live Web
Creating Topical Collections:Web Archives vs. Live WebCreating Topical Collections:Web Archives vs. Live Web
Creating Topical Collections: Web Archives vs. Live WebMartin Klein
 
Information sharing about Columbia University Library’s recent web archiving ...
Information sharing about Columbia University Library’s recent web archiving ...Information sharing about Columbia University Library’s recent web archiving ...
Information sharing about Columbia University Library’s recent web archiving ...Anna Perricci
 
2015 04-21-eexcess emtacl
2015 04-21-eexcess emtacl2015 04-21-eexcess emtacl
2015 04-21-eexcess emtaclTamara Pianos
 
SAFETY NETS: RESCUE AND REVIVAL FOR ENDANGERED BORN-DIGITAL RECORDS- Program ...
SAFETY NETS: RESCUE AND REVIVAL FOR ENDANGERED BORN-DIGITAL RECORDS- Program ...SAFETY NETS: RESCUE AND REVIVAL FOR ENDANGERED BORN-DIGITAL RECORDS- Program ...
SAFETY NETS: RESCUE AND REVIVAL FOR ENDANGERED BORN-DIGITAL RECORDS- Program ...Micah Altman
 
Collaboration and Cash: Web Archiving Incentive Awards
Collaboration and Cash: Web Archiving Incentive AwardsCollaboration and Cash: Web Archiving Incentive Awards
Collaboration and Cash: Web Archiving Incentive AwardsAnna Perricci
 
ResourceSync - Overview and Real-World Use Cases for Discovery, Harvesting, a...
ResourceSync - Overview and Real-World Use Cases for Discovery, Harvesting, a...ResourceSync - Overview and Real-World Use Cases for Discovery, Harvesting, a...
ResourceSync - Overview and Real-World Use Cases for Discovery, Harvesting, a...Martin Klein
 
Resource sync overview and real-world use cases for discovery, harvesting, an...
Resource sync overview and real-world use cases for discovery, harvesting, an...Resource sync overview and real-world use cases for discovery, harvesting, an...
Resource sync overview and real-world use cases for discovery, harvesting, an...openminted_eu
 
Investigating the PROMISE of a Belgian web archive
Investigating the PROMISE of a Belgian web archive Investigating the PROMISE of a Belgian web archive
Investigating the PROMISE of a Belgian web archive Sally Chambers
 
Session 1.4 a distributed network of heritage information
Session 1.4   a distributed network of heritage informationSession 1.4   a distributed network of heritage information
Session 1.4 a distributed network of heritage informationsemanticsconference
 
A distributed network of digital heritage information - Semantics Amsterdam
A distributed network of digital heritage information - Semantics AmsterdamA distributed network of digital heritage information - Semantics Amsterdam
A distributed network of digital heritage information - Semantics AmsterdamEnno Meijers
 
Resources in uct libraries acc_hons_shellyh_2017
Resources in uct libraries acc_hons_shellyh_2017Resources in uct libraries acc_hons_shellyh_2017
Resources in uct libraries acc_hons_shellyh_2017Susanne Noll
 
WS-DL’s Work towards Enabling Personal Use of Web Archives
WS-DL’s Work towards Enabling Personal Use of Web ArchivesWS-DL’s Work towards Enabling Personal Use of Web Archives
WS-DL’s Work towards Enabling Personal Use of Web ArchivesMichele Weigle
 
WebART: Facilitating Scholarly Use of Web Archives (IIPC, Apr. 2013)
WebART: Facilitating Scholarly Use of Web Archives (IIPC, Apr. 2013)WebART: Facilitating Scholarly Use of Web Archives (IIPC, Apr. 2013)
WebART: Facilitating Scholarly Use of Web Archives (IIPC, Apr. 2013)TimelessFuture
 
OCLC Research Update at ALA Chicago. June 26, 2017.
OCLC Research Update at ALA Chicago. June 26, 2017.OCLC Research Update at ALA Chicago. June 26, 2017.
OCLC Research Update at ALA Chicago. June 26, 2017.OCLC
 
Estermann Wikidata GLAM Example Projects 20170914
Estermann Wikidata GLAM Example Projects 20170914Estermann Wikidata GLAM Example Projects 20170914
Estermann Wikidata GLAM Example Projects 20170914Beat Estermann
 
Making the Black Hole Gray: Implementing the Web Archiving of Specialist Art ...
Making the Black Hole Gray: Implementing the Web Archiving of Specialist Art ...Making the Black Hole Gray: Implementing the Web Archiving of Specialist Art ...
Making the Black Hole Gray: Implementing the Web Archiving of Specialist Art ...The Frick Collection
 
Uk discovery-jisc-project-showcase
Uk discovery-jisc-project-showcaseUk discovery-jisc-project-showcase
Uk discovery-jisc-project-showcaseRDTF-Discovery
 
Digital collections: Increasing awareness and use
Digital collections:  Increasing awareness and useDigital collections:  Increasing awareness and use
Digital collections: Increasing awareness and useButtes
 
Towards a Repository for Dutch Development Organizations
Towards a Repository for Dutch Development OrganizationsTowards a Repository for Dutch Development Organizations
Towards a Repository for Dutch Development OrganizationsIAALD Community
 
Toward complex e service for management of reserach outcomes, poland
Toward complex e service for management of reserach outcomes, polandToward complex e service for management of reserach outcomes, poland
Toward complex e service for management of reserach outcomes, polandSistemaBibliotecarioSapienza
 

Semelhante a Focused Crawl of Web Archives to Build Event Collections (20)

Creating Topical Collections: Web Archives vs. Live Web
Creating Topical Collections:Web Archives vs. Live WebCreating Topical Collections:Web Archives vs. Live Web
Creating Topical Collections: Web Archives vs. Live Web
 
Information sharing about Columbia University Library’s recent web archiving ...
Information sharing about Columbia University Library’s recent web archiving ...Information sharing about Columbia University Library’s recent web archiving ...
Information sharing about Columbia University Library’s recent web archiving ...
 
2015 04-21-eexcess emtacl
2015 04-21-eexcess emtacl2015 04-21-eexcess emtacl
2015 04-21-eexcess emtacl
 
SAFETY NETS: RESCUE AND REVIVAL FOR ENDANGERED BORN-DIGITAL RECORDS- Program ...
SAFETY NETS: RESCUE AND REVIVAL FOR ENDANGERED BORN-DIGITAL RECORDS- Program ...SAFETY NETS: RESCUE AND REVIVAL FOR ENDANGERED BORN-DIGITAL RECORDS- Program ...
SAFETY NETS: RESCUE AND REVIVAL FOR ENDANGERED BORN-DIGITAL RECORDS- Program ...
 
Collaboration and Cash: Web Archiving Incentive Awards
Collaboration and Cash: Web Archiving Incentive AwardsCollaboration and Cash: Web Archiving Incentive Awards
Collaboration and Cash: Web Archiving Incentive Awards
 
ResourceSync - Overview and Real-World Use Cases for Discovery, Harvesting, a...
ResourceSync - Overview and Real-World Use Cases for Discovery, Harvesting, a...ResourceSync - Overview and Real-World Use Cases for Discovery, Harvesting, a...
ResourceSync - Overview and Real-World Use Cases for Discovery, Harvesting, a...
 
Resource sync overview and real-world use cases for discovery, harvesting, an...
Resource sync overview and real-world use cases for discovery, harvesting, an...Resource sync overview and real-world use cases for discovery, harvesting, an...
Resource sync overview and real-world use cases for discovery, harvesting, an...
 
Investigating the PROMISE of a Belgian web archive
Investigating the PROMISE of a Belgian web archive Investigating the PROMISE of a Belgian web archive
Investigating the PROMISE of a Belgian web archive
 
Session 1.4 a distributed network of heritage information
Session 1.4   a distributed network of heritage informationSession 1.4   a distributed network of heritage information
Session 1.4 a distributed network of heritage information
 
A distributed network of digital heritage information - Semantics Amsterdam
A distributed network of digital heritage information - Semantics AmsterdamA distributed network of digital heritage information - Semantics Amsterdam
A distributed network of digital heritage information - Semantics Amsterdam
 
Resources in uct libraries acc_hons_shellyh_2017
Resources in uct libraries acc_hons_shellyh_2017Resources in uct libraries acc_hons_shellyh_2017
Resources in uct libraries acc_hons_shellyh_2017
 
WS-DL’s Work towards Enabling Personal Use of Web Archives
WS-DL’s Work towards Enabling Personal Use of Web ArchivesWS-DL’s Work towards Enabling Personal Use of Web Archives
WS-DL’s Work towards Enabling Personal Use of Web Archives
 
WebART: Facilitating Scholarly Use of Web Archives (IIPC, Apr. 2013)
WebART: Facilitating Scholarly Use of Web Archives (IIPC, Apr. 2013)WebART: Facilitating Scholarly Use of Web Archives (IIPC, Apr. 2013)
WebART: Facilitating Scholarly Use of Web Archives (IIPC, Apr. 2013)
 
OCLC Research Update at ALA Chicago. June 26, 2017.
OCLC Research Update at ALA Chicago. June 26, 2017.OCLC Research Update at ALA Chicago. June 26, 2017.
OCLC Research Update at ALA Chicago. June 26, 2017.
 
Estermann Wikidata GLAM Example Projects 20170914
Estermann Wikidata GLAM Example Projects 20170914Estermann Wikidata GLAM Example Projects 20170914
Estermann Wikidata GLAM Example Projects 20170914
 
Making the Black Hole Gray: Implementing the Web Archiving of Specialist Art ...
Making the Black Hole Gray: Implementing the Web Archiving of Specialist Art ...Making the Black Hole Gray: Implementing the Web Archiving of Specialist Art ...
Making the Black Hole Gray: Implementing the Web Archiving of Specialist Art ...
 
Uk discovery-jisc-project-showcase
Uk discovery-jisc-project-showcaseUk discovery-jisc-project-showcase
Uk discovery-jisc-project-showcase
 
Digital collections: Increasing awareness and use
Digital collections:  Increasing awareness and useDigital collections:  Increasing awareness and use
Digital collections: Increasing awareness and use
 
Towards a Repository for Dutch Development Organizations
Towards a Repository for Dutch Development OrganizationsTowards a Repository for Dutch Development Organizations
Towards a Repository for Dutch Development Organizations
 
Toward complex e service for management of reserach outcomes, poland
Toward complex e service for management of reserach outcomes, polandToward complex e service for management of reserach outcomes, poland
Toward complex e service for management of reserach outcomes, poland
 

Mais de Martin Klein

On the Persistence of Persistent Identifiers of the Scholarly Web
On the Persistence of Persistent Identifiers of the Scholarly WebOn the Persistence of Persistent Identifiers of the Scholarly Web
On the Persistence of Persistent Identifiers of the Scholarly WebMartin Klein
 
On the Persistence of Persistent Identifiers of the Scholarly Web
 On the Persistence of Persistent Identifiers of the Scholarly Web On the Persistence of Persistent Identifiers of the Scholarly Web
On the Persistence of Persistent Identifiers of the Scholarly WebMartin Klein
 
An Institutional Perspective to Rescue Scholarly Orphans
An Institutional Perspective to Rescue Scholarly OrphansAn Institutional Perspective to Rescue Scholarly Orphans
An Institutional Perspective to Rescue Scholarly OrphansMartin Klein
 
Who is Asking - Humans and Machines Experience a Different Scholarly Web
Who is Asking - Humans and Machines  Experience a Different Scholarly WebWho is Asking - Humans and Machines  Experience a Different Scholarly Web
Who is Asking - Humans and Machines Experience a Different Scholarly WebMartin Klein
 
The Memento Tracer Framework: Balancing Quality and Scalability for Web Arch...
The Memento Tracer Framework: Balancing Quality and Scalability  for Web Arch...The Memento Tracer Framework: Balancing Quality and Scalability  for Web Arch...
The Memento Tracer Framework: Balancing Quality and Scalability for Web Arch...Martin Klein
 
Memento Tracer An Innovative Approach Towards Balancing Scale and Fidelity f...
Memento Tracer An Innovative Approach Towards Balancing  Scale and Fidelity f...Memento Tracer An Innovative Approach Towards Balancing  Scale and Fidelity f...
Memento Tracer An Innovative Approach Towards Balancing Scale and Fidelity f...Martin Klein
 
Comparing the Performance of OAI-PMH with ResourceSync
Comparing the Performance of OAI-PMH with ResourceSyncComparing the Performance of OAI-PMH with ResourceSync
Comparing the Performance of OAI-PMH with ResourceSyncMartin Klein
 
Evaluating Memento Service Optimizations
Evaluating Memento Service OptimizationsEvaluating Memento Service Optimizations
Evaluating Memento Service OptimizationsMartin Klein
 
An Institutional Perspective to Rescue Scholarly Orphans
An Institutional Perspective to Rescue Scholarly OrphansAn Institutional Perspective to Rescue Scholarly Orphans
An Institutional Perspective to Rescue Scholarly OrphansMartin Klein
 
A Vision of the Library’s Role in Archiving Scholarly Artifacts
A Vision of the Library’s Role  in Archiving Scholarly ArtifactsA Vision of the Library’s Role  in Archiving Scholarly Artifacts
A Vision of the Library’s Role in Archiving Scholarly ArtifactsMartin Klein
 
First Steps in Research Data Management Under Constraints of a National Secur...
First Steps in Research Data Management Under Constraints of a National Secur...First Steps in Research Data Management Under Constraints of a National Secur...
First Steps in Research Data Management Under Constraints of a National Secur...Martin Klein
 
Smart Routing of Memento Requests
Smart Routing of Memento RequestsSmart Routing of Memento Requests
Smart Routing of Memento RequestsMartin Klein
 
A Web-Centric Pipeline for Archiving Scholarly Artifacts
A Web-Centric Pipeline for Archiving Scholarly ArtifactsA Web-Centric Pipeline for Archiving Scholarly Artifacts
A Web-Centric Pipeline for Archiving Scholarly ArtifactsMartin Klein
 
Robust Linking to Web Resources
Robust Linking to Web ResourcesRobust Linking to Web Resources
Robust Linking to Web ResourcesMartin Klein
 
Signposting for Repositories
Signposting for RepositoriesSignposting for Repositories
Signposting for RepositoriesMartin Klein
 
Discovering Scholarly Orphans Using ORCID
Discovering Scholarly Orphans Using ORCIDDiscovering Scholarly Orphans Using ORCID
Discovering Scholarly Orphans Using ORCIDMartin Klein
 
Using the Memento Framework to Assess Content Drift in Scholarly Communication
Using the Memento Framework to Assess Content Drift in Scholarly CommunicationUsing the Memento Framework to Assess Content Drift in Scholarly Communication
Using the Memento Framework to Assess Content Drift in Scholarly CommunicationMartin Klein
 
Uniform Access to Raw Mementos
Uniform Access to Raw MementosUniform Access to Raw Mementos
Uniform Access to Raw MementosMartin Klein
 
Robust Links - a proposed solution to reference rot in scholarly communication
Robust Links - a proposed solution to reference rot in scholarly communicationRobust Links - a proposed solution to reference rot in scholarly communication
Robust Links - a proposed solution to reference rot in scholarly communicationMartin Klein
 
To the Rescue of the Orphans of Scholarly Communication
To the Rescue of the Orphans of Scholarly CommunicationTo the Rescue of the Orphans of Scholarly Communication
To the Rescue of the Orphans of Scholarly CommunicationMartin Klein
 

Mais de Martin Klein (20)

On the Persistence of Persistent Identifiers of the Scholarly Web
On the Persistence of Persistent Identifiers of the Scholarly WebOn the Persistence of Persistent Identifiers of the Scholarly Web
On the Persistence of Persistent Identifiers of the Scholarly Web
 
On the Persistence of Persistent Identifiers of the Scholarly Web
 On the Persistence of Persistent Identifiers of the Scholarly Web On the Persistence of Persistent Identifiers of the Scholarly Web
On the Persistence of Persistent Identifiers of the Scholarly Web
 
An Institutional Perspective to Rescue Scholarly Orphans
An Institutional Perspective to Rescue Scholarly OrphansAn Institutional Perspective to Rescue Scholarly Orphans
An Institutional Perspective to Rescue Scholarly Orphans
 
Who is Asking - Humans and Machines Experience a Different Scholarly Web
Who is Asking - Humans and Machines  Experience a Different Scholarly WebWho is Asking - Humans and Machines  Experience a Different Scholarly Web
Who is Asking - Humans and Machines Experience a Different Scholarly Web
 
The Memento Tracer Framework: Balancing Quality and Scalability for Web Arch...
The Memento Tracer Framework: Balancing Quality and Scalability  for Web Arch...The Memento Tracer Framework: Balancing Quality and Scalability  for Web Arch...
The Memento Tracer Framework: Balancing Quality and Scalability for Web Arch...
 
Memento Tracer An Innovative Approach Towards Balancing Scale and Fidelity f...
Memento Tracer An Innovative Approach Towards Balancing  Scale and Fidelity f...Memento Tracer An Innovative Approach Towards Balancing  Scale and Fidelity f...
Memento Tracer An Innovative Approach Towards Balancing Scale and Fidelity f...
 
Comparing the Performance of OAI-PMH with ResourceSync
Comparing the Performance of OAI-PMH with ResourceSyncComparing the Performance of OAI-PMH with ResourceSync
Comparing the Performance of OAI-PMH with ResourceSync
 
Evaluating Memento Service Optimizations
Evaluating Memento Service OptimizationsEvaluating Memento Service Optimizations
Evaluating Memento Service Optimizations
 
An Institutional Perspective to Rescue Scholarly Orphans
An Institutional Perspective to Rescue Scholarly OrphansAn Institutional Perspective to Rescue Scholarly Orphans
An Institutional Perspective to Rescue Scholarly Orphans
 
A Vision of the Library’s Role in Archiving Scholarly Artifacts
A Vision of the Library’s Role  in Archiving Scholarly ArtifactsA Vision of the Library’s Role  in Archiving Scholarly Artifacts
A Vision of the Library’s Role in Archiving Scholarly Artifacts
 
First Steps in Research Data Management Under Constraints of a National Secur...
First Steps in Research Data Management Under Constraints of a National Secur...First Steps in Research Data Management Under Constraints of a National Secur...
First Steps in Research Data Management Under Constraints of a National Secur...
 
Smart Routing of Memento Requests
Smart Routing of Memento RequestsSmart Routing of Memento Requests
Smart Routing of Memento Requests
 
A Web-Centric Pipeline for Archiving Scholarly Artifacts
A Web-Centric Pipeline for Archiving Scholarly ArtifactsA Web-Centric Pipeline for Archiving Scholarly Artifacts
A Web-Centric Pipeline for Archiving Scholarly Artifacts
 
Robust Linking to Web Resources
Robust Linking to Web ResourcesRobust Linking to Web Resources
Robust Linking to Web Resources
 
Signposting for Repositories
Signposting for RepositoriesSignposting for Repositories
Signposting for Repositories
 
Discovering Scholarly Orphans Using ORCID
Discovering Scholarly Orphans Using ORCIDDiscovering Scholarly Orphans Using ORCID
Discovering Scholarly Orphans Using ORCID
 
Using the Memento Framework to Assess Content Drift in Scholarly Communication
Using the Memento Framework to Assess Content Drift in Scholarly CommunicationUsing the Memento Framework to Assess Content Drift in Scholarly Communication
Using the Memento Framework to Assess Content Drift in Scholarly Communication
 
Uniform Access to Raw Mementos
Uniform Access to Raw MementosUniform Access to Raw Mementos
Uniform Access to Raw Mementos
 
Robust Links - a proposed solution to reference rot in scholarly communication
Robust Links - a proposed solution to reference rot in scholarly communicationRobust Links - a proposed solution to reference rot in scholarly communication
Robust Links - a proposed solution to reference rot in scholarly communication
 
To the Rescue of the Orphans of Scholarly Communication
To the Rescue of the Orphans of Scholarly CommunicationTo the Rescue of the Orphans of Scholarly Communication
To the Rescue of the Orphans of Scholarly Communication
 

Último

Mary Meeker Internet Trends Report for 2019
Mary Meeker Internet Trends Report for 2019Mary Meeker Internet Trends Report for 2019
Mary Meeker Internet Trends Report for 2019Eric Johnson
 
SQL Server on Azure VM datasheet.dsadaspptx
SQL Server on Azure VM datasheet.dsadaspptxSQL Server on Azure VM datasheet.dsadaspptx
SQL Server on Azure VM datasheet.dsadaspptxJustineGarcia32
 
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...hasimatwork
 
Google-Next-Madrid-BBVA-Research inv.pdf
Google-Next-Madrid-BBVA-Research inv.pdfGoogle-Next-Madrid-BBVA-Research inv.pdf
Google-Next-Madrid-BBVA-Research inv.pdfMaria Adalfio
 
Tungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and BeyondTungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and BeyondContinuent
 
overview of Virtualization, concept of Virtualization
overview of Virtualization, concept of Virtualizationoverview of Virtualization, concept of Virtualization
overview of Virtualization, concept of VirtualizationRajan yadav
 
Benefits of Fiber Internet vs. Traditional Internet.pptx
Benefits of Fiber Internet vs. Traditional Internet.pptxBenefits of Fiber Internet vs. Traditional Internet.pptx
Benefits of Fiber Internet vs. Traditional Internet.pptxlibertyuae uae
 
Generalities about NFT , as a new technology
Generalities about NFT , as a new technologyGeneralities about NFT , as a new technology
Generalities about NFT , as a new technologysoufianbouktaib1
 
如何办理朴茨茅斯大学毕业证书学位证书成绩单?
如何办理朴茨茅斯大学毕业证书学位证书成绩单?如何办理朴茨茅斯大学毕业证书学位证书成绩单?
如何办理朴茨茅斯大学毕业证书学位证书成绩单?krc0yvm5
 
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85APNIC
 

Último (10)

Mary Meeker Internet Trends Report for 2019
Mary Meeker Internet Trends Report for 2019Mary Meeker Internet Trends Report for 2019
Mary Meeker Internet Trends Report for 2019
 
SQL Server on Azure VM datasheet.dsadaspptx
SQL Server on Azure VM datasheet.dsadaspptxSQL Server on Azure VM datasheet.dsadaspptx
SQL Server on Azure VM datasheet.dsadaspptx
 
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...
Section 3 - Technical Sales Foundations for IBM QRadar for Cloud (QRoC)V1 P10...
 
Google-Next-Madrid-BBVA-Research inv.pdf
Google-Next-Madrid-BBVA-Research inv.pdfGoogle-Next-Madrid-BBVA-Research inv.pdf
Google-Next-Madrid-BBVA-Research inv.pdf
 
Tungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and BeyondTungsten Webinar: v6 & v7 Release Recap, and Beyond
Tungsten Webinar: v6 & v7 Release Recap, and Beyond
 
overview of Virtualization, concept of Virtualization
overview of Virtualization, concept of Virtualizationoverview of Virtualization, concept of Virtualization
overview of Virtualization, concept of Virtualization
 
Benefits of Fiber Internet vs. Traditional Internet.pptx
Benefits of Fiber Internet vs. Traditional Internet.pptxBenefits of Fiber Internet vs. Traditional Internet.pptx
Benefits of Fiber Internet vs. Traditional Internet.pptx
 
Generalities about NFT , as a new technology
Generalities about NFT , as a new technologyGeneralities about NFT , as a new technology
Generalities about NFT , as a new technology
 
如何办理朴茨茅斯大学毕业证书学位证书成绩单?
如何办理朴茨茅斯大学毕业证书学位证书成绩单?如何办理朴茨茅斯大学毕业证书学位证书成绩单?
如何办理朴茨茅斯大学毕业证书学位证书成绩单?
 
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
 

Focused Crawl of Web Archives to Build Event Collections

  • 1. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL Focused Crawl of Web Archives to Build Event Collections Martin Klein Lyudmila Balakireva Herbert Van de Sompel Research Library Los Alamos National Laboratory
  • 2. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 2 • Often orchestrated by subject matter experts, archivists, special collection librarians, technicians • Potentially with guidance from institutional collection policy • Results in a list of seeds (URIs, social media accounts, etc) • Utilization of crawling services such as Archive-It, Social Feed Manager Background – Event Collection Building
  • 3. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 3 • Temporal: time passed since event is of concern  Use of web archives • Selection: seeds often picked manually  Use of references from Wikipedia pages • Relevance: seed assessment often done by humans  Use of focused crawling with content and temporal relevance assessment Inspiration from: “Extracting Event-Centric Document Collections from Large-Scale Web Archives” Gerhard Gossen, Elena Demidova, Thomas Risse https://doi.org/10.1007/978-3-319-67008-9_10 Problems and our Approach
  • 4. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 4 • Web archives are an invaluable resource for researchers, historians, journalists, etc. • Often broad in scope, large in scale, covering different temporal intervals • Makes discovery, access, and analysis difficult Background – Archived Web
  • 5. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 5
  • 6. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 6
  • 7. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 7 • Can we create event collections by focused crawling online- available web archives? • How do event collections created from the archived web compare to those created from the live web? • How does the amount of time passed since the event affect the collections built from the live and the archived web? • How do event collections built from the archived web compare to manually curated collections? Questions
  • 8. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 8 • Topics limited to terror attacks and mass shootings in the U.S. • From different times in the past • Focused crawl of: a) 22 archives, simultaneously, via Memento infrastructure b) the live web • Take content and temporal relevance into account, equally weighted • Use events’ Wikipedia page as input for focused crawler Experiment
  • 9. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 9 1. Content of Wikipedia page + random 60% of page’s references • Generate topic vector (TF-IDF of 1grams + 2grams) 2. Content of remaining 40% of Wikipedia page’s outlinks • Generate topic vector (TF-IDF of 1grams + 2grams) • Compute cosine similarity value between vectors 1 and 2 • Run 10 times • Take average similarity value as content threshold Content Relevance
  • 10. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 10 • Define temporal interval for which crawled pages are considered relevant • Event date extracted from Wikipedia event page • Change point determined from graph of proportional Wikipedia page edits per day Temporal Relevance 1 Event Date Change Point Today 0 0
  • 11. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 11 • Extract datetime from pages via: • URI http://www.cnn.com/2017/12/09/us/wildfire-fighting-tactics/ • Meta tags <meta property="article:published" itemprop="datePublished" content="2017-12-09T10:14:50-05:00" /> • ODU’s Carbondate tool http://carbondate.cs.odu.edu/ • Memento datetime • X-Header Datetime Extraction
  • 12. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 12 • Use version of Wikipedia page that was live at change point • Crawl stop conditions: • No more relevant documents left • 5 levels deep • Utilized crawl priority queue Crawls Level 2 Level 1 Level 0 Child 1 Seed Child 2 Child 3 Child 3.2Child 3.1Child 2.1Child 1.1 Child 3.2Child 1.2
  • 13. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 13 • New York City, October 31st 2017 • San Bernadino, December 2nd 2015 • Tucson, January 8th 2011 • Binghampton, April 3rd 2009 Collections Crawled (in November 2017)
  • 14. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 14 NYC, 10/31/2017 – URIs per Level 0 1 2 3 4 5 Crawl depth NumberofURIs 0500100015002000 Web Archive Crawl 0102030405060708090100 All URIs Relevant URIs 0 1 2 3 4 5 Crawl depth 0500100015002000 Live Web Crawl 0102030405060708090100 Percent All URIs Relevant URIs
  • 15. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 15 TUC, 01/08/2011 – URIs per Level 0 1 2 3 4 5 Crawl depth NumberofURIs 020000400006000080000 Web Archive Crawl 0102030405060708090100 All URIs Relevant URIs 0 1 2 3 4 5 Crawl depth 020000400006000080000 Live Web Crawl 0102030405060708090100 Percent All URIs Relevant URIs
  • 16. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 16 NYC, 10/31/2017 – Relevance over… Crawled Documents Crawl Time
  • 17. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 17 TUC, 01/08/2011 – Relevance over… Crawled Documents Crawl Time
  • 18. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 18 TUC, 01/08/2011 – Comparison to Archive-IT 0 5000 10000 15000 050001000015000 Documents AccumulatedRelevance Web Archive Crawl Archive−It Crawl
  • 19. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 19 TUC, 01/08/2011 – Web Archive Contributions web.archive.org 75% wayback.archive−it.org 14% webarchive.loc.gov 7% web.archive.bibalex.org 2% archive.is 2%
  • 20. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 20 • Web archives are great resources to build event collections of web resources • Crawling web archives is much slower than the live web • Collections about very recent events benefit more from the live web than the archived web but • Collections about events from the distant past benefit more from the archived web than the live web • Utilizing multiple web archives is beneficial for the collection • Focused crawls have the potential to outperform manual collection building Takeaways
  • 21. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL 21 https://web.archive.org/web/20171206181955/https:/twitter.com/TVNewsArchive/status/938466726190096384
  • 22. Focused Crawl of Web Archives to Build Event Collections @mart1nkle1n WebSci 2018, 05/30/2018, Amsterdam, NL Focused Crawl of Web Archives to Build Event Collections Martin Klein Lyudmila Balakireva Herbert Van de Sompel Research Library Los Alamos National Laboratory