SlideShare uma empresa Scribd logo
1 de 6
Full Text Biomedical Literature Processing:More Than a Scaling Challenge Christophe Roeder, Tom Christiansen, Helen Johnson, Karin Verspoor, (UC Denver) Gully Burns (ISI) , Lawrence Hunter (UC Denver)
Obtaining Documents Identify documents by querying PubMed Challenging due to variations in names Not all documents are freely available One project identified 3034 documents 1253 (41%) licensed, available without charge 418 (14 %) available in PubMed Central 	 Availability effects experiment reproducibility Downloading can be problematic Manual download is slow. PMC Open Access is limited Arrange bulk download from publishers based on existing licenses
File Formats Documents are available in many formats: HTML, XML, PDF, plain text Convert to plain text for NLP tool input Stripping XML or HTML markup is relatively easy ISI is working on PDF Extract to find correct flow Keep document zoning, other markup  headings, sections, captions, italics Identify source character encoding properly XML stores the encoding in file, others do not
Character Representation Encoding is a mapping from bytes to characters Difficult to discern wich encoding a file uses  ASCII, UTF-8, MacRoman,  ISO-8859-1, or other? Reading a file with the wrong encoding can produce unreported errors and spurious ‘?’ characters Java regular expression classes (, ) don’t match non-ASCII characters Some characters look like others:  dash, en dash, minus  space, em space, non-breaking-space
Scaling Use a cluster when you need more than a desktop Prefer an easy migration from desktop to cluster Concurrency (threading) issues are minimized since most NLP processes are independent Finding success using Sun/Oracle Grid Engine (SGE) and Network File System (NFS) on a small (48 core) cluster NFS shares disks between nodes SGE starts and manages processes on cluster
Acknowledgements UC Denver Helen Johnson Tom Christiansen Karin Verspoor, NIH grant R01 LM010120-01 Larry Hunter,  NIH 2R01LM009254-04 NIH 2R01LM008111-04A1 NIH 5R01GM083649-02 ISI Gully Burns, NSF grant #0849977

Mais conteúdo relacionado

Mais procurados

Anton Dorfman - Reversing data formats what data can reveal
Anton Dorfman - Reversing data formats what data can revealAnton Dorfman - Reversing data formats what data can reveal
Anton Dorfman - Reversing data formats what data can reveal
DefconRussia
 
Adaptive information extraction
Adaptive information extractionAdaptive information extraction
Adaptive information extraction
unyil96
 

Mais procurados (8)

Ld4 l triannon
Ld4 l triannonLd4 l triannon
Ld4 l triannon
 
SANAPHOR: Ontology-based Coreference Resolution
SANAPHOR: Ontology-based Coreference ResolutionSANAPHOR: Ontology-based Coreference Resolution
SANAPHOR: Ontology-based Coreference Resolution
 
Survey On Building A Database Driven Reverse Dictionary
Survey On Building A Database Driven Reverse DictionarySurvey On Building A Database Driven Reverse Dictionary
Survey On Building A Database Driven Reverse Dictionary
 
Anton Dorfman - Reversing data formats what data can reveal
Anton Dorfman - Reversing data formats what data can revealAnton Dorfman - Reversing data formats what data can reveal
Anton Dorfman - Reversing data formats what data can reveal
 
Open source software for building open access repositories
Open source software for building open access repositoriesOpen source software for building open access repositories
Open source software for building open access repositories
 
Brain Imaging Data Structure and Center for Reproducible Neuroscince
Brain Imaging Data Structure and Center for Reproducible NeuroscinceBrain Imaging Data Structure and Center for Reproducible Neuroscince
Brain Imaging Data Structure and Center for Reproducible Neuroscince
 
Adaptive information extraction
Adaptive information extractionAdaptive information extraction
Adaptive information extraction
 
Corpus Linguistics :Analytical Tools
Corpus Linguistics :Analytical ToolsCorpus Linguistics :Analytical Tools
Corpus Linguistics :Analytical Tools
 

Destaque (9)

Roeder posterismb2010
Roeder posterismb2010Roeder posterismb2010
Roeder posterismb2010
 
Spring survey
Spring surveySpring survey
Spring survey
 
Uml
UmlUml
Uml
 
Roeder rocky 2011_46
Roeder rocky 2011_46Roeder rocky 2011_46
Roeder rocky 2011_46
 
Spring Framework 101
Spring Framework 101Spring Framework 101
Spring Framework 101
 
Sge
SgeSge
Sge
 
Maven
MavenMaven
Maven
 
Spring Intro
Spring IntroSpring Intro
Spring Intro
 
Spring MVC Basics
Spring MVC BasicsSpring MVC Basics
Spring MVC Basics
 

Último

Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 

Último (20)

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 

Rocky2010 roeder full_textbiomedicalliteratureprocesing

  • 1. Full Text Biomedical Literature Processing:More Than a Scaling Challenge Christophe Roeder, Tom Christiansen, Helen Johnson, Karin Verspoor, (UC Denver) Gully Burns (ISI) , Lawrence Hunter (UC Denver)
  • 2. Obtaining Documents Identify documents by querying PubMed Challenging due to variations in names Not all documents are freely available One project identified 3034 documents 1253 (41%) licensed, available without charge 418 (14 %) available in PubMed Central Availability effects experiment reproducibility Downloading can be problematic Manual download is slow. PMC Open Access is limited Arrange bulk download from publishers based on existing licenses
  • 3. File Formats Documents are available in many formats: HTML, XML, PDF, plain text Convert to plain text for NLP tool input Stripping XML or HTML markup is relatively easy ISI is working on PDF Extract to find correct flow Keep document zoning, other markup headings, sections, captions, italics Identify source character encoding properly XML stores the encoding in file, others do not
  • 4. Character Representation Encoding is a mapping from bytes to characters Difficult to discern wich encoding a file uses ASCII, UTF-8, MacRoman, ISO-8859-1, or other? Reading a file with the wrong encoding can produce unreported errors and spurious ‘?’ characters Java regular expression classes (, ) don’t match non-ASCII characters Some characters look like others: dash, en dash, minus space, em space, non-breaking-space
  • 5. Scaling Use a cluster when you need more than a desktop Prefer an easy migration from desktop to cluster Concurrency (threading) issues are minimized since most NLP processes are independent Finding success using Sun/Oracle Grid Engine (SGE) and Network File System (NFS) on a small (48 core) cluster NFS shares disks between nodes SGE starts and manages processes on cluster
  • 6. Acknowledgements UC Denver Helen Johnson Tom Christiansen Karin Verspoor, NIH grant R01 LM010120-01 Larry Hunter, NIH 2R01LM009254-04 NIH 2R01LM008111-04A1 NIH 5R01GM083649-02 ISI Gully Burns, NSF grant #0849977