Since last year’s TDWG we’ve taken this talk on the road, here’s an update on the BHL global cluster for those who were here last year, and an introduction for those who weren’t. We'll talk about reasons for needing the cluster, concepts and software developed to support the cluster (which is all available as open source software) and of course, the famous lessons learned.
NOTE: please click on the 'Notes' tab below the presentation for more detail on each slide.
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Updates on the BHL Global Cluster
1. Updates on the
BHL Global
Cluster
biodiversity heritage library
anthony goddard phil cryer
2. Us?
o We do this talk a lot.. generally our shirts match.
3. What is the BHL?
• BHL - The Biodiversity Heritage Library
o digitization component of the Encylopedia of Life
o a consortium of a global partners
o aims to share historic biodiversity literature texts
o provide open access of all content
o free for all
4. Why do we need a cluster?
• All BHL data is at the Internet Archive in San Francisco
o no redundancy
o single point of failure (earthquake risk)
o limited in how we could serve
o no easy way to analyze data
• First global BHL cluster gives us
o redundancy
o no single point of failure
o various new serving options
o new ways to run analytics
#win!
5.
6.
7. Use Linux and open source software running on
commodity hardware to create a scalable, distributed filesystem.
15. the plan
• Internet2 - woohoo
o “This will take forever” (it took longer)
o “We need more space” (not 24TB)
o “something’s overloading the network” (oops)
o “this checksum is wrong” (what the...)
• Lessons learned would we do it again? Probably not.
16. code: grabbyd
1
Internet Archive, San Francisco BHL Global, Woods Hole
22. BHL content distribution
1 ?
Internet Archive, San Francisco BHL Global, Woods Hole BHL China, Beijing
2 2 ?
BHL, St. Louis BHL Europe, London BHL Australia, Melbourne
23. BHL content + local data
Internet Archive, San Francisco BHL Global, Woods Hole BHL China, Beijing
Content sourced from China, scanned by
Internet Archive, replicated into BHL Global
24. BHL content + regional data
Internet Archive, San Francisco BHL Global, Woods Hole
?
BHL Europe, Paris BHL Europe, London BHL Europe, Berlin
Content sourced from BHL Europe partners may, or may
not, be passed back to Internet Archive and BHL Global
25. other replication challenges
• deleting content - "going dark"
• new content coming in from other sources (localization of content)
• distributing modified content
26. fedora-commons integration
Repository platform
• storage, access and management digital content
• a base for software developers to build tools for sharing
• free, community supported, open source software
27. fedora-commons integration
Repository platform
• storage, access and management digital content
• a base for software developers to build tools for sharing
• free, community supported, open source software
• Maintains a persistent, stable, digital archive
o provides backup, redundancy and disaster recovery
o complements existing architecture by incorporating open standards
o stores data in a neutral manner
o shares data via OAI
28. BHL content distribution
Internet Archive, San Francisco BHL Global, Woods Hole Fedora-commons
BHL, St. Louis BHL Europe, London
29. BHL content distribution
Internet Archive, San Francisco BHL Global, Woods Hole Fedora-commons
OAI
BHL node Fedora-commons
30. BHL content distribution
Internet Archive, San Francisco BHL Global, Woods Hole Fedora-commons
OAI
BHL node Fedora-commons
32. thanks.
anthony goddard phil cryer
all code available bit.ly/bhl-bits
presentation slides on slidesha.re/bhl-slides
Notas do Editor
PHIL/ANT Ohai! This is Phil, this is Ant.. Since last year’s TDWG we’ve taken this talk on the road, here’s an update for those who were here last year, an introduction for those who weren’t.
ANT if anyone does not know, BHL is a digitization component of the Encylopedia of Life, a consortium of a global libraries and national history museums, it aims to digitize and share historic biodiversity literature texts, open access and free for all
PHIL currently all BHL data is stored at IA, this is bad for many reasons - with our own cluster we can have control and new options on how we can use and store out data
PHIL so how did we get from a proof of concept, put together with some various, outdated hardware
PHIL to our formal production cluster
PHIL with our first global cluster, our concept was to create a scalable storage system to store and server our data, that others *could* emulate, using open source software
PHIL our systems run Debian Linux, using the latest filesystem, ext4, which supports far larger file systems (up to 1 Extabyte) and file sizes (16TB). We use the GlusterFS distributed/networked filesystem to handle the replication
ANT cluster contains 6 boxes like this, 24 hard drives per, broken up and mirrored via a networked clustered storage. This gives us 216TB raw space, or 108TB of usable, mirrored data.
ANT it’s a cluster.. six boxes but we see it as one giant machine.. 64GB RAM, 100TB Hard drive, 48 processors
PHIL just an example of a record type that we store, all of the derivative files of a book can range anywhere from 200MB to over 3TB. Here’s an average record, and it’s about 650MB, the size of a standard cd-rom (our mirror currently has over 80k such records)
ANT we looked at different ways of transferring the files from Internet archive to our own cluster
ANT after considering all the options, was decided to download the data from IA
ANT this shows some of the downloading in progress (250MB/sec), all told we have downloaded 74TB so far - had some problems...
ANT talk about the 1st, 2nd and 3rd one
PHIL do the 4th, and the ‘lessons learned’
PHIL parts of the code that did the initial download has been reworked to be an ongoing process, grabbyd will handle downloading new items from IA to the cluster weekly
PHIL we currently have reporting to give updates on download progress, overall size of the data and transfer rates. This will be expanded as we go forward
PHIL to keep various nodes in sync we’ve written a backend ‘open source’ Dropbox like server application. Using other software we can have a service listening for any changes and kicking off the syncing scripts
ANT all of our code that we write is available as open source software, hosted on the BHL code repository
ANT We have begun initial speed and sync tests within the US and to London, work will be starting on these tests to Australia shortly
ANT the global aspect of BHL has become more clear now after last week’s global meeting, with Egypt, Brazil joining others like China, AU, and EU
ANT There are many options for syncing, due to the degree of control we require, we chose to use IA as a point of data ingestion, Woods Hole as a master site to seed data from
ANT but, in the case of China, we ingest data into IA and then sync that data to our cluster in Woods Hole - so our model is flexible
ANT In the case of BHL-Europe, content may or may not be ingested via IA, depending on the desire for BHL-Europe to take advantage of IA services such as OCR
Phil there are other challenges such as deleting or content “going dark”, localization of content and especially how to deal with modified or annotated content
Phil to track changes to the content we’re using Fedora-commons, which provides access and management of digital content, and is a base to build other apps on to use the data in other ways
Phil Fedora maintains a persistent archive, used for backup and disaster recovery for the files, compliment existing arch by using open standards and not requiring anything of the existing system. Offers more sharing options via OAI
Phil as seen in the mix, Fedora runs independently of the system
Phil while it could provide a conduit to share metadata about the archive
Phil and can even talk 1:1 with another Fedora instance
ANT we have this hardware, we are intending on making use of it for computation services such as taxon name finding and text mining.
PHIL we have tested running Hadoop on our cluster, and work on running statistical jobs in R have been run in Missouri and we’re looking to integrate the cluster for this
PHIL/ANT in closing, while the BHL global cluster is to serve a certain purpose, we’d like to highlight that anyone could cluster can be built a similar cluster in many ways, and for almost no money, contact us for any advice or assistance for this. Thanks