Preservation, Publishing, and People: A SEAD View
•
1 gostou•329 visualizações
Slides from a workshop organized by the SEAD Virtual Archive team (see http://sead-data.net/)
Recomendados
Recomendados
Mais conteúdo relacionado
Destaque
Destaque (10)
Semelhante a Preservation, Publishing, and People: A SEAD View
Semelhante a Preservation, Publishing, and People: A SEAD View (20)
Último
Último (20)
Preservation, Publishing, and People: A SEAD View
- 3. Data lifecycle • Research occurs over months to years. Praveen Kumar study of Mississippi River Basin flood of late April, early May 2011. • Arrange funding, define objectives (2011) • Data gathering: sample flood plain at designated locations, take pictures, obtain satellite data, contract with independent organization to fly over the area with Lidar • Data cleaning and analysis • Publish 2‐3 papers (2014) • Decide what data to package for publishing alongside publications • Publish the datasets • Each published package we call a Research Object
- 4. Publish‐reuse window • We focus on one window in time in lifecycle of research data : starts when researcher is ready to make data publically available … through to its first case of use by unrelated party (reuse). “publish‐reuse window” • Why this window? • Repository services have to be self‐documenting to achieve reproducibility. I derive new object from object in SEAD VA, I revise object in SEAD VA – these are different actions, by different people, with different implications.
- 6. Research Object: what the RO is • The Research object (RO) is an aggregation of resources that can be transferred, produced, and consumed by common services across organizational boundaries. The RO encapsulates digital knowledge and provides a mechanism for sharing and discovering re‐usable research. • ROs are a bundle of primary results, metadata describing the generated data, software used, configuration parameters used with the software, input data sources, … • An RO can and will likely have multiple manifestations. • Research object is the publishable object.
- 7. Why is Research Object view important? • Addresses weaknesses in existing solutions: The hierarchical “belongs to” organization of information is extremely inadequate for all but simplest cases. • Facilitates reproducibility: We can no longer look just at data products: software is critical for reproducibility (even if repeatability is not the goal.) • Allows for uniform handling: Research object is dropped into a BagIT bag (1 bag = 1 RO). SEAD VA accepts bags of all colors, but all are bags. Lifecycle of ROs tracked in SEAD VA • Just makes sense: When is the result of a scientific dissertation a uniform collection of files with fixed directory structure? <answer: never>
- 8. Research Object, Components of • Identity : unique ID • Entities : core data or software objects themselves • Properties : Aggregation : “belongs to” relationship, used to aggregate within Research Object • Properties : Relationships : “related to” relationship • Properties: Descriptive/Annotative : metadata • Properties: Provenance : “derived from”, “versioned from” relationship as well as others • Properties: Agents : data creator (author list), curator, data scientist • State : external to the RO
- 10. Research Object State Transition An RO in one of three states: LO, PO, and CO as follows: • Live object (LO) – a work in progress. Data creator assembling content for publication • Curation object (CO) – an object after creator signaled intention to publish. Curator works on the curation object; changes are selective. • Publication object (PO) ‐ a final version ready to be disseminated widely. Published Objects (PO’s) are mutable under certain conditions only. RO described by model: RO = {s, dm, c} Where s is state of an RO at any point in time, dm is its descriptive metadata, and c is the entities (core content) and relationships amongst entities
- 11. RO = {s, dm, c} Where s is state of an RO at any point in time, dm is its descriptive metadata, and c is the resources and relationships amongst resources. State transition graph
- 19. Overview The Data Creator collects data and, once done with a study, gathers materials that support the study and submits them for publication and preservation in institutional repositories. Example: A dissertation that is based • images from USGS • spreadsheets with numbers and calculations • computing scripts • videos of experiments In VA a data creator can: • Upload research objects (ROs) • Preview, review and download ROs • Check status of ROs in queue to IR
- 20. Background : SEAD Services • SEAD Research Network • Project Spaces • Packaging and Mapping
- 21. Research Network • Network of data creators, curators and re‐use scientists across disciplines • Rich ontology to support links to data, projects and publications • Visualizations of co‐authorship and co‐citation
- 22. ORCiD / SEAD Research Network Integration • Create empty profile in VIVO • Execute harvester • Ingest data
- 23. Project Spaces • 15 project spaces (incl. an open demo space and an internal testing space) • Thousands of collections in active curation • Once a collection is marked for publication, it can be ingested into Virtual Archive
- 26. Packaging and Mapping (BagIT / ORE) • BagIt format • standardized “envelopes” (bags) • no requirements for “knowing” internal semantics • 3 elements: a bag declaration (bag.txt), a manifest file (manifest‐ <algorithm>.txt, folder with content (data) • Tools available for bagging • SEAD BagIt service • LOC Bagger tool (http://sourceforge.net/projects/loc‐xferutils/files/loc‐ bagger/2.1.2/)
- 27. Resource Maps • OAI/ORE standard • Exposes rich content • Captures semantic of relationships among RO items • Identifies aggregations • SEAD VA OAI/ORE relationship classes: • Aggregation • Description • Authorship • Copyright / rights • Modification • Derivation • Citation • Processing (calculation, computation, etc.)
- 29. OAI/ORE Map Example <rdf:RDF … <rdf:Description rdf:about=URI> <!‐‐ data item‐‐> <ore:isAggregatedBy>ID</ore:isAggregatedBy> <dcterms:identifier rdf:datatype=URI>ID</dcterms:identifier> <dcterms:title rdf:datatype=URI>Vortex_Mining.xlsx</dcterms:title> <dcterms:source rdf:datatype=URI>test_bag/data/Vortex_Mining.xlsx</dcterms:source> <!‐‐ A related resource from which the described resource is derived. ‐‐> </rdf:Description> ….. </rdf:RDF>
- 33. Upload Research Object • On the Upload Data tab, click “Choose File” • Select a test dataset in the dialog window • Click upload Upload Data Tab
- 34. Review Research Object • Check that the object is correct • Change project name and description • Agree to the license terms • Click “Submit Dataset for Review”
- 36. Trace Activity • Go to activity tab • See all actions performed by you • Click on the dataset name to see details Activity tab
- 38. Receive Notification • After the next part of the tutorial, check your inbox for email from SEAD VA
- 40. Overview The Curator works on Research Objects created and submitted by Data Creators: reviews submission, modifies metadata, and takes action to move submission to their Institutional Repository In VA curator can: • Select Item for review from curation queue • Enhance Metadata • Deposit to Institutional Repository
- 42. Automatic IR Recommendation (SEAD VA Matchmaker) • Matches RO’s to compatible Institutional Repository • Recommends best Institutional Repository match for RO • Facilitates transfer and deposit of heterogeneous ROs
- 44. IR – SEAD VA “contract”: the Service Level Agreement • Service Level Agreement (SLA) is a contract of sorts between SEAD VA and an Institutional Repository. It captures • Repository requirements and privileges • Repository services • The IR Recommendation system uses excerpts from IR’s SLA to identify compatible pairs of datasets and repositories during RO deposit.
- 45. Service Level Agreement ‐ Requirements and Privileges (summary) • RO properties – Requirements • Data contributor Institutional Affiliation • Scientific Domain • Data Organization (e.g.: BagIt or SWORD) • Size • Versioning • Minimal Metadata • Licensing (eg: open, embargoed) • Repository privileges • Repository is free to re‐distribute the RO received from SEAD VA, except in case of embargo. • Repository can migrate RO into other formats and re‐distribute migrate ROs. • Repository curators can annotate data collections to comply with standards or upgrades in our policies.
- 46. SLA – Repository Service Guarantees • Long‐term preservation • Format Migration • Archival support • Embargo • Access • DOI generation • Technical guarantees: • Limited Downtime • Data Ingest Time • Backup • Integrity checks
- 47. Excerpt from from SLA for IU Scholarworks • Institutional Affiliation • At least one author, at the time of deposit, belongs to the same institution as our repository. • RO Size • 150 MB for items uploaded directly to IUScholarWorks, 10 GB total • 5 TB for items hosted on the SDA • Versioning • Only final PO is accepted, subsequent versions will substitute the version of record. • Scientific Domain – Curator review might be needed • ROs are associated with research in the domains of ANY (identify specific domains or put “sustainability science” for a broader match)
- 48. The IR Recommender use of an SLA • IR Recommender implements an IR’s SLA as a set of executable rules in the Matchmaker. The rules are executed with a rules engine called “Drools” • Rules can be added on the fly, meaning new IR can be added just by specifying a SLA. • Incorporate modifications in SLA to rules at runtime • Clean mapping of SLA terms to Drools Drools rules
- 49. Mapping SLA to Drools rules rule "IU Scholaworks Affiliation rule” dialect "mvel” salience 20 when SeadDeliverableUnit( title != null ) //Per IU SLA collection should have title SeadDeliverableUnit($contributors:dataContributors ) eval( $contributors.size>0 ) //Creators should not be empty per IU SLA $seadPerson: SeadPerson( idType == "vivo" && getEmail(id)=="Indiana University") from $contributors; $seadDu : SeadDeliverableUnit(sizeBytes < 10000000000 ) //Total collection size less than10 GB approximately SeadDeliverableUnit(fileNo < 1000 ) //Total file count less than 1000 SeadDeliverableUnit( "CC" in (rights) ) //Open access data then addRepository("iu", 2); //Adding IU repository to the queue of matched repositories with priority 2 end Rule declaration Condition Execution Affiliated data contributor found
- 58. Trace Activity • Go to activity tab • See all actions performed by you • Click on the Research Object name to see details Activity tab
- 61. Overview: The Data Scientist Data Scientist uses research objects that were created by someone else for his/her purposes and creates new research objects by modifying existing objects. Super Simple Example: Putting images in given RO 3 into a single presentation and creating a new RO Data scientist can: • Search • Download (bags) • Modify • Re‐upload
- 63. Provenance • What is Provenance? • Provenance is information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness • Also called “Lineage” or “Pedigree” • Advantages of provenance for preservation • Derive ownership • Asses quality and trustworthiness • Reproducibility • Validation • Failure Tracing Not used in Preservation Provenance
- 64. Provenance in Repositories • The provenance important here is provenance of a Research Object • Why important? • For the data scientists in “Search” • To check ownership of RO • To asses quality and trustworthiness of RO • For the Curators • To check curation history • Provenance role in “Publish ‐ Reuse window” • Published Object (PO) Provenance • Curation Object (CO) Provenance
- 65. Provenance Capture in SEAD VA • Uses Komadu provenance system • Captures activity in real time, assembles new activity into internal representation as provenance graphs • W3C PROV spec compliant • Terminology • Activity : Some Processing Event in SEAD VA • Entity : A Research Object (in CO or PO state) • Agent : Data Creator, Curator, Data Scientist
- 67. Maintaining Provenance among Published ROs • Two identifiers maintained: DOI and Internal Identifier. • Why two identifiers? • DOI: each RO has a unique DOI. • Internal Identifier: lineage maintained through internal identifier which maintains the relationship between original object and derived object
- 69. Provenance among Published ROs • Provenance relationships captured in Komadu • Entity‐Entity (derivation) : When the second publish is done • This RO provenance capture continues up to any number of publish:download:re‐upload cycles • At second publish (RO’), “wasDerivedFrom” element is added in the oaiore.xml referring to the original Internal Identifier
- 71. Curation Time Provenance Capture Create RO Publish RO Creator Curator Provenance within Curation
- 72. Curation Time Provenance Capture • Curation Activities • Curation‐Edit‐Event • Publish‐Event • Provenance relationships captured in Komadu • Agent‐Activity : When some Agent triggers one of above Activities • Activity‐Entity : When an Activity Generates (Updates) a Research Object • Example Scenario • Curator X edits metadata on research object Y • Agent‐Activity relationship (association) between X and Curation‐Edit‐Event • Activity‐Entity relationship (generation) between Curation‐Edit‐Event and Y
- 77. Search for Data
- 78. Find data Filter
- 83. Re‐Upload data
- 86. Check Activity • Go to activity tab • See activities performed (Curation time provenance) • Click on the Research Object name to see details Activity tab
- 88. Thank You