The Right Combination: Using DDI and PREMIS for data preservation Parul Sharma & Sally Vermaaten

1. The Right Combination:
Using DDI and PREMIS for data
preservation
Parul Sharma & Sally Vermaaten
March 2012

2. Outline
1. The context – drivers for preservation
2. The problem – challenges faced when trying to re-
use data
3. Our solution – metadata for data management
&preservation
4. Our recommendations– strategies for making the
right metadata choices
2

3. 1. THE CONTEXT:DRIVERS FOR
PRESERVATION
3

4. Data is a cross-domain concern
Geospatial data
Scientific data
Statistical data
Financial and
commercial data
4

5. There are many drivers
for data preservation
Legal mandates Cost of data
Verification collection
Uniqueness of data Data re-use
5

6. An example of data re-use at
Statistics New Zealand
6

7. 2. THE PROBLEM:
CHALLENGESFACEDWHEN
TRYING TO RE-USE DATA
7

8. Common challenges
to re-use/preservation of any type of digital object

9. Common challenges
to re-use/preservation of any type of digital object
I can’t find it
I can’t open it (wrong hardware/software)
I’m not sure it is the right thing

10. Unique challenges
to re-use/preservation of structured data

11. Unique challenges
to re-use/preservation of structured data
I’m not sure it is the authoritative data
I don’t understand the meaning of the data - data is
not self-descriptive
I can’t use the data because I can’t harmonize it
with other data
11

12. 3. OUR SOLUTION: METADATA
FOR DATA MANAGEMENT
&PRESERVATION
12

13. Our solutions
Have subject Archivists put
I can’t find the data
(common) experts record it in a safe
locations place
I can’t open the data
(common)
I’m not sure it’s the
right thing / it’s the
authoritative data
(particularly hard
with data)
I don’t understand
the meaning of the
data (particularly
hard with data)
I can’t reuse the data
because it’s not
harmonised (unique
to data)
13

14. Our solutions
Have subject Archivists put
I can’t find the data
(common)
experts record it in a safe
locations place
Archivists
I can’t open the data
(common) monitor file
formats
I’m not sure it’s the
right thing / it’s the
authoritative data
(particularly hard
with data)
I don’t understand
the meaning of the
data (particularly
hard with data)
I can’t reuse the data
because it’s not
harmonised (unique
to data)
14

15. Our solutions
Have subject
I can’t find the data Archivists put it in a
experts record
(common) safe place
locations
I can’t open the data Archivists monitor
(common) file formats
I’m not sure it’s the
right thing / it’s the Subject experts &
Have subject
archivists capture
authoritative data experts identify key
what has happened
(particularly hard datasets
to the data
with data)
I don’t understand
the meaning of the
data (particularly
hard with data)
I can’t reuse the data
because it’s not
harmonised (unique
to data)
15

16. Our solutions
Have subject
I can’t find the data Archivists put it in a
experts record
(common) safe place
locations
I can’t open the data Archivists monitor
(common) file formats
I’m not sure it’s the
right thing / it’s the Subject experts &
Have subject
archivists capture
authoritative data experts identify key
what has happened
(particularly hard datasets
to the data
with data)
I don’t understand
Have subject
the meaning of the Archivists capture
experts capture
data (particularly or QA metadata
important data
hard with data)
I can’t reuse the data
because it’s not
harmonised (unique
to data)
16

17. Our solutions
Have subject
I can’t find the data Archivists put it in a
experts record
(common) safe place
locations
I can’t open the data Archivists monitor
(common) file formats
I’m not sure it’s the
right thing / it’s the Subject experts &
Have subject
archivists capture
authoritative data experts identify key
what has happened
(particularly hard datasets
to the data
with data)
I don’t understand
Have subject
the meaning of the Archivists capture
experts capture
data (particularly or QA metadata
important data
hard with data)
I can’t reuse the data Archivists and
Tools to create
because it’s not subject experts
more standardised
harmonised (unique capture detailed
data
to data) metadata
17

18. To support these processes…
Metadata is key
We could invent our own standard for recording
metadata but there is a better way …
18

19. How?
PREservation Metadata:
Data Documentation
Implementation Strategies
Initiative (DDI)
(PREMIS)
Dublin Core
+ +
Discover !
Preserve!
Describe!
19

20. Comparison of standards coverage
Dublin Core DDI PREMIS
Discovery information Surveys and outputs Objects (significant
about a resource (e.g. (Series and Studies) characteristics,
Title, Creator, Publication checksums, basic
date) identifying information)
Methodology & quality Events (preservation
information actions)
Classifications used Agents
Dataset descriptions Rights
Variables used
Links to documentation
20

21. Metadata to support re-use
I can’t find the Have subject
Archivists put it in a
data
experts record
safe place DDI
locations
PREMIS
I’m not sure it’s Have subject
Subject experts &
archivists capture
the authoritative experts identify key
what has happened
datasets
data to the data
I don’t Have subject
understand the experts capture Archivists capture
meaning of the important or QA metadata
metadata
data
I can’t open the Archivists monitor
data file formats
I can’t reuse the Archivists and
Tools to create
subject experts
data because it’s capture detailed
more standardised
data
not harmonised metadata
21

22. 4. OUR RECOMMENDATIONS:
STRATEGIES FOR MAKING THE
RIGHT METADATA CHOICES
22

23. Metadata Top Tips
1. Create structures that will allow you to re-use metadata
tools
2. Use standards that are fit for your content so users can
re-use
3. Consider overlap between standards so you’re using the
right standard for the right job
4. Provide standard based tools and capture at point of
creation to improve quality and efficiency
23

24. 1. Create structures that will allow you
to re-use metadata tools
Set yourself up to be able to use the same tools to
harvest and mine your metadata (e.g. handy reports,
searching across content types) by:
– developing a standard structure that can support all your content
types
– and recording generic information in generic metadata standards
24

25. Data_1500 Database_0120
DublinCore.xml Non-format DublinCore.xml
PREMIS.xml specific metadata
PREMIS.xml
Original Original
data.sas7bdat database.mdb
questionnaire.doc ArchiveMaster
ArchiveMaster Header
Data metadata.xsd
data.csv Format metadata.xml
specific structure &
Documentation metadata Content
questionnaire.pdf Schema1
Metadata Table1
DDI.xml table.xsd
table.xml 25

26. 2. Use standards that are fit for your
content so users can re-use
Enable future re-use and understanding by recording format
or content-specific metadata in fit-for-purpose standards e.g.
DDI for statistical data
SIARD for databases
MIX for images
26

27. 3. Consider overlap between standards so
you’re using the right standard for the
right job
Information DDI PREMIS Dublin Core Useful to
duplicate?
Basic identifying •Title •Title yes
information •Creator •Creator
•PublicationDate •Date
•ID •Identifier
Access •Access Conditions •Rights entity •Rights No – PREMIS is
information most expressive
and generic
location
27

28. 4. Provide standard based tools and
capture at point of creation to
improve quality and efficiency
At first, you may need to capture or collate all
metadata about data yourself
Think ahead about tools you might be able to
provide to data experts to allow them to record the
information directly in the standard if possible
28

29. 29

30. Takeaways
1. Organisations have many reasons to re-use data over time
2. There are unique challenges to preserving data
3. Where possible, save yourself some work and make your
metadata more harvestable and data more understandable by
using international standards like DDI and PREMIS
4. When you use metadata standards like DDI and PREMIS together:
• create generic structures
• use fit-for-purpose standards for specific content
• consider information overlap
• ‘delegate’ metadata capture where possible
30

31. Thanks!
Sally Vermaatensally.vermaaten@stats.govt.nz
Parul Sharma parul.sharma@stats.govt.nz
31