1. Unified Digital Format Registry
a semantic registry for digital preservation
Digital Library Federation Forum
Baltimore, October 31-November 2, 2011
UDFR: A Semantic Registry for Format
Representation Information
Lisa Dawn Colvin
Abhishek Salve
Stephen Abrams
UC Curation Center
California Digital Library

2. Unified Digital Format Registry
a semantic registry for digital preservation
Outline
 What
 Why
 How
 When

3. Unified Digital Format Registry
a semantic registry for digital preservation
Why formats?
“Format” is the dividing line between bits and
information
ffd8ffe000104a46 SOI
4946000102010083 APP0 JFIF 1.2
00830000ffed0fb0 APP13 IPTC
50686f746f73686f APP2 ICC
7020332e30003842 DQT
494d03e90a507269 SOF0 183x512
6e7420496e666f00 DRI
Syntax Semantics
0000007800000000 DHT
0048004800000000 SOS
02f40240ffeeffee ECS0
0306025203470528 RST0
03fc000200000048 ECS1
00480000000002d8 RST1
0228000100000064 ECS2
0000000100030... ...

4. Unified Digital Format Registry
a semantic registry for digital preservation
Why formats?
There are many necessary preservation activities that
can be usefully performed on bits qua bits
But to preserve information you most act on
formatted bits and know what those formats mean
• Preservation of syntax and semantics

5. Unified Digital Format Registry
a semantic registry for digital preservation
Unified Digital Format Registry
“A reliable, publicly accessible, and sustainable
knowledge base of file format representation
information for use by the digital preservation
community”
• “Unification” of the function and holdings of PRONOM
and GDFR
http://www.nationalarchives.gov.uk/PRONOM
http://gdfr.info/
• Open source platform / GPL
• Semantic wiki
• Funded by the Library of Congress

6. Unified Digital Format Registry
a semantic registry for digital preservation
Timeline
PRONOM – National Archives [UK], 2002
http://www.nationalarchives.gov.uk/PRONOM
“ready access to reliable technical information about the
nature of electronic records”
JHOVE – Harvard, 2003
http://hul.harvard.edu/jhove
“digital object validation and characterization”
GDFR – Harvard/OCLC, 2006
http://gdfr.info/
“a distributed and replicated registry of format information
populated and vetted by experts and enthusiasts world-
wide”

7. Unified Digital Format Registry
a semantic registry for digital preservation
Timeline
UDFR – Ad hoc stakeholder community, 2009
• Resolve PRONOM IPR issues and develop a community-
supported open source solution
• Advance beyond legacy RDBMS and XML database
technology
UDFR – CDL, January 2011
http://udfr.org/
“a semantic registry for digital preservation”
• Stakeholder meeting, April 2011
• Beta release, November 2011
• Production release, January 2012

8. Unified Digital Format Registry
a semantic registry for digital preservation
Representation information
What you need to know about something in order to
exploit that thing meaningfully [OAIS/ISO 14720]
Information that lets you answer important
preservation questions
• What format is it?
• What are its significant properties?
• Is it valid?
• Is it at risk?
• How can I render/play/read it?
• What can it be transformed into?
• And how?

9. Unified Digital Format Registry
a semantic registry for digital preservation
Why semantic?
Everyone wants to say something about everything
• The semantic web lets anyone say anything about
anything
• Understandable to both people and machines

10. Unified Digital Format Registry
a semantic registry for digital preservation
Data modeling
Abstract Controlled
Base Vocabulary …
holder
dependency
holder creator
owner Abstract product Abstract
Process IPR Agent Holding Digest
Product Signature
maintainer reference
embodies ipr specification file
digest
Abstract External
Software Hardware Media Document File
Format Signature
Internal
Signature
input / output signature
Character Compression
Assessment Grammar File Format
Encoding Algorithm
grammar
assessment

11. Unified Digital Format Registry
a semantic registry for digital preservation
Provenance
“Trust, but verify”
• Complete change history
at the assertion level,
including
– Who made the assertion, and when?
– Confidence based on personal and institutional
reputation
• Imprimatur by technically knowledgeable
reviewers

12. Unified Digital Format Registry
a semantic registry for digital preservation
Ontologies
Prefixu Namespace
udfrs http://udfr.org/onto#
udfr http://udfr.org/udfr/
dc http://purl.org/dc/elements/1.1/
dcterms http://purl.org/dc/terms/
foaf http://xmls.com/foaf/0.1/
owl http://www.w3.org/2002/07/owl#
pronom http://reference.data.gov.uk/technical-registry/
rdf http://www.w3.org/1999/02/22-rdf-syntax-ns#
rdfs http://www.w3.org/2000/01/rdf-schema#
skos http://www.w3.org/2004/02/skos/core#
xds http://www.w3.org/2001/XMLSchema#

13. Unified Digital Format Registry
a semantic registry for digital preservation
Technology stack
HTTP / SPARQL
JavaScript / CSS
Ontowiki Erfurt / RDFAuthor
http://aksw.org/Projects/Erfurt
http://ontowiki.net/
https://github.com/AKSW/RDFauthor
Zend framework Virtuoso 4store
http://www.zend.com/ http://virtuoso.openlinksw.com/
PHP RDF
http://www.php.net/ http://www.w3.org/RDF
Apache httpd
http://httpd.apache.org/

14. Unified Digital Format Registry
a semantic registry for digital preservation
Initial population
Export from PRONOM
• Working with TNA to identify appropriate subset
• Transform to cross-walk modeling differences

15. Unified Digital Format Registry
a semantic registry for digital preservation
Licensing
Code is available under GPLv3
http://www.gnu.org/copyleft/gpl.html
• Hosted on BitBucket
http://www.bitbucket.org/udfr
Data is contributed and available under CC-BY
http://creativecommons.org/licenses/by/3.0/
• Consistent with UK open government license applicable
to PRONOM data
http://www.nationalarchives.gov.uk/doc/open-government-licence

16. Unified Digital Format Registry
a semantic registry for digital preservation
Demo

17. Unified Digital Format Registry
a semantic registry for digital preservation
Lessons learned
 People with semantic experience are scarce
 Too much time evaluating/prototyping potential
technology choices
 More difficulty than anticipated integrating disparate
open source products
 0.x software is often numbered that for a reason
 Feature lists aren’t (always)

18. Unified Digital Format Registry
a semantic registry for digital preservation
Lessons learned
 Availability of a worldwide selection of products is a
good thing (except when you don’t read German)
• Excellent support from AKWS/Universität Leipzig
 Modeling differences
• RDF (non-)standards
 VM deployment
• Disparate IT organizations supporting dev/prod instances

19. Unified Digital Format Registry
a semantic registry for digital preservation
Next steps
 Long-term governance and operational support
 Technical maintenance and enhancement
 Replication/synchronization
 Building contributor and reviewer communities

20. Unified Digital Format Registry
a semantic registry for digital preservation
For more information
UDFR UC3
http://udfr.org/ http://www.cdlib.org/uc3
http://bitbucket.org/udfr uc3@ucop.edu
Stephen Abrams Mark Reyes
PRONOM Lisa Colvin Abhishek Salve
http://www.nationalarchives.gov.uk/PRONOM Patricia Cruse Tracy Seneca
Scott Fisher Joan Starr
GDFR Erik Hetzner Carly Strasser
http://gdfr.info/ Greg Janée Marisa Strong
John Kunze Adrian Turner
OntoWiki Margaret Low
David Loy
Perry Willett
http://ontowiki.net/Projects/OntoWiki
Virtuoso
http://www.openlinksw.com/dataspace/dav/wiki/Main/VOSRDFWP
Agile Knowledge and Semantic Web (AKSW), Universität Leipzig
http://aksw.org/