How to Publish Open Data

Digital Enterprise Research Institute www.deri.ie

How to publish Open Data

Richard Cyganiak
Opening Up Government Data – Galway, 8 Nov 2011

Stefan.Decker@deri.org
http://www.StefanDecker.org/

Copyright 2010 Digital Enterprise Research Institute. All rights reserved.

TimBL’s 5-star plan for open data

★Make your stuff available on the Web
★★Make it available as structured data
(e.g., an Excel sheet instead of image scan of a table)

★★★Use a non-proprietary format
(e.g., a CSV file instead of an Excel sheet)

★★★★Use linked data format
(i.e., URIs to identify things, and RDF to represent data)

★★★★★Link your data to other people’s data to provide
context

Source: http://inkdroid.org/journal/2010/06/04/the-5-stars-of-open-linked-data/

Five-shamrock scheme


 1. Publish data on the web



 2. Publish data in a machine-processableformat



 2. Publish data in a machine-processable format

 3. Use an open standard format





 4. Publish under an open license





 4. Publish under an open license

 5. List your data in a data catalog


1. Publish data on the web

Why?

 The web is where people look for it first
 Google can index it
 Less phone calls and emails (and FoI requests) to
answer

Lots of data is already there

 Databases
 Reports
 Spreadsheets
 Maps


2. Publish data
in a machine-
processableformat

Why?

 Allow others to do their own processing, analysis and
visualisation of your data
 New services, new ideas

Examples

 CSO Quarterly National Household Survey
 http://cso.ie/qnhs/calendar_quarters_qnhs.htm
 EPA enforcement files and ScraperWiki
 http://www.epa.ie/whatwedo/enforce/lic/info/
 https://views.scraperwiki.com/run/irish-epa-visuals/
 Galway and Fingal planning applications
 http://lab.linkeddata.deri.ie/2010/planning-apps/
 Getting the data: 210 lines of code vs. 30 lines of code

Symptom: screenscraping

 People use tools like ScraperWiki to get at data that isn't
machine-readable
 https://scraperwiki.com/tags/ireland
 Scraping is not the right way of doing this
 Expensive
 Brittle
 Strain on computing resources

Formats

 Good: MS Excel, CSV, XML, JSON, Microdata
 Not so good: Pure websites, MS Word
 Bad: PDF
 Really bad: Only charts/maps without numbers

Good practices

 Publish in multiple formats, at least one machine-
readable
 Publish Excel files alongside large PDF reports
 Publish CSV alongside database-backed web
applications


3. Use an open standard format

Why?

 Not all formats are created equal
 Some formats bring many tools and applications that
people can already use

Quick tour of formats

 CSV – Comma-Separated Values
 More open (and simpler) alternative to Excel format
 Can be opened in and exported from Excel, Google
Spreadsheets, Google Refine, …
 KML – Keyhole Markup Language
 Simple format for presenting geographic data
 Can be opened in Google Maps
 RSS – Really Simple Syndication
 Notifications of updates of any kind
 Can be opened in RSS readers and many email clients

Developer-oriented formats

 XML – Extensible Markup Language
 W3C (World Wide Web Consortium) standard, 1997
 established, reliable, ubiquitous
 JSON – Javascript Object Notation
 IETF (Internet Engineering Task Force) standard, 2006
 great for web APIs
 very simple; very fashionable right now
 RDF – Resource Description Framework
 W3C standard, 2004
 great for data integration
 steeper learning curve

Also: standard classifications

 Within your data, use the same categories as everybody
else
 CSO
 http://www.cso.ie/surveysandmethodologies/classifications_stan.
htm
 StatCentral list of classifications
 http://www.statcentral.ie/classifications.asp

Also: standard identifiers

 Example: School roll numbers
 Department of Education publishes an Excel file with all school
roll numbers
 Can be used to Google the same school on other websites,
school evaluation reports etc
 Example: Ordnance Survey UK geo identifiers
 Uses URIs (web addresses) as identifiers
 http://data.ordnancesurvey.co.uk/doc/7000000000037256
 Great for use in RDF

Linked Open Data Cloud

Summary

 Prefer open, widely used standards
 But: also prefer what you know best
 Support multiple formats for different audiences where it
makes sense
 Great: CSV, KML, RSS, XML, JSON


4. Publish under an
open license

Why?

 Regulates what others can and cannot do with the data
 For re-users, uncertainty about rights is a major concern
 A good way to ensure that your organisation gets
acknowledged
 You need some non-discriminatory policy for giving
rights to the data anyway (PSI directive)

Complex topic

 Destroying a potential income stream?
 Content licenses vs database licenses
 Mixing and compatibility of licenses
 Wikipedia, OpenStreetMap

Irish PSI License

 Created in response to PSI Directive
 Available at http://psi.gov.ie/
 Problems: Documents may not be used “for the principal
purpose of advertising or promoting a particular product
or service”
 Can't be combined with Wikipedia or OpenStreetMap
 Not an open license according to Open Definition
 http://opendefinition.org/

Open database licenses

http://opendefinition.org/licenses/

License features

 You're allowed to do pretty much anything, provided
you…
 Attribution (“By”) – give credit
 ShareAlike (“SA”) – adapted data must be published in
the same way

Does Open Data have to be free?

 Many would say yes
 A matter of terminology and definitions
 Either way there is nothing wrong with charging for
certain data

Data protection

 Personal information is not open data
 Freedom of Information legislation
 http://foi.gov.ie/

Summary

 Stating an explicit license is important
 Irish PSI License: It's readily available, but not “open
enough” for some applications
 Open Data Commons licenses with various constraints


5. List your data in a
data catalog

Why?

 So that people know it exists
 This is how the world learns about available data
 This is how you learn what they do and need

Some key information about a dataset

 What data is being published?
 What's the license?
 When was the data collected?
 When will it be updated, if at all?
 How was/is this data collected?
 What was/is the data used for?
 Contact person?
 Where to give feedback?

How to do this in practice?

 Have a simple page on your website
 Use an open community data catalog
 Set up your own catalog
 Use a national Irish data catalog???

Open community catalogs

 The Data Hub
 http://thedatahub.org
 Irish CKAN
 http://ie.ckan.net

Set up your own catalog

 Requires a budget
 Roll your own software?
 data.fingal.ie
 Use open source, e.g., CKAN?
 data.gov.uk
 Berlin Open Data
 …

National Irish data catalog?

 CSO'sStatCentral?
 Marine Institute's ISDE?
 Who publishes the catalog in other countries?
 UK: Cabinet Office
 US: White House
 Australia: Dept of Finance and Deregulation
 New Zealand: Dept of Internal Affairs

Summary

 Data catalogs make it easy to find data
 Basic metadata, how to give feedback etc
 Important: How often are datasets accessed?
 “Request a dataset” feature
 Also: Open Data Ireland Google Group
 http://groups.google.com/group/open-data-ireland

How to Publish Open Data

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a How to Publish Open Data

Semelhante a How to Publish Open Data (20)

Mais de Richard Cyganiak

Mais de Richard Cyganiak (9)

Último

Último (20)

How to Publish Open Data