This document discusses consuming and using open and linked data with open source tools. It outlines the available data and distribution technologies like RDF, SPARQL and linked data. It then explains the typical software stack used including tools for querying, storing and visualizing the data. The document provides an example of actually retrieving data and doing something useful with it. It also notes some potential issues or "gotchas" and provides further resources for working with open data.

1. 5/31/2011
Consuming Open and
Linked Data with Open
Source Tools
Or, how to do something useful
with all this free data
Jo Cook, Geospatial Consultant, Astun Technology

2. Talk Outline
● The data and the distribution technologies
explained
● The software stack explained
● Actually getting some data and doing something
with it
● Gotchas
● Further resources and links

3. The Data and Technologies

4. The Data and Technologies

5. The Data and Technologies
Point in Latitude/Longitude format (WGS84)

6. The Data and Technologies

7. The Data and Technologies

8. The Data and Technologies
PREFIX naptan: <http://transport.data.gov.uk/def/naptan/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>
SELECT DISTINCT ?stopPoint_airport ?name ?iataRef ?lat ?long
WHERE {
?item a naptan:StopArea ;
naptan:administrativeArea <http://transport.data.gov.uk/id/administrative-
area/145> ;
skos:prefLabel ?name ;
geo:lat ?lat ;
geo:long ?long ;
naptan:stopPoint [ naptan:airport ?stopPoint_airport ] .
?stopPoint_airport naptan:iataRef ?iataRef .
}

9. The Data and Technologies

10. The Data and Technologies

11. The Data and Technologies
http://wallpapertree.com/wallpaper/Sad-Kitten/

12. The Data and Technologies

13. The Data and Technologies

14. The Data and Technologies

15. The Data and Technologies
http://freeimagesarchive.com

16. The Data and Technologies

17. The Data and Technologies
1m4ge.blogspot.com

18. The Data and Technologies

19. The Data and Technologies

20. Actually doing something with the data

21. Actually doing something with the data

22. Gotchas
smashingmagazine.com

23. Gotchas

24. Mitigation
Keep your data here so you're
not relying directly on the service
for updates to customers
(Maybe even
keep it here too)

25. Conclusions
A dash of all these, some readily
available python libraries, and
some relatively simple scripting
gets you useful data for your own
applications...
... and by using PostgreSQL to
store your data, you can provide
a buffer in case of api changes or
outages...
... and even do some nifty
manipulation to analyse and
present the data differently!

26. Conclusions
Simples!

27. Conclusions
Not *that* simple, but...

28. Resources
Linked Data Resources and Example Queries
Best API for transport data: http://openuplabs.tso.co.uk/
O'Reilly Book: http://www.learningsparql.com/
Blog: http://talis-systems.com/blog/
Airport Query: http://getthedata.org/questions/137/where-can-i-get-a-list-of-all-airports-codes-names-locations
Beginner Guide: http://www.pezholio.co.uk/2011/01/a-beginners-guide-to-sparqling-linked-data-part-1/
NHS Choices- RSS
http://www.nhs.uk/servicedirectories/Pages/ServiceSearch.aspx
Police Data- JSON/REST
API: http://policeapi2.rkh.co.uk/api/docs/
CSV/SHP: http://www.police.uk/data
Basic Software Resources
Mapserver dynamic charting: http://mapserver.org/output/dynamic_charting.html
Python SparQLWrapper: http://sparql-wrapper.sourceforge.net/
Python SimpleJSON: https://github.com/simplejson/simplejson
Python PygreSQL: http://www.pygresql.org/
Loader: https://github.com/AstunTechnology/Loader

29. Thank You!
www.archaeogeek.com
@archaeogeek
jocook@astuntechnology.com