Presentation given at TDWG 2013 conference on our visual tool to spot data quality issues in Darwin Core Archives http://www.vesper.org.uk

1. VESPER
Visual Exploration of Species-Referenced Repositories
Martin Graham & Jessie Kennedy
Edinburgh Napier University

2. Introduction
• VESPER – an exploration into data quality issues for
Darwin Core Archives (DWCA)
• DWCA’s are files for storing detailed species-based
data sets
• How does a user know which data sets are useful
and complete?

3. Valid vs. Useful
• GBIF has tools to test DWCA validity
• This work is about visualising data we assume is
“valid” but are unsure of “usefulness”
– Taxonomy is broken
– Dates are wrong
– Lions in the sea
• In many cases the usefulness of such data is only
seen when visualised in context

4. Approach
• Web-based visualisation of DWCAs
– Uses HTML5
• SVG, CSS3, FileWriters, ArrayBuffers
– D3 toolkit
– Client side only
• Visualise basic dimensions of data
– Taxonomy
– Geography
– Time
– & Miscellaneous Stats

5. Darwin Core Archives
Eml.xml
Meta.xml
Core
Taxa/Occurrence Data
Extension
Extension
Meta Files (XML)
Data Files (CSV)
Describes
Exactly one
Zero or more
Extension ID == Core ID

6. Zapped by Zip
• Zip files make things smaller
– Good for network transport
– But analysing the data means we have to make things big
again
Expand a lot
Expand even more
(String copying, UTF-16 etc)

7. Zip Zapped
• Partial Unzip
• Analyse fields listed in meta file
– Disregard verbose fields
• Find combinations of fields that can be used
to generate a visualisation
Implicit Taxonomy acceptedNameUsageID, parentNameUsageID
Explicit Taxonomy Any of Kingdom, order, family, genus etc
Map decimalLongitude, decimalLatitude
Timeline eventDate
• List choice of available visualisations for a
meta.xml and just extract chosen fields

8. Taxonomy
• Sunburst / Icicle plot
– Some difficulties with high fan-out taxa
– Though a lot of these are data quality issues

9. Taxonomy
• Sunburst / Icicle plot
– Some difficulties with high fan-out taxa
– Though a lot of these are data quality issues

10. Geography
• Based on popular leaflet.js library
– And Markercluster plugin
– Some adaptations to show selected items

11. Temporal
• Simple bar chart
– With rangeslider
– Zoom in and see yearly patterns (i.e not much at xmas)

12. Miscellaneous
• Sanity check - Empty data count

13. Miscellaneous
• Taxonomic fan-out for hollow curve anomalies
• Export selected IDs
– These can be saved or sent somewhere else

14. Selection
• Selections in one view are reflected in the other
views for the same data
– Multiple views, linking

15. Conclusion
• Javascript visualisations for DWCA archives
• Quickly shows areas of quality issue
• Can handle large archives if only key fields are
analysed

16. Fin
• http://www.soc.napier.ac.uk/~cs22/vesperDemo/ves
per/demoNew.html
– Feedback welcome
• Thanks to GBIF, Canadensys, EMBL for data
• Funded by BBSRC
• Ask for a demo