Together with Marco Büchler. Presentation at CAS 2018.

1. Measuring Completeness
as Metadata Quality Metric in Europeana
Péter Király1 – Marco Büchler2
1Göttingen eResearch Alliance (GWDG) – pkiraly@gwdg.de, @kiru
2Georg-August-Universität Göttingen – mbuechler@etrap.eu, @mabuechler
IEEE Big Data 2018, 3rd Computational Archival Science (CAS) workshop
http://dcicblog.umd.edu/cas/ieee-big-data-2018-3rd-cas-workshop/
Seattle, WA, Wednesday, Dec. 12, 2018
slides URL: http://bit.ly/qa-cas2018

2. the problem
https://twitter.com/fxru/status/1052838758066868224
http://bit.ly/qa-cas2018
2

3. top 20 patterns, ‘date’ field, MoMa collection
Harald Klinke (LMU München) https://twitter.com/HxxxKxxx/status/1066805548866289664
3
http://bit.ly/qa-cas2018

4. generic title and bad thumbnail
4
http://bit.ly/qa-cas2018

5. multilinguality problem
5
★ Mona Lisa → 456
results
★ La Gioconda → 365
results
★ La Joconde → 71
results
http://www.europeana.eu/portal/en/record/90402/RP_F_00_351.html
http://bit.ly/qa-cas2018

6. strange values
6
from a template?
more examples in Report and Recommendations from the Task Force on Metadata Quality (2015)
http://bit.ly/qa-cas2018

7. consequence of metadata quality issues
7
main purpose of metadata: to access content
vs.
no metadata
no access to data no data usage
more explanation:
Data on the Web Best Practices, W3C Working Draft, https://www.w3.org/TR/dwbp/
bad metadata
http://bit.ly/qa-cas2018

8. purpose of assessment
8
we feel that there are “good” and “bad” metadata
records
we would like to achieve metrics like this:
functional requirements
good
acceptable
bad
http://bit.ly/qa-cas2018

9. metadata quality metrics in literature
★ completeness: number of metadata elements filled out
★ accuracy: data correspond to the resource that is being described
★ consistency: values compliant to what is defined by the metadata scheme
★ objectiveness: values describe the resource in an unbiased way
★ appropriateness: values are facilitating the deployment of search
★ correctness: syntactically and grammatically correct language
★ ...
Bruce and Hillman (2004); Ochoa and Duval (2009); Palavitsinis (2014); Zaveri et al. (2015)
https://www.zotero.org/groups/488224/metadata_assessment
9
http://bit.ly/qa-cas2018

10. Measuring Europeana
http://bit.ly/qa-cas2018

11. hypothesis
11
by measuring structural elements we
can approximate metadata record quality
≃ metadata smell
http://bit.ly/qa-cas2018

12. organisational proposal
12
Europeana Data Quality Committee
★ Analysing/revising metadata schema
★ Functional requirement analysis
★ Problem catalog
★ Multilinguality
others: DLF Metadata Assessment Group, ADOCHS
http://bit.ly/qa-cas2018

13. technical proposal
13
“Metadata Quality Assessment Framework”
a generic tool for measuring metadata quality
★ adaptable to different metadata schemes
★ scalable (to Big Data)
★ understandable reports for data curators
★ open source
http://bit.ly/qa-cas2018

14. measuring workflow
14
★ OAI-PMH
★ Europeana API
★ Hadoop
★ NoSQL
★ Spark
★ Hadoop
★ Java
★ Apache Solr
★ Spark
★ R
★ PHP
★ D3.js
★ highchart.js
★ NoSQL
json csv json, png html, svg
ingest measure statistical
analysis
web
interface
http://bit.ly/qa-cas2018

15. What to measure?
15
★Structural and semantic features
Completeness, cardinality, uniqueness, length, dictionary entry, data type
conformance, multilinguality (generic metrics)
★Functional requirement analysis / Discovery scenarios
Requirements of the most important functions
★Problem catalog
Known metadata problems
http://bit.ly/qa-cas2018

20. multilinguality
20
<#record> a ore:Proxy ;
dc:subject “Ballet”, “Opera” .
<#record> a ore:Proxy ; edm:europeanaProxy true ;
dc:subject <http://data.europeana.eu/concept/base/264>
, <http://data.europeana.eu/concept/base/247> .
<http://data.europeana.eu/concept/base/264> a skos:Concept .
skos:prefLabel "Ballett"@no, "बैले"@hi, "Ballett"@de, "Балет"@be, "Балет"@ru
, "Balé"@pt, "Балет"@bg, "Baletas"@lt, "Balet"@hr, "Balets"@lv .
<http://data.europeana.eu/concept/base/247>
skos:prefLabel "Opera"@no, "ओपेरा (गीतिनाटक)"@hi, "Oper"@de, "Ooppera"@fi
, "Опера"@be, "Опера"@ru, "Ópera"@pt, "Опера"@bg, "Opera"@lt .
0
0
11 19
Distinct languages Tagged literals 1,7 Literals per language
dereferencing
http://bit.ly/qa-cas2018

22. 19%
58%
63%
13.3% 23.7%

27. counter example

29. K-means clustering
Spark (Scala)
increasing number of clusters
decreasing the distance from
the centroids
after a point this gain is not
so big (“elbow effect”) -- in
theory
Big number or low
quality records
small clusters with ‘in
between’ quality records
the acceptable average
clusters with good quality
records
29
http://bit.ly/qa-cas2018

30. more information
quality dashboard: http://144.76.218.178/europeana-qa/
https://pro.europeana.eu/project/data-quality-committee
https://github.com/pkiraly (GPL-3.0, binaries, scripts)
http://pkiraly.github.io, https://twitter.com/kiru
Would you like to cooperate? (I do!) peter.kiraly@gwdg.de
30
http://bit.ly/qa-cas2018