Harvesting metadata in repository networks, including ARIADNE, GLOBE, MACE and MELT

1. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
HARVESTING METADATA
IN REPOSITORY
NETWORKS
Bram Vandeputte
K.U.Leuven
ARIADNE
1
1

2. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
WHO AM I
• Working since 2006 with Erik Duval
• ARIADNE harvester, OAI-PMH target support,
validation service, etc.
• International projects : MACE, MELT, Globe, Ariadne,
Aspect, iCoper,...
• Other interests : Information visualisation,
Interoperability, TEL
2
2

3. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
Overview
• OAI-PMH
• Validation Service
• Harvester Infrastructure
• Projects
• MACE
• MELT
• GLOBE
• OAI-PMH Target
3
3

4. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI-PMH
• Server - Client model
• Pull mechanism
• options :
• selective harvesting
• incremental harvesting
4
4

5. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI-PMH
• Verbs : Identify, ListRecords, GetRecord
• Parameters :
• baseUrl
• from & until date
• metadataPrefix
• sets
5
5

6. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
Validation Service
• Interoperability : Application Profile (AP)
• Manual check : very time consuming
• Need a tool for enforcing an AP => validation scheme
• A set of validation rules
• Reusable
6
6

7. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
Validation Service
LOM loose
MELTv1.0/
MACEv1.0
minimal
melt.xsd lomloose.xsd mace.xsd
classification
vocab service
MELTv1.0/full vcard validator
Legend
empty attribute uses
fields
extends
MACEv1.0 validationScheme
conditional LRE taxon field
fields check numerical vcard validator validation component
7
7

8. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
8
8

9. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
The ARIADNE Harvester
Infrastructure
8
8

10. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
harvester log
8
8

11. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
ARIADNE Repository
SPI SQI
harvester log
validation service
8
8

12. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
ARIADNE Repository
SPI SQI
External Repository
OAI
OAI-PMH
LOM
LOM
LOM
harvester log
validation service
8
8

13. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
ARIADNE Repository
OAI-PMH
SPI SQI
External Repository
LOM
LOM
LOM
OAI
OAI-PMH
LOM
LOM
LOM
harvester log
validation service
8
8

14. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
ARIADNE Repository
OAI-PMH
SPI SQI
External Repository
LOM
LOM
OAI
OAI-PMH
LOM
LOM
LOM
LOM
harvester log
validation service
8
8

15. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
ARIADNE Repository
OAI-PMH
SPI SQI
External Repository
LOM
LOM
OAI
OAI-PMH
LOM
LOM
LOM
LOM
LOM
harvester log
validation service
8
8

16. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
ARIADNE Repository
OAI-PMH
SPI SQI
External Repository
LOM
LOM
OAI
OAI-PMH
LOM
LOM
LOM
LOM
Validation
Msg
harvester log
validation service
8
8

17. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
ARIADNE Repository
OAI-PMH
SPI SQI
External Repository
LOM
LOM
OAI
OAI-PMH
Validation
LOM
Msg
LOM
LOM
LOM
harvester log
validation service
8
8

18. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
ARIADNE Repository
OAI-PMH
SPI SQI
External Repository
LOM
LOM
OAI
OAI-PMH
LOM
LOM
LOM
LOM
Validation
Msg
harvester log
validation service
8
8

19. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
ARIADNE Repository
OAI-PMH
SPI SQI
External Repository
LOM
LOM
OAI
LOM
OAI-PMH
LOM
LOM
LOM
Validation
Msg
harvester log
validation service
8
8

20. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
ARIADNE Repository
OAI-PMH
SPI SQI
External Repository
LOM
LOM
LOM
OAI
2 6 LOM
OAI-PMH
1
Validation
LOM
Msg
LOM
LOM
LOM
4
5 3
Validation Validation
LOM
Msg Msg
harvester log
validation service
9
9

21. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
Harvesting in networks
• MACE
• MELT
• GLOBE
10
10

22. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
MACE
• Metadata for Architectural Contents in Europe
• www.mace-project.eu
• Harvesting, Searching, Enriching, (geo-) Tagging,
attention metadata
11
11

23. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
MACE
Uses Lucene for index search
WINDS
Validation OAI-PMH
service
SPI
Harvested
metadata OAI-PMH
SPI
IRB
ARIADNE
Harvester
OAI-PMH
MACE
metadata
SPI OAI-PMH DYNAMO
Enriched
metadata CuminCAD
12
12

24. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
MACE
13
13

25. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
MELT
• MELT : a Metadata Ecology for Learning and Teaching
• Ministries of education across europe
• enriching, translating, better quality metadata
• practical test for scalability of harvester framework
14
14

26. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
MELT
Multilingual
Vocabulary
Thesaurus
Registry Validation BMBWK EUN MEC RVP
Enrichment service
Toolkit
EUN AMG
OAI-PMH
SQI ARIADNE C-H FWU MESC TLF
Harvester
EUN MDS
SPI
CFL INDIRE MSU UL-FRI
ARIADNE OAI
Harvester SQI
(EUN)
ARIADNE
METADATASTORE EDUCATIO KUL NCTE XTEC
15
15

27. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
16
16

28. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
GLOBE
• Global Learning Objects Brokered Exchange
• Worldwide network of networks of repositories
• => Both SQI and OAI-PMH are used on various levels
in the network
17
17

29. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
GLOBE
BMBWK EUN MEC RVP
SQI UDDI OAI
OAI-PMH
Registry SQI ARIADNE C-H FWU MESC TLF
SQI Client Harvester
OAI CFL INDIRE MSU UL-FRI
MELT
SQI SQI
Federated EDUCATIO KUL NCTE XTEC
GLOBE Search
SPI OAI
OAI-PMH
ARIADNE
SQI Harvester
WINDS DYNAMO
ARIADNE
Harvester
MACE IRB CuminCAD
OAI OAI OAI SQI SQI SQI SQI
SQI SQI SQI
LACLO NIME EUN EdNA CGIAR Pubelo Merlot
OAI OAI SQI SQI SQI SQI
SQI SQI
ISKME KERIS III COSL LORNET Ariadne
18
18

30. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
19
19

31. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI-PMH target
20
20

32. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI-PMH Target Software
2. Mapping Process
Repository metadata Repository LOM metadata
1. Get metadata - identifier - lom.general.identifier
out of database - title - lom.general.title
- url - lom.general.description
- project description - lom.technical.location
- ... - ...
3. Copy results in
OAI-PMH service
Repository OAI-PMH result
DYNAMO LOM 4. Serve results ARIADNE
DYNAMO LOM
metadata LOM Harvester
DYNAMO
metadata
Repository LOM
metadata
metadata
21
21

33. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• Questions ?
22
22

34. ARIADNE K.U.Leuven
www.ariadne-eu.org www.cs.kuleuven.be/~hmdb
References
• SPI : http://ariadne.cs.kuleuven.be/lomi/index.php/SimplePublishingInterface
• SQI : http://ariadne.cs.kuleuven.be/lomi/index.php/LorInteroperability
• IEEE LOM : http://ltsc.ieee.org/wg12/
• OAI-PMH : http://www.openarchives.org/
• OAI-PMH target wiki :
http://ariadne.cs.kuleuven.be/lomi/index.php/Setting_Up_OAI-PMH
23
23