This paper describes an infrastructure for the automated evaluation of semantic technologies and, in particular, semantic search technologies. For this purpose, we present an evaluation framework which follows a service-oriented approach for evaluating semantic technologies and uses the Business Process Execution Language (BPEL) to define evaluation workflows that can be executed by process engines. This framework supports a variety of evaluations, from different semantic areas, including search, and is extendible to new evaluations. We show how BPEL addresses this diversity as well as how it is used to solve specific challenges such as heterogeneity, error handling and reuse. Presented at Data infrastructurEs for Supporting Information Retrieval Evaluation (DESIRE 2011) Workshop, Co-located with CIKM 2011, the 20th ACM Conference on Information and Knowledge Management Friday 28th October 2011, Glasgow, UK http://www.promise-noe.eu/events/desire-2011/

1. Infrastructure and Workflow
for the Formal Evaluation of
Semantic Search
Technologies
Stuart N. Wrigley1, Raúl García-Castro2 and Cassia Trojahn3
1University of Sheffield, UK
2Universidad Politécnica de Madrid, Spain
3INRIA, France
Data infrastructurEs for Supporting Information Retrieval Evaluation:
DESIRE 2011 Workshop

2. SEALS Project
• SEALS: Semantic Evaluation At Large Scale
• EU FP7 funded Infrastructures project
• June 2009 – June 2012.
• Initial areas: ontology engineering, ontology storage and reasoning
tools, ontology matching, semantic web service discovery, semantic
search
tiv ing
Re tivi
• Objectives:
s
Ac o rk
Ac
itie
se ties
tw
ar
– SEALS Platform.
ch
Ne
• A lasting reference infrastructure.
• Evaluations executed on-demand on SEALS Platform.
Service
– SEALS Evaluation Campaigns. Activities
• Two public evaluation campaigns
– SEALS Community.
28.10.2011
2

3. Key (non-technical) features
• Infrastructure characteristics:
– Open (both in terms of use and development – Apache 2.0 license)
– Scalable (to users and data size – cluster-based)
– Extensible (new evals, new tool types, new metrics)
– Sustainable (beyond funded period)
– Independent (unbiased, trustworthy)
– Repeatable (eval results can be reproduced)
• Core criteria:
– Interoperability
– Scalability
– Tool-specific measures (e.g., alignment precision, etc)
28.10.2011
3

4. Evaluation dependencies
Tools
ER
Execution
Evaluation Results
Request
Test data
Evaluation
descriptions
28.10.2011
4

5. Platform architecture
28.10.2011
5

6. Evaluation overview
Infrastructure Tools Tools Infrastructure
request deployment undeployment release
Execution Execution Evaluation Execution
request environment description environment
analysis preparation execution clean-up
ER
Test data Activity Results Test data
stage-in Execution storage stage-out
28.10.2011
6

7. REPOSITORIES
28.10.2011
7

8. Repositories
External Interface
HTTP
File
Request Restlet Resources
Internal Interface
Java Test Data Tools Results
objects Repository Repository Repository
Implementation Implementation Implementation
Storage
RDF RDF File
File
Triples Repository Repository
(Metadata) (Artifacts)
28.10.2011
8

9. Test Data Repository Service (TDRS)
• Storage of, and access to:
– persistent test data sets (aka suites)
– test data generators
• Suites are stored as ZIP files and accompanied by metadata.
• Suites can be versioned.
• ZIP-internal metadata allows structuring and repository-based iteration.
Data
Metadata Artifact
Data
Entity Artifact
Artifact
Item
Discovery Exploitation
28.10.2011
9

10. Results Repository Service (RRS)
• Storage of, and access to, suites of:
– raw results
– interpretations
• Suites stored as metadata and optional ZIP files.
• Metadata allows structuring and linking to ensure backlinks:
– interpretation links to raw result dataItem
– raw result links to tool and test suite dataItem
28.10.2011
10

11. TOOLS
28.10.2011
11

12. Tool wrapping and storage
• Tools are evaluated on within the Platform (ie, locally)
• Tools must have bi-directional communication with Platform.
• Each campaign (e.g., search) defines its own Java API specific to their tool
type.
• Participating tools create a tool ‘wrapper’ to implement API.
• Also includes ‘setup’ and ‘tear down’ scripts and any 3rd party libraries /
packages required.
• Bundles stored in Tool Repository Service (TRS).
• Tutorials on SEALS portal (http://www.seals-project.eu/).
28.10.2011
12

13. Semantic Search API
Method Functionality
boolean loadOntology( Load an ontology
URL ontology,
String ontologyName,
String ontologyNamespace)
void showGUI(boolean show) Switch the GUI on or off
boolean executeQuery(String query) Execute a query
boolean isResultSetReady() Are query results ready?
URL getResults() Retrieve the URL of the results file
boolean isUserInputComplete() Has user hit ‘go’ (or equivalent)?
String getUserQuery() Retrieve the query as entered by the user
28.10.2011
13

14. WORKFLOWS
28.10.2011
14

15. Evaluation overview (workflow)
Infrastructure Tools Tools Infrastructure
request deployment undeployment release
Execution Execution Evaluation Execution
request environment description environment
analysis preparation execution clean-up
ER
Test data Activity Results Test data
stage-in Execution storage stage-out
28.10.2011
15

16. Workflow
• Business Process Execution Language (BPEL)
– orchestrates manipulation of information using (only) web services interfaces.
• All entities involved must be exposed as web services (TDRS, RRS, the tool
wrapper, custom services) defined using WSDL interfaces.
• ‘Custom services’ allow out-of-band processing:
– computation of analyses
– data / metadata manipulation
– timestamping
– etc.
• Workflow defined according to campaign requirements.
• Stored in the Evaluation Repository Service (ERS).
28.10.2011
16

17. Conceptual workflow for search
Start
Start
Ontology No Get Ontology Test Data
Loaded? URL Repository
Load Test Suite Test Data
Yes
Repository
Test Data Tool: Load
Get Query
Repository Ontology
Loaded No
Record error
successfully?
Tool: Execute
Query
Yes
Tool: Switch off
GUI
Tool: Results No
Pause
Ready?
Yes
More test No
Store Results End
cases?
Tool: Get Results
Yes
Store Results Results
Results Repository
Process
Repository
Test Case
End
28.10.2011
17

18. BPEL workflow snippets
28.10.2011 Stored as XML
S
18

19. BPEL exception handling
28.10.2011
19

20. Summary
• SEALS Platform provides functionality to simplify and automate
evaluations.
• Powerful cluster-based compute.
• Storage of test data, results and interpretations in perpetuity.
• Workflows specified in industry-standard BPEL.
• All for free!
28.10.2011
20

21. Thank you for your attention!
http://www.seals-project.eu/

22. Links to resources
• Evaluation campaigns
– http://www.seals-project.eu/seals-evaluation-campaigns/
• Test suites
– Mooney (NL questions)
• http://seals.sti2.at/tdrs-web/testdata/persistent/Mooney+NL+Testsuite/2010/
– EvoOnt (SPARQL questions)
• http://seals.sti2.at/tdrs-web/testdata/persistent/EvoOnt+1K+SPARQL+Testsuite/2010/
• http://seals.sti2.at/tdrs-web/testdata/persistent/EvoOnt+10K+SPARQL+Testsuite/2010/
• http://seals.sti2.at/tdrs-web/testdata/persistent/EvoOnt+100K+SPARQL+Testsuite/2010/
• http://seals.sti2.at/tdrs-web/testdata/persistent/EvoOnt+1000K+SPARQL+Testsuite/2010/
• http://seals.sti2.at/tdrs-web/testdata/persistent/EvoOnt+10000K+SPARQL+Testsuite/2010/
– QALD
• http://seals.sti2.at/tdrs-web/testdata/persistent/QALD-MusicBrainz+SPARQL+Testsuite/2010/
• Last year’s campaign outcomes
– http://www.seals-project.eu/seals-evaluation-campaigns/semantic-search-tools/results-2010
28.10.2011
22