The document discusses the HOBBIT platform for benchmarking big data platforms. It aims to provide a unified benchmarking platform as a community-driven effort. The platform will include reference datasets and implementations of key performance indicators to standardize benchmarking and allow comparison of results. It will focus on benchmarking tasks related to big linked data and the entire data lifecycle.
Hobbit presentation at Apache Big Data Europe 2016
1. Unified Benchmarking of Big Data Platforms
The HOBBIT Platform
Axel-Cyrille Ngonga Ngomo
Horizon 2020
GA No 688227
01/12/2016–30/11/2018
Apache Big Data
Sevilla, Spain
November 15, 2016
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 1 / 44
3. Summary
Rationale
A community-driven unified benchmarking platform for the community
Focus on Big (Linked) Data
Provide benchmarks and baselines
Provide reference implementation of KPIs
Extensible and referenceable
Result analysis
Open-Source
http://project-hobbit.eu
@hobbit_project
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 2 / 44
4. A Lot of Data
1
1http://www.ibmbigdatahub.com/infographic/four-vs-big-data
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 3 / 44
5. A Lot of Tools
2
2https://cloudramblings.me/
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 4 / 44
6. A Lot ... of Tools
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 5 / 44
7. Questions
Developers: How good is my tool?
Vendors: Who is my tool good for?
Users: Which tool(s) should I use for
my application?
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 6 / 44
8. Many Questions
Where are the current bottlenecks?
Which steps of the data lifecycle are
critical?
Which solutions are available?
Which key performance indicators
are relevant?
How well do or should tools
perform?
How do existing solutions perform
w.r.t. relevant indicators?
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 7 / 44
9. A Lot of Views
4
4https://steemit.com/philosophy/@l0k1/
subjectivity-and-truth-how-blockchains-model-consensus-buildingNgonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 8 / 44
11. Solution
Benchmark
Components
Dataset(s), e.g., Twitter stream, sensor data
Task(s), e.g., entity recognition, storage
Performance indicators, e.g., precision, recall, queries per second
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 9 / 44
12. Solution
Benchmark
TPCH-H (3,000 GB Results): −5.6 × 106
QphH between 2014 and 20165
QALD: ≈ 5% increase in Micro F-Measure
ACE2004: ≈ 6% increase in Micro F-measure
5http://www.tpc.org/tpch/results/tpch_perf_results.asp
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 10 / 44
13. Challenges
Dataset and KPI Mismatch
Year
ACE
Wiki
AQUAINT
MSNBC
IITB
Meij
AIDA/CoNLL
N3
collection
KORE50
Wiki-Disamb30
Wiki-Annot30
SpotlightCorpus
SemEval-2013task12
SemEval-2007task7
SemEval-2007task17
Senseval-3
NIF-basedcorpus
Microposts2014
Softwareavailable?
Webserviceavailable?
Cucerzan 2007
Wikipedia
2008 *
Miner
Illinois Wikifier 2011 *
Spotlight 2011
AIDA 2011 **
TagMe 2 2012
Dexter 2013
KEA 2013
WAT 2013
AGDISTIS 2014
Babelfy 2014
NERD-ML 2014
BAT-
2013 *
Framework
NERD
2014
Framework
GERBIL 2014 *
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 11 / 44
14. Challenges
Unclear KPI Semantics
Example
Federated queries in distributed storage solutions
Which time do we measure?
First or last result?
With or without network delay?
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 12 / 44
15. Challenges
Unclear KPI Semantics
Example
Entity recognition and linking
When is an annotation correct?
Weak or strong annotation?
Semantically equivalent or exact URI?
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 13 / 44
16. Solution!
Unified Benchmarking Framework
Rationale
Provide all benchmark components in one package
Include reference datasets and baselines
Devise standardized tasks and reference KPI implementations
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 14 / 44
17. Solution!
Unified Benchmarking Framework
Rationale
Provide all benchmark components in one package
Include reference datasets and baselines
Devise standardized tasks and reference KPI implementations
Benchmark Core
Web service calls
Dataset
Wrapper
Web service calls
Interface View
Annotator
Wrapper
Interface View
Open Datasets
Configuration
(Model)
...
Benchmark Core
Your
Annotator
Your Dataset
DataHub.io
GERBIL
Core
Controller
Persistent
Experiment Database
(Model)
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 14 / 44
18. GERBIL
HOBBIT v0.1
Features
Unified benchmarking platform
for NER/NEL
18 reference annotation systems
32 reference datasets
Reference implementations of
KPIs
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 15 / 44
19. GERBIL
HOBBIT v0.1
Features
Unified benchmarking platform
for NER/NEL
18 reference annotation systems
32 reference datasets
Reference implementations of
KPIs
Advantages
Benchmarking ≈ 30× faster
Archiving of results
Citeable URIs
Additional analysis
http://gerbil.aksw.org
http://github.org/aksw/gerbil
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 15 / 44
20. GERBIL
HOBBIT v0.1
Features
Unified benchmarking platform
for NER/NEL
18 reference annotation systems
32 reference datasets
Reference implementations of
KPIs
Advantages
Benchmarking ≈ 30× faster
Archiving of results
Citeable URIs
Additional analysis
Availability
Open-source project
Local deployment
Online instance
Feedback for developers and users
http://gerbil.aksw.org
http://github.org/aksw/gerbil
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 15 / 44
21. GERBIL
HOBBIT v0.1
Annotator Tasks
NIF-based Annotators 2519
Babelfy 958
DBpedia Spotlight 922
TagMe 2 811
WAT 787
Kea 763
Wikipedia Miner 714
NERD-ML 639
Dexter 587
AGDISTIS 443
Entityclassifier.eu NER 410
FOX 352
Cetus 1
Overall 24.3K exps
50+ papers
http://gerbil.aksw.org
http://github.org/aksw/gerbil
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 16 / 44
23. HOBBIT
Rationale
Rationale
A community-driven unified benchmarking platform for the community
Build upon 24.3K GERBIL experiments
Experiments focus on Big Linked Data
Designed to accomodate all Big Data
Cover all steps of the Big (Linked) Data
lifecycle
Open benchmarks based on industrial data
and use cases
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 17 / 44
24. HOBBIT
Aims
1 Gather real requirements
Performance indicators
Performance thresholds
2 Develop benchmarks based on real data
3 Provide universal benchmarking platform
Standardized hardware
Comparable results
4 Periodic benchmarking challenges
5 Periodic reporting
6 Found independent Hobbit association
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 18 / 44
25. HOBBIT
Overview
Data Collection
Industry
data
Measure Collection
Benchmark Creation
Benchmark 1
KPIs
Tasks
KPIs
Tasks
KPIs
Tasks
KPIs
Tasks
KPIs
Tasks
KPIs
Tasks
Benchmark 2
Benchmark n
HOBBIT
Platform
Solution 1
Solution k
Solution 2
Challenges
Reports
Participants/Community
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 19 / 44
26. Survey
Questions
Questions
In what areas are organizations active?
What do people expect from benchmarks?
How are benchmarks being used?
Profile Count
Solution providers 56
Technology users 67
Scientific community 65
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 20 / 44
27. Survey
Can your solution be benchmarked?
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 21 / 44
28. Survey
Do you benchmark your solution?
Own datasets and settings in many cases
Own implementations of measures
Results not comparable
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 22 / 44
30. HOBBIT Platform
Features
Uses established deployment
technologies (Docker)
Decoupled components
Benchmark and systems can be
written in different languages
Uses scalable message queues for
communication
Open-source implementation
Supports distributed benchmarks
and systems
Online instance on server cluster
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 24 / 44
31. HOBBIT Platform
Features
Features
Unified benchmarking platform for Big
Data
20+ reference annotation systems
40+ reference datasets
Reference implementations of KPIs
Advantages
Benchmarks derived from real industrial
data and use cases
Scalable size of benchmarks
Archiving of results
Citeable URIs
Result analysis
Availability
Open-source project
Local deployment
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 25 / 44
34. HOBBIT Platform
Benchmark Execution
Platform
Controller
data flow
creates component
Storage
Data
Generator
Task
Generator
Data
Generator
Data
Generator
Task
Generator
Task
Generator
Benchmarked System
Benchmark
Controller
Eval. Storage
ex:Entity rdf:type ex:Class
...
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 28 / 44
35. HOBBIT Platform
Benchmark Execution
Platform
Controller
data flow
creates component
Storage
Data
Generator
Task
Generator
Data
Generator
Data
Generator
Task
Generator
Task
Generator
Benchmarked System
Benchmark
Controller
Eval. Storage
v
ex:Entity
...
SELECT ?v
WHERE { ?v a ex:Class }
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 29 / 44
36. HOBBIT Platform
Benchmark Execution
Platform
Controller
data flow
creates component
Storage
Data
Generator
Task
Generator
Data
Generator
Data
Generator
Task
Generator
Task
Generator
Benchmarked System
Benchmark
Controller
Eval. Storage
X
v
ex:Entity
...
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 30 / 44
37. HOBBIT Platform
Benchmark Evaluation
data flow
creates component
Platform
Controller
Storage
Benchmark
Controller
Evaluation
Module
Eval. Storage
precision=...
recall=...
F1-score=... precision=...
recall=...
F1-score=...
benchmark
parameters:
...
v
ex:Entity
...
v
ex:Entity
...
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 31 / 44
38. HOBBIT Platform
Benchmarks
Streaming and static deterministic benchmarks
Realistic benchmarks
Controlled volume and velocity
Generation and Acquisition
Conversion of XML into RDF
Entity recognition and linking
Relation extraction
Analysis and Processing
Link Discovery
Machine Learning
Supervised and unsupervised
Storage and Curation
Triple stores
Versioning
Incl. updates
Visualization and Services
Question Answering
Faceted Browsing
Usage-based benchmarks
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 32 / 44
39. Datasets
TWIG
Goal: Simulate real Twitter Firehose
Relies on 476 million tweets as training data
Mimicking algorithm based on
Distribution of character frequencies
Distribution of transportation frequency
Network topology
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 33 / 44
40. Datasets
LinkedConnections
Goal: Simulate real transport network
Real transportation data from Belgium for training
Mimicking algorithm based on
Observed correlation between population density and transportation
Distribution of transportation frequency
Network topology
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 34 / 44
41. Datasets
Printing Machinery
Goal: Simulate events from printing machinery
Mimicking algorithm using event correlations and distributions
Changing plate
Double sheet
Early sheet
Finish job
Misaligned sheet
Missing sheet
Operation partially completed
Performance
Printing interval
Produktion Good Sheet
Side guide warning
Start job
Washing blanket
Washing impression cylinder
Washing ink rollers
with washing ink fountain roller
with washing plates
Mai 01 00:00 Mai 01 06:00 Mai 01 12:00 Mai 01 18:00 Mai 02 00:00
Time
Events
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 35 / 44
42. Datasets
Weidmüller
Goal: Simulate events from injection molding machinery
Mimicking algorithm using event correlations and distributions
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 36 / 44
43. Datasets
Semantic Publishing
Goal: Simulate data from the BBC
Generator based on manually configurable set of correlations
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 37 / 44
48. Summary
Rationale
A community-driven unified benchmarking platform for the community
Provide benchmarks and baselines
Provide reference implementation of KPIs
Extensible and referenceable
Open-Source
@hobbit_project
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 41 / 44
49. Join HOBBIT
α completed
Join the HOBBIT community
Provide KPIs
Provide datasets
Join the platform development
Follow us on Twitter
https://project-hobbit.eu
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 42 / 44
50. Thank You
Axel Ngonga
AKSW Research Group
Institute for Applied Informatics
ngonga@informatik.uni-leipzig.de
Michael Röder
AKSW Research Group
Institute for Applied Informatics
roeder@informatik.uni-leipzig.de
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 43 / 44
51. Acknowledgment
This work was supported by grants from the EU H2020 Framework Programme
provided for the project HOBBIT (GA no. 688227).
Ngonga Ngomo (InfAI) Benchmarking Big Data November 15th, 2016 44 / 44