On the need to include functional testing in RDF stream engine benchmarks

On the need to include
functional testing in RDF
stream engine
benchmarks
Daniele Dell’Aglio,
Marco Balduini, and
Emanuele Della Valle
1st International Workshop on
Benchmarking RDF Systems (BeRSys
2013)
co-located with ESWC 2013
May 26th, 2013
Montpellier, France

Agenda
Background on
Data Stream Management Systems (DSMS)
RDF Stream Engines
Benchmarking RDF Stream Engine
Operational semantics of RDF Stream Engines
Testing the correctness of continuous queries'
results
Conclusions
BeRSys 2013 - May 26, 2013Emanuele Della Valle - http://streamreasoning.org

Background
Data Stream Management Systems
What are data streams?
Formally:
Data streams are unbounded sequences of time-
varying data elements
Less formally:
an (almost)“continuous”flow of information
with the recent information being more relevant as it
describes the current state of a dynamic system
time

Background
The nature of streams requires a paradigmatic
change*
from persistent data
to be stored and queried on demand
a.k.a. one time semantics
to transient data
to be consumed on the fly by continuous queries
a.k.a. continuous semantics
* This paradigmatic change first arose in DB community BeRSys 2013 - May 26, 2013Emanuele Della Valle - http://streamreasoning.org

Background
Continuous queries registered over streams
that, in most of the cases, are observed trough
windows
Streams of answer
produced by the
relation to stream
operators
Registered
Continuou
s Query
Window:
stream to
relation
operators
input streams
Express using
relation to relation
operators

Background
Types of windows (a.k.a., stream to relation
operators)
physical: a given number of data elements
logical: a variable number of data elements which occur
during a given time interval (e.g., 1 hour)
Sliding: they are progressively advanced of
a given STEP (e.g., 5 minutes)
Tumbling: they are advanced of exactly their time
interval

Background
RDF Stream Engines
RDF Stream Engines ports DSMS concepts into
the Semantic Web extending
RDF data model with the notion of RDF Stream
…
<si pi oi> : [τ1]
<si+1 pi+1 oi+1> : [τ1+1]
…
SPARQL to express and process continuous queries
Existing languages/engines
CQELS
SPARQLSTREAM
C-SPARQL
Timestamps are
non-decreasing to allow
for expressing
contemporaneity

Background
E.g., where C-SPARQL extends
SPARQL

Background
An example of C-SPARQL query
Who are the opinion makers? i.e., the users who are
likely to influence the behaviour of other users who
follow them
REGISTER STREAM OpinionMakers COMPUTED EVERY 5m AS
CONSTRUCT { ?opinionMaker sd:about ?resource }
FROM STREAM
<http://streamingsocialdata.org/interactions>
[RANGE 30m STEP 5m]
WHERE {
?opinionMaker ?opinion ?resource .
?follower sioc:follows ?opinionMaker.
?follower ?opinion ?resource.
FILTER ( cs:timestamp(?follower) >
cs:timestamp(?opinionMaker)
&& ?opinion != sd:accesses ) BeRSys 2013 - May 26, 2013

Background
An example of C-SPARQL query
Who are the opinion makers? i.e., the users who are
likely to influence the behaviour of other users who
follow them
REGISTER STREAM OpinionMakers COMPUTED EVERY 5m AS
CONSTRUCT { ?opinionMaker sd:about ?resource }
FROM STREAM
<http://streamingsocialdata.org/interactions>
[RANGE 30m STEP 5m]
WHERE {
?opinionMaker ?opinion ?resource .
?follower sioc:follows ?opinionMaker.
?follower ?opinion ?resource.
FILTER ( cs:timestamp(?follower) >
cs:timestamp(?opinionMaker)
&& ?opinion != sd:accesses )
Query registration
(for continuous
execution)
FROM STREAM
clause
WINDOW
RDF Stream added
as
new output format
Builtin to
access
timestamps
Aggregates
as in SPARQL
1.1
BeRSys 2013 - May 26, 2013

Background
Benchmarking RDF stream engines
SRBench
Dataset: LinkedSensorData (real meteorological
sensor data)
Queries: 17 continuous queries, some requiring RDFS
reasoning
KPI: feature coverage and correctness
LSBench
Dataset: synthetic social network inspired data set
Queries: 12 continuous queries involving multiple
stream and static knowledge
KPI: input throughput and correctness
Not
verified
Verified
comparing the
number of
results produced
by different

Is verifying correctness hard?
Not for SPARQL
http://www.w3.org/2009/sparql/docs/tests/
Queries + expected results
However, it is hard for continuous (SPARQL)
queries
1 query  multiple correct results
Input data and query are not enough to determine
the correct result

A simple test
Take the motivation scenario of CQELS
there are two connected rooms, r1 and r2;
each room has a sensor able to detect the individuals
inside, m1 and m2.
The stream ST contains the following triples:
<:m1 :detectedAt :r1>:[1]
S1
S2
S3
S4

The query of the simple test
We want to know when the two individuals m1 and m2 are in the
same room using time-based tumbling window of 10 seconds.
REGISTER QUERY SimpleTest AS
SELECT ?room
FROM STREAM <http://ex.org/ST> [RANGE 10s STEP 10s]
WHERE {
:m1 :detectedAt ?room .
:m2 :detectedAt ?room
}

All the results you can obtain
Running the test in C-SPARQL, CQELS and
SPARQLSTREAM the following results can be
obtained.
Are they all correct?
How can this be?

These engines have
different operational
semantics!

The devil is in the details!
S1 S2 S3 S4
W0
S1 = <:m1 :detectedAt
:r1>:[1]
:r1>:[3]
:r2>:[12]
S4 = <:m2 .detectedAt
:r2>:[15]
ST
t3 12 151
W1
W2
W3
W4
W5
W6

Can operational semantics of RDF
Stream Engines be modelled?
A model has been proposed to explain the
differences that appear between different DSMS:
SECRET
The results of a DSMS not only depends on the input
and the query, but also on the system

ScopE in the SECRET model
It is the time range of the active window
[topen,tclose)
it is determined using the size ω and
slideβparameters of the window as written by the
query issuer
tapp
1 2 3 4 5 6 7
tW1
W2
W3
Closed
Open
Active
ω=3
β= 2

Content in the SECRET model
It is the subset of the stream included of the
active window
It is determined using
the size ω and slide β parameters of the window and
t0 the time instant on which the first window starts,
W0
W1
W2
W3
t3 12 151
ω=β=10
Different values for
t0 BeRSys 2013 - May 26, 2013Emanuele Della Valle - http://streamreasoning.org

Report in the SECRET model
It defines the conditions under which the window
contents become visible for further query
evaluation and result reporting
It can take a logical combination of the following:
content change
window close
non-empty content
periodic

Explaining the results of C-SPARQL
S1 S2 S3 S4
W0
W1
W2
W3
W4
W5
W6
ST
t3 12 151
ω=β=10
The reporting strategy of
C-SPARQL is window
close and non-empty
result
:r1>:[1]
:r1>:[3]
:r2>:[12]
:r2>:[15]

Explaining the results of
SPARQLSTREAM
S1 S2 S3 S4
W0
W1
W2
W3
W4
W5
W6
ST
t3 12 151
ω=β=10
SPARQLSTREAM is window
close
:r1>:[1]
:r1>:[3]
:r2>:[12]
:r2>:[15]

Explaining the results of
CQELS
S1 S2 S3 S4
W0
W1
W2
W3
W4
W5
W6
ST
t3 12 151
ω=β=10
CQELS is content
change, and non-empty
result
:r1>:[1]
:r1>:[3]
:r2>:[12]
:r2>:[15]

And so what?

Let's test correctness!
Data
Query
SELECT ?room WHERE {
STREAM <http://ex.org/s1> [RANGE 3s SLIDE 3s] {
?p1 :detectedAt ?room .
?p2 :detectedAt ?room }
FILTER (?p1 != ?p2) }
Timeline
S1 = <:m1 :detectedAt :r1>:[0]
S1 S2 S3 S4
t0 5 10 15
S1
ω=β=3

Results of the test
Why?

Trying to make sense of it …
Data
Query
SELECT ?room WHERE {
STREAM <http://ex.org/s1> [RANGE 3s SLIDE 3s] {
?p1 :detectedAt ?room .
?p2 :detectedAt ?room }
FILTER (?p1 != ?p2) }
Timeline
S1 S2 S3 S4
t0 5 10 15
S1
ω=β=3
Let's remove this
filter

Results of the new test
Is this caused by
incorrect removal of
the triples from the
window?

Does throughput matters
without correctness?

Conclusions
The different operational semantics of existing
RDF stream engines affect the outputs and the
performance of those systems
Throughput measurements must be
performed
while testing correctness
Modeling RDF stream Engines using SECRET
allows for checking correctness
SRbench and LSBench should be extended with
an "oracle" that checks correctness

Thank you for you
attention: questions?
Daniele Dell’Aglio,
Marco Balduini, and
Emanuele Della Valle

On the need to include functional testing in RDF stream engine benchmarks

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (7)

Destaque

Destaque (6)

Semelhante a On the need to include functional testing in RDF stream engine benchmarks

Semelhante a On the need to include functional testing in RDF stream engine benchmarks (20)

Mais de Emanuele Della Valle

Mais de Emanuele Della Valle (20)

Último

Último (20)

On the need to include functional testing in RDF stream engine benchmarks