1. DICE
Horizon
2020
Project
Grant
Agreement
no.
644869
h>p://www.dice-­‐h2020.eu
Funded
by
the
Horizon
2020
Framework
Programme
of
the
European
Union
DICE:
Quality-­‐Driven
Development
of
Data-­‐
Intensive
Cloud
ApplicaPons
G.
Casale,
D.
Ardagna,
M.
Artac,
F.
Barbier,
E.
Di
Ni6o,
A.
Henry,
G.
Iuhasz,
C.
Joubert,
J.
Merseguer,
V.
I.
Munteanu,
J.
F.
Pérez,
D.
Petcu,
M.
Rossi,
C.
Sheridan,
I.
Spais,
D.
Vladušič

2. MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015
Overview
and
goals
o MDE
oYen
features
quality
assurance
(QA)
techniques
for
developers
o How
should
quality-­‐aware
MDE
support
data-­‐
intensive
soYware
systems?
o ExisPng
models
and
QA
techniques
largely
ignore
properPes
of
data
o Characterize
the
behavior
of
new
technologies
o DICE:
a
quality-­‐aware
MDE
methodology
inspired
by
DevOps
for
data-­‐intensive
cloud
applicaPons
2
©DICE

3. o SoYware
market
rapidly
shiYing
to
Big
Data
§ 32%
compound
annual
growth
rate
in
EU
through
2016
§ 35%
Big
data
projects
are
successful
[CapGemini
2015]
o European
call
for
soYware
quality
assurance
(QA)
§
ISTAG:
call
to
define
environments
“for
understanding
the
consequences
of
different
implementaNon
alternaNves
(e.g.
quality,
robustness,
performance,
maintenance,
evolvability,
...)”
o QA
evolving
too
slowly
compared
to
the
trends
in
soYware
development
(Big
data,
Cloud,
DevOps
...)
MoPvaPon
3
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

4. DataInc
example
o DataInc
is
a
small
soYware
vendor
selling
cloud-­‐based
environmental
soYware
o DICEnv,
a
warning
system
for
floods
in
rural
regions
o monitoring
local
environmental
condiPons
o fetching
precipitaPons
data
from
satellite
image
stream
o DICEnv
exploits
Big
Data
technologies
and
cloud
capacity
for
online
water
simulaPons
and
MapReduce
for
batch
processing
of
historical
data
o DICEnv
is
a
criPcal
system:
o is
expected
to
remain
up
24/7
o should
quickly
ramp
up
data
intake
rates,
as
well
as
memory
and
compute
capaciPes,
to
update
more
frequently
the
hazard
management
control
room
4
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

5. DataInc
example
o The
contract
requires
delivering
an
iniPal
version
of
DICEnv
within
3
months
serving
a
small
area,
increasing
coverage
on
a
monthly
basis
o Challenges:
o How
to
implement
a
complex
cloud
applicaPon
in
such
a
short
Pme?
o How
to
saPsfy
all
the
quality
requirements?
o What
architecture
should
be
adopted?
5
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

6. Plalorm-­‐Indep.
Model
Domain
Models
Quality-­‐Aware
MDE
Today
6
©DICE
QA
Models
Architecture
Model
Plalorm-­‐Specific
Model
C#JavaC++
Plalorm
DescripPon
MARTE
AnalyPcal
Models
Cost-­‐Quality
Models
Code
generaPon
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

7. Plalorm-­‐Indep.
Model
Domain
Models
Quality-­‐Aware
MDE
Today
7
©DICE
Architecture
Model
Plalorm-­‐Specific
Model
Code
generaPon
C#JavaC++
Plalorm
DescripPon
MARTE
Issues
PIM
layer:
•
staNc
characterisNcs
of
data
•
dynamic
characterisNcs
of
data
•
data
dependencies
DICEnv
modeling
issues:
•
individual
dependencies
between
components
and
data
streams
•
relaPonships
between
compute
and
memory
requirements
•
lack
of
an
explicit
annotaPon
for
data
characterisPcs
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

8. Plalorm-­‐Indep.
Model
Domain
Models
Quality-­‐Aware
MDE
Today
8
©DICE
Architecture
Model
Plalorm-­‐Specific
Model
Code
generaPon
C#JavaC++
Plalorm
DescripPon
MARTE
Issues
at
PSM
layer:
•
heterogeneity
of
Big
Data
technologies
•
automaNc
translaNon
of
PSM
models
into
deployment
plans
QA
tools
limitaPons:
•
contenPon
at
processing
resources
with
limited
features
for
memory
consumpPon
•
fork
and
joining
are
complex
to
be
described
analyPcally
preserving
tractability
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

9. Plalorm-­‐Indep.
Model
Domain
Models
An
HolisPc
Approach:
DICE
9
©DICE
ConPnuous
ValidaPon
ConPnuous
Monitoring
Data
Awareness
Architecture
Model
Plalorm-­‐Specific
Model
Plalorm
DescripPon
DICE MARTE
Deployment
&
ConPnuous
IntegraPon
DICE IDE
Big Data
QA
Models
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

10. Embracing
DevOps
o SoYware
development
process
is
evolving
§
Developer:
“I
want
to
change
my
code”
§
Operator:
“I
want
systems
to
be
stable”
o ...but
code
changes
are
the
cause
of
most
instabiliPes!
o
DevOps
closes
the
gap
between
Dev
and
Ops
§ Lean
release
cycles
with
automated
tests
and
tools
§ Deep
modelling
of
systems
is
the
key
to
automaPon
10
©DICE
Agile
Development
DevOps
Business
Dev
Ops
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

11. Embracing
DevOps
11
©DICE
o QA
must
become
lean
as
well
§ ConPnuous
quality
checks
and
model
versioning
o Modelling
of
the
operaPons
§ Dev
needs
awareness
of
infrastructure
and
costs
o ConPnuous
feedback
§ Forward
and
backward
model
synchronisaPon
§ Tracking
of
self-­‐adaptaPon
events
(e.g.
auto-­‐scaling)
o Big
data
coming
from
conPnuous
monitoring
§ QA
has
its
own
Big
data,
use
machine
learning?
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

12. Benefits
o Tackling
skill
shortage
and
steep
learning
curves
§ Data-­‐aware
methods,
models,
and
OSS
tools
o Shorter
Pme
to
market
for
Big
Data
applicaPons
§ Cost
reducPon,
without
sacrificing
product
quality
o Decrease
development
and
tesPng
costs
§ Select
opPmal
architectures
that
can
meet
SLAs
o Reduce
number
and
severity
of
quality
incidents
§ IteraPve
refinement
of
applicaPon
design
12
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

13. DICE
Plalorm
Independent
Model
(DPIM)
13
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

14. DICE
Plalorm
and
Technology
Specific
Model
(DTSM)
14
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

15. DICE
Plalorm,
Technology
and
Deployment
Specific
Model
(DDSM)
15
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

16. DICE
Profile:
PIM
Level
o FuncPonal
approach
to
data
to
be
expanded
o Data
dependencies
§ graph
relaPonships
between
data,
archives
and
streams
o QA
focuses
on
quanPtaPve
aspects
of
data
o StaPc
characterisPcs
of
data
§ volumes,
value,
storage
locaPon,
replicaPon
pa>ern,
consistency
policies,
data
access
costs,
known
schedules
of
data
transfers,
data
access
control
/
privacy,
...
o Dynamic
characterisPcs
of
data
§ cache
hit/miss
probabiliPes,
read/write/update
rates,
bursPness,
...
16
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

17. DICE
Profile:
PSM
Level
o Need
for
technology-­‐specific
abstracPons
§ Hadoop:
Number
of
mappers
and
reducers
,
...
§ In-­‐memory
DBs:
Peak
memory
and
variable
threading
§ Streaming:
merge/split/operators,
networking,
...
§ Storage:
Supported
operaPons,
cost/byte
,
...
§ NoSQL:
Consistency
policies
,
...
o GeneraPon
of
deployment
plan
§ Proposed
Chef
+
TOSCA
extension
o Interest
is
both
on
private
and
public
clouds
17
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

18. § Risk
of
harm
§ Privacy
&
data
protecPon
DICE
QA:
Quality
Dimensions
o Reliability
o Efficiency
o Safety
18
©DICE
§ Performance
§ Time
behaviour
§ Costs
§ Availability
§ Fault-­‐tolerance
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

19. DICE
QA:
Tools
o Discrete-­‐event
simulaNon:
assess
reliability
and
efficiency
in
Big
Data
applicaPons,
accounPng
for
stochasPc
evoluPon
of
the
environment
o stochasPc
Petri
nets
or
queueing
networks,
rely
on
simulaPon
o Formal
verificaNon
tools:
assess
safety
risks
in
Big
Data
applicaPons,
e.g.
find
design
flaws
causing
order
and
Pming
violaPons
in
message
and
state
sequences
o temporal
logic
formulae
and
bounded
model
checking,
saPsfiability
modulo
theories
solvers
o quanPfier-­‐eliminaPon
techniques
to
extend
temporal
logic-­‐
based
verificaPon
19
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

20. DICE
QA:
Tools
o Architecture
opNmizaNon
tool:
find
architectural
improvements
to
opPmise
costs
and
quality
o decomposiPon-­‐based
analysis
approach
o resort
to
fluid
approximaPon
of
stochasPc
models
o Feedback
analysis:
automated
extracPon
from
the
monitored
data
of
key
parameters
required
to
define
simulaPon
and
verificaPon
models
o extract
model
parameters
through
log
mining
and
staPsPcal
esPmaPon
methods
o breakdown
resource
consumpPon
into
its
atomic
components
on
the
end-­‐to-­‐end
path
of
requests
20
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015

21. DICE
Project
h>p://www.dice-­‐h2020.eu
o Horizon
2020
Research
&
InnovaPon
AcPon
§ Quality-­‐Aware
Development
for
Big
Data
applicaPons
§ Feb
2015
-­‐
Jan
2019,
4M
Euros
budget
§ 9
partners
(Academia
&
SMEs),
7
EU
countries
21
©DICE
MiSE
2015@ICSE,
Florence,
Italy,
May
17
2015