Reactive architectures are emerging as the way to build systems that are responsive, scalable, resilient and event-driven. In other terms, systems that deliver highly responsive user experiences with a real-time feel, that are scalable, resilient, and ready to be deployed on multicore and cloud computing architectures. The Reactive Manifesto (see http://www.reactivemanifesto.org/) captures the key traits that characterize reactive architectures.
The Data Distribution Service (DDS) incarnates the principles enumerated by the reactive manifesto and provides a very good platform for building reactive systems. In this webcast I will (1) introduce the key principles of Reactive Architectures, (2) explain the DDS features that are essential to build reactive systems, and (3) introduce some programming techniques that remove inversion of control while maintaining applications even-driven.
1. Building Reactive
Applications with DDS
Angelo Corsaro, PhD
Chief Technology Officer
OMG DDS SIG Co-Chair
angelo.corsaro@prismtech.com
PrismTech
2. • Reactive Systems were characteristic of “niche” applications domains such
as Industrial, Military, Telco, Finance and Aerospace
• To address the challenges imposed contemporary Internet Applications an
increasing movement of architects and developers is recognising the
importance of the techniques and technologies used for building Reactive
Systems
PrismTech
Copyright PrismTech, 2014
Reactive Systems Go Mainstream
3. Integrated Modular Vetronics
Air Traffic Control & Management
PrismTech
Training & Simulation Systems
Unmanned Air Vehicles
Naval Combat Systems
Aerospace Applications
Copyright PrismTech, 2014
Defense and Aerospace
4. Autonomous Vehicle Systems
Train Control / Management Systems
PrismTech
Smart Grids / Power Generation
Smart Cities
Connected Medical Devices
Financial Services
Copyright PrismTech, 2014
Industrial Internet
5. Reactive Manifesto
• The techniques and
Responsive
Event-Driven
Resilient
• Recently, the Reactive
Manifesto was defined to
summarise the key traits of
“Reactive Applications”
Scalable
http://www.reactivemanifesto.org/
PrismTech
Copyright PrismTech, 2014
technologies traditionally
used in a class of Reactive
and Interactive Systems is
gaining quite a bit of
attention from the
mainstream developers
community
6. Reactive Manifesto
• Event-Driven: modular,
• Responsive: Timely react to
stimuli
• Resilient: Tolerate failures
functionally and ideally
temporally
• Scalable: Gracefully scale up
and down depending on
load demands
Event-Driven
Resilient
Scalable
http://www.reactivemanifesto.org/
PrismTech
Copyright PrismTech, 2014
Responsive
asynchronous
8. Reactive Systems
1
• A Reactive System designates a permanently operating system that
responds, reacts, to external stimuli at a speed determined by its
environment, e.g. an industrial control system.
[1] Nicolas Halbwachs, Synchronous Programming of Reactive Systems
PrismTech
Copyright PrismTech, 2014
Reactive Systems
9. Transformational Systems
• A Transformational System designates a system that terminates after
transforming its input into its output, e.g. a compiler
PrismTech
Copyright PrismTech, 2014
Transformational Systems
10. Interactive Systems
• An Interactive System designates a systems that permanently
communicates with its environment at its own speed, e.g. a computer
game
PrismTech
Copyright PrismTech, 2014
Interactive Systems
11. Reactive vs. Interactive Systems
Interactive
• react to an environment which
• react to an environment which
• response time needs to be
• response time needs to be
• distributed concurrent systems
• distributed concurrent systems
• data-centric
• data-centric
“cannot wait”, e.g. reacting late
has harsh consequences
deterministic
“doesn’t like to wait”, e.g. reacting
late induces QoS degradation
acceptable
The term reactive systems is increasingly used to denote interactive systems with stringent
response time, performance and scalability constraints.
PrismTech
Copyright PrismTech, 2014
Reactive
12. Reactive Systems Key Traits
Concurrency
• Aside from the concurrency between the system and its environment, it is natural to
Strict Temporal Requirements
• Reactive Systems have strict requirements with respect to the rate at which the need to
process input as well as the response time for their outputs
Determinism
• The output of these systems are generally determined by the input and the occurrence
time, e.g. no scheduling effects on the temporal properties of the output
Reliability
• Reactive Systems are often life/mission critical, as such reliability is of utmost importance
PrismTech
Copyright PrismTech, 2014
decompose these systems as an ensemble of concurrent components that cooperate
to achieve the intended behaviour
14. Stream Processing
• Stream Processing is a commonly adopted architectural style for building
• Stream Processing is often applied in:
-
Interactive Systems
-
Signal Processing Systems
-
Functional Stream Programming
-
PrismTech
Reactive Systems
Data Flow Systems
Copyright PrismTech, 2014
systems that operate over continuous (theoretically infinite) streams of data
15. Stream Processing
for modelling systems that react to streams
of data produced by the cyber-phisical
world, such as data produced by sensors,
the stock market, etc.
• Stream Processing Systems operate in realtime over streams and generate in turns
other streams of data providing
information on what is happening or
suggesting actions to perform, such as by
stock X, raise alarm Y, or detected spatial
violation, etc.
PrismTech
Copyright PrismTech, 2014
• Stream Processing Architectures are ideal
16. Stream Processing
• Stream Processing Systems are typically modelled
• Modules usually play one of the following roles:
-
-
PrismTech
Filter
Sources: Injecting data into the System
Filter
Filter
Filters/Actors: Performing some computation over
sources
Sinks: Consuming the data produced by the system
Copyright PrismTech, 2014
as collection of concurrent modules
communicating via typed data channels
Source
Stream
Sink
18. Streams in DDS
infinite sequence of data samples
of a given type T
Filter
• As such, in DDS, a stream can be
naturally represented with a Topic
• Example:
-
PrismTech
Topic<PressureType>
Topic<MarketData>
Filter
Filter
Stream
Copyright PrismTech, 2014
• A stream represents a potentially
19. Topic
• A Topic defines a domain-wide information’s class
Copyright PrismTech, 2014
• A Topic is defined by means of a (name, type, qos)
tuple, where
type: is the programming language type associated
with the topic. Types are extensible and evolvable
Name
-
qos: is a collection of policies that express the nonfunctional properties of this topic, e.g. reliability,
persistence, etc.
Topic
Qo
S
-
PrismTech
name: identifies the topic within the domain
e
Typ
-
20. •As explained in the previous slide a topic defines a class/type of information
•Topics can be defined as Singleton or can have multiple Instances
•Topic Instances are identified by means of the topic key
•A Topic Key is identified by a tuple of attributes -- like in databases
•Remarks:
• A Singleton topic has a single domain-wide instance
• A “regular” Topic can have as many instances as the number of
different key values, e.g., if the key is an 8-bit character then the topic
can have 256 different instances
PrismTech
Copyright PrismTech, 2014
Topic and Instances
21. • Sources in DDS are mapped to
Filter
DataWriters for a given Topic
• Sinks in DDS are mapped to
Filter
Filter
DataReaders for a given Topic
Source
PrismTech
Copyright PrismTech, 2014
Source and Sinks in DDS
Sink
22. Filters in DDS
application business logic
• Each Filter has one or more input stream
and one or more output streams
• Obviously, a Filter consumes data from an
input stream through a DDS data reader
and pushed data into an output stream
through a DDS data writer
PrismTech
Filter
Filter
Filter
Copyright PrismTech, 2014
• Filters implement bit and pieces of the
24. Loose Coupling
• Anonymous Communication => No Spatial Coupling
• Non-Blocking and Asynchronous Interaction => No control or temporal
coupling
In DDS the only thing that matters is the kind of information a module is
interested in consuming or producing — who is producing the data or when
the data has been produced is not relevant
PrismTech
Copyright PrismTech, 2014
Stream Processing Architectures in general and DDS-based architecture in
particular promote loose coupling
25. Location Transparency
• Location Transparency is an important
architectural property as it makes it
possible to completely decouple the
logical decomposition of a system from
its actual physical deployment
• DDS Automatic Discovery brings
location transparency to the next level
by enabling systems that are zero-conf,
self-forming and self-healing
Module
Logical
Deployment Unit
Physical
Deployment Unit
Module
Module
PrismTech
Module
Module
Copyright PrismTech, 2014
Module
26. Temporal Decoupling
Volatile
Durability
decoupling by means of its Durability
Policy
• As such the data produced by a
Source can be:
-
-
-
PrismTech
Source
t
Sink
t
Sink
t
Volatile: available for those that are
around at the time at production
Transient: available as far as the
system/source is running
Durable: available forever
Transient
Durability
Source
t
Sink
t
Sink
t
Copyright PrismTech, 2014
• DDS provides full control over temporal
27. Failure Detection
traditional faults as well as performance
failures
• The Fault-Detection mechanism is
Source
TD
Sink
t
controlled by means of the DDS Liveliness
policy
• Performance Failures can be detected
using the Deadline Policy which allows to
receive notification when data is not
received within the expected delays
Fault
Notification
Source
t
Sink
P
P
P
Performance
Failure
Notification
PrismTech
t
Copyright PrismTech, 2014
• DDS provides mechanism for detecting
28. Fault-Masking
mechanism that allow to replicate
Sources and transparently switch
over when a failure occurs
• At any point in time the “active”
source is the one with the highest
strength. Where the strength is an
integer parameter controller by the
user
PrismTech
Source
Source
t
Sink
t
Copyright PrismTech, 2014
• DDS provides a built-in fault-masking
29. Fault-Masking
mechanism that allow to replicate
Sources and transparently switch
over when a failure occurs
• At any point in time the “active”
source is the one with the highest
strength. Where the strength is an
integer parameter controller by the
user
PrismTech
Source
Source
t
Sink
t
Copyright PrismTech, 2014
• DDS provides a built-in fault-masking
30. Performance
-
15-20 usec Inter-Core Latency
-
45-75 usec over GBps Ethernet
• Throughput
-
PrismTech
Up to 10+M msg/sec inter-core
Up to 5M msg/sec inter node
Copyright PrismTech, 2014
• Latency
31. Standard and Interoperable
introduced as an OMG standard in 2006 to
address Real-time and interoperable Data
Sharing in Network Centric Systems
• Key requirements for the standard were
high and predictable performance, and
scalability to ultra-large-scale deployments
• DDS is recommended by key administration
worldwide, e.g. DoD, MoD, EUROCAE, etc.
and widely adopted across application
domains, e.g., Automated Trading,
Simulations, SCADA, Telemetry, etc.
PrismTech
Copyright PrismTech, 2014
• The Data Distribution Service (DDS) was
33. Dealing with Events
environment
• Stimuli coming from the cyber-physical world are often represented as
events to be handled by the Reactive/Interactive System
-
e.g. think about the event from a GUI, or the data available event in DDS
• Events are often handled through some form of callbacks, such as listeners,
functors, etc. This stile of event handling leads to inversion of control
PrismTech
Copyright PrismTech, 2014
• Reactive and Interactive Systems continuously interact with their
34. The Callback Hell
• The Inversion of Control induced by callback-based code in known to
• As an mental exercise, think about the call-back style code you’d have to
write to in Java for drawing when the mouse is being dragged. Although
the logic is simple, it is far cry from having a “declarative style”
• The problem of callback management, infamously known as Callback Hell,
has become even more important due to the surge of Reactive Systems…
• Can we escape the Callback Hell?
PrismTech
Copyright PrismTech, 2014
make applications brittle and harder to understand
35. Reactive Programming
• Reactive Programming is a paradigm based on a relaxed form of
• The Reactive Programming is built around the notion of continuous timevarying values and propagation of change
• Reactive Programming facilitates the declarative development of non-
blocking event driven applications — in essence you express what should
be done and the runtime decides when to do it
• Reactive Programming was popularised in the context of Functional
Programming Languages by the seminal done by Elliot and Hudak in 1997
as part of Fran — a framework for composing richly interactive multimedia
animations
PrismTech
Copyright PrismTech, 2014
Synchronous Dataflow Programming
36. execute typical instruction
fetch from L1 cache memory
branch misprediction
fetch from L2 cache memory
Mutex lock/unlock
fetch from main memory
send 2K bytes over 1Gbps network
read 1MB sequentially from memory
fetch from new disk location (seek)
read 1MB sequentially from disk
send packet US to Europe and back
1/1,000,000,000 sec = 1 nanosec
0.5 nanosec
5 nanosec
7 nanosec
25 nanosec
100 nanosec
20,000 nanosec
250,000 nanosec
8,000,000 nanosec
20,000,000 nanosec
150 milliseconds = 150,000,000 nanosec
http://norvig.com/21-‐days.html
PrismTech
Copyright PrismTech, 2014
Time for Common Operations
37. Reactive Programming Libraries
There is an increasing number of Reactive Programming framework. The Rx
framework introduced by Erik Meijer is becoming a reference
.NET Rx
• Arguably the first framework that brought Reactive Programming to the
masses. Available for the .NET platform
RxJava (http://bit.ly/rxjava)
• An Open Source implementation of Rx for the JVM contributed by NetFlix, it
supports, Java, Scala, Kotlin, Clojure, Groovy and JRuby
PrismTech
Copyright PrismTech, 2014
!
38. RxJava Overview
for composing asynchronous and event-based programs by using
observable sequences.
• RxJava allow you to compose streams of data declaratively while
abstracting away low level concerns such as threading, synchronization,
concurrent data structures, and non-blocking I/O
• The two main concepts at the foundation of RxJava are Observables and
Observers
PrismTech
Copyright PrismTech, 2014
• RxJava is a Java implementation of .NET Reactive Extensions — a library
39. RxJava Observables
• RxJava Observables allow the composition of flows and sequences of
• You can think of Observables as some kinds of asynchronous collections
that push data to you
• In essence an observable can be created from any synchronous or
asynchronous stream of data.
-
-
PrismTech
you can create an observables that represents the data coming into a DDS
Data Reader, the events from a Button, or even time
you can also create an observable that represent an actual container
Copyright PrismTech, 2014
asynchronous data
40. Observable Examples
0
1
2
3
4
5
t
n
val
evens:
Observable[Long]
=
ticks.filter(n
=>
n
%
2
==
0)
x
0
1
x
2
3
4
x
5
t
n
val
bufs:
Observable[Seq[Long]]
=
evens.buffer(2,1)
0
PrismTech
2
4
6
8
n
t
Copyright PrismTech, 2014
val
ticks:
Observable[Long]
=
Observable.interval(1
seconds)
41. • The Observer receives
notification concerning new
values, errors or the
completion of the stream —
e.g. no more data
PrismTech
trait
Observer[-‐T]
{
def
onNext(value:
T):
Unit
def
onError(error:
Throwable):
Unit
def
onCompleted():
Unit
}
Copyright PrismTech, 2014
RxJava Observers
42. • Beside from the standard RxJava/RxScala Observable, each framework
that wants to integrate with RxJava/RxScala has to provide its own factory
methods to create observables
• For DDS you have to use the DdsObservable defined as follows:
object
DdsObservable
{
def
fromDataReaderData[T](dr:
DataReader[T]):
Observable[T]
def
fromDataReaderEvents[T](dr:
DataReader[T]):
Observable[ReaderEvent[T]]
//
more
methods
which
we’ll
ignore
for
the
time
being
}
PrismTech
Copyright PrismTech, 2014
DDS Observables
44. Shapes Application
Application to illustrate the benefits of
using Reactive Programming with DDS
• Three Topics
-
Circle, Square, Triangle
• One Type:
struct ShapeType {
string color;
Pierced shapes represent publications
long x;
long y;
long shapesize;
Spotted shapes represent subscriptions
};
#pragma keylist ShapeType color
PrismTech
Copyright PrismTech, 2014
• We will use the omnipresent Shapes
45. • To appreciate the joys of Reactive lets look first at how to implement a
Shapes Logger using traditional “callback” programming
PrismTech
Copyright PrismTech, 2014
Logging Shapes
46. Step 1: Implement Listener
class
ShapesListener
extends
DataReaderListener[ShapeType]
{
!
def
onRequestedDeadlineMissed(p1:
RequestedDeadlineMissedEvent[ShapeType]):
Unit
def
onRequestedIncompatibleQos(p1:
RequestedIncompatibleQosEvent[ShapeType]):
Unit
!
def
onSampleRejected(p1:
SampleRejectedEvent[ShapeType]):
Unit
!
def
onLivelinessChanged(p1:
LivelinessChangedEvent[ShapeType]):
Unit
!
def
onDataAvailable(p1:
DataAvailableEvent[ShapeType]):
Unit
!
def
onSubscriptionMatched(p1:
SubscriptionMatchedEvent[ShapeType]):
Unit
!
def
onSampleLost(p1:
SampleLostEvent[ShapeType]):
Unit
}
PrismTech
Copyright PrismTech, 2014
!
47. Step 2: Read and Log Data
• From the listener code you’d retrieve the data reader (dr) and do
something like this:
def
onDataAvailable(p1:
DataAvailableEvent[ShapeType]):
Unit
=
{
dr.read.foreach(s
=>
println("("
+
s.color
+
",
"
+
s.x
+
",
"
+
s.y))
}
!
PrismTech
Copyright PrismTech, 2014
• To keep things simple and more compact we’ll use the Scala binding
48. Molière Makes it Better
write:
dr
listen
{
case
DataAvailable(_)
=>
{
dr
read()
foreach
(s
=>
println("("
+
s.color
+
",
"
+
s.x
+
",
"
+
s.y))
}
}
PrismTech
Copyright PrismTech, 2014
• To be fair, if you use Moliere, the Scala API for OpenSplice you could simply
50. To show a bit more the expressive power of Reactive programming, let’s see
how we can solve the following problems
PrismTech
Copyright PrismTech, 2014
“Shapes Problems”
51. • We want to show a triangle that
positioned midway between every
single pair of circle and square of the
same colour
PrismTech
Copyright PrismTech, 2014
Problem 1: Shape in the Middle
52. Shapes Marble Diagrams
t
• Conceptually, we need to select on stream, look at it one element at the time
and match it with the corresponding item in the other stream
• Then, we can produce the average
PrismTech
Copyright PrismTech, 2014
t
54. Shapes Marble Diagrams
x x x x
t
t
P.S.: The dropping may seem a bit counter-intuitive but it is in part due to how the DDS streams work
PrismTech
Copyright PrismTech, 2014
t
55. val
circles
=
DdsObservable.fromDataReaderData
{
DataReader[ShapeType](Topic[ShapeType](circle))
}
val
squares
=
DdsObservable.fromDataReaderData
{
DataReader[ShapeType](Topic[ShapeType](square))
}
val
ttopic
=
Topic[ShapeType](triangle)
val
tdw
=
DataWriter[ShapeType](ttopic)
!
!
//
Compute
the
average
between
circle
and
square
of
matching
color
with
flatMap
val
triangles
=
circles.flatMap
{
c
=>
squares.dropWhile(_.color
!=
c.color).take(1).map
{
s
=>
new
ShapeType(s.color,
(s.x
+
c.x)/2,
(s.y
+
c.y)/2,
(s.shapesize
+
c.shapesize)/4)
}
}
!
triangles.subscribe(tdw.write(_))
PrismTech
Copyright PrismTech, 2014
Midpoint Triangle
56.
//
Compute
the
average
between
circle
and
square
of
matching
colour
with
flatMap
val
triangles
=
circles.flatMap
{
c
=>
squares.dropWhile(_.color
!=
c.color).take(1).map
{
s
=>
new
ShapeType(s.color,
(s.x
+
c.x)/2,
(s.y
+
c.y)/2,
(s.shapesize
+
c.shapesize)/4)
}
}
//
Compute
the
average
between
circle
and
square
of
matching
colour
with
for
comprehension
val
triangles
=
for
{
c
<-‐
circles;
s
<-‐
squares.dropWhile(_.color
!=
c.color).take(1)
}
yield
new
ShapeType(s.color,
(s.x
+
c.x)/2,
(s.y
+
c.y)/2,
(s.shapesize
+
c.shapesize)/4)
PrismTech
Copyright PrismTech, 2014
Using for-comprehension
58. Concluding Remarks
• Many of the requirements of traditional Reactive Systems are applicable to
• Stream Processing Architecture and Reactive Programming can greatly
help in the design and implementation of Reactive Systems that are
efficient, responsive, resilient, scalable and highly available
• As proven through years of deployments DDS provides the ideal
infrastructure technology for Reactive Systems
• The combination of DDS with Reactive Programming makes for a very
powerful platform!
PrismTech
Copyright PrismTech, 2014
many mainstream application domains
59. • OpenSplice DDS [http://prismtech.com]
• RxJava [http://bit.ly/rxjava] (OpenSouce Rx Implementation from NetFlix)
• Moliere [http://bit.ly/nuvo-moliere] (This includes Rx Binding for DDS)
PrismTech
Copyright PrismTech, 2014
References