1. Your
systems.
Working
as
one.
Introduc)on
to
OMG
DDS
OMG
Technical
Mee9ng,
Washington
D.C.,
March
2013
Gerardo
Pardo-­‐Castellote,
Ph.D.
[gerardo@r9.com]
CTO,
Real-­‐Time
Innova9ons,
Inc.
[www.r9.com]
Co-­‐chair
of
the
OMG
Data-­‐Distribu9on
SIG

2. Systems
that
interact
with
the
Real
World
• Must
adapt
to
changing
environment
• Cannot
stop
processing
the
informa9on
• Live
within
world-­‐imposed
9ming
Beyond
tradi9onal
interpreta9on
of
real-­‐9me
©
2012
RTI
•
ALL
RIGHTS
RESERVED
2

3. Challenge:
More
Data,
More
Speed,
More
Sources
TRENDS:
• Growing
Informa9on
Volume
• Lowering
Decision
Latency
• Increasing
System
Availability
• Accelera9ng
technology
inser9on
and
deployment
Next-­‐genera)on
systems
needs:
• Scalability
• Integra9on
&
Evolu9on
• Robustness
&
Availability
• Performance
• Security
©
2012
RTI
•
ALL
RIGHTS
RESERVED
3
3

4. “Real
World”
Systems
are
integrated
using
a
Data
Model
• Grounded
on
the
“physics”
of
the
problem
domain
– Tied
to
the
nature
of
the
sensors
and
real
objects
in
the
system
(vehicles,
device
types,
…)
• Provides
governance
across
disparate
teams
&
organiza9ons
– The
“N^2”
integra9on
problem
is
reduced
to
a
“N”
problem
• Increased
decoupling
from
use-­‐cases
and
components
– Avoids
over
constraining
applica9ons
• Open,
Evolvable,
Plaborm-­‐Independent
– The
use-­‐cases,
algorithms
might
change
between
missions
or
versions
of
the
system
App
App
App
Realizing
this
data-­‐model
requires
a
middleware
infrastructure
©
2012
RTI
•
ALL
RIGHTS
RESERVED
4

5. DDS:
Standards-­‐based
Integra9on
Infrastructure
for
Cri9cal
Applica9ons
Streaming
Sensors
Events
Data
Real-­‐Time
Enterprise
Actuators
Applica)ons
Applica)ons
©
2012
RTI
•
ALL
RIGHTS
RESERVED
5

6. Family
of
Specifica9ons
2008
2009
2010
2010
2013
2013
UML
Profile
DDS
for
DDS
DDS-­‐STD-­‐C++
DDS
RPC
over
for
DDS
Lw
CCM
X-­‐Types
DDS-­‐JAVA5
Security
DDS
App
2004
App
App
DDS
Spec
DDS
2006
DDS
DDS
Implementa9on
DDS
Implementa9on
Implementa9on
Interoperablity
Network
/
TCP
/
UDP
/
IP
©
2012
RTI
•
ALL
RIGHTS
RESERVED
6
6

7. Broad
Adop9on
• Vendor
independent
Cross-­‐vendor
portability!
– API
for
portability
– Wire
protocol
for
interoperability
• Mul9ple
implementa9ons
– 10
of
API
DDS
API
– 8
support
RTPS
• Heterogeneous
Middleware
– C,
C++,
Java,
.NET
(C#,
C++/CLI),
ADA,
Python,
Scala,
…
DDS
Real-­‐Time
Publish-­‐Subscribe
– Linux,
Windows,
VxWorks,
Integrity,
Wire
Protocol
(RTPS)
other
embedded
&
real-­‐9me
• Loosely
coupled
Cross-­‐vendor
interoperability!
©
2012
RTI
•
ALL
RIGHTS
RESERVED
7

8. Interoperability
between
the
applica)ons
implemented
by
six
different
vendors
(March
2012)
OCI
ETRI
PrismTech
IBM
RTI
TwinOaks
8

9. DDS
mandated
by
key
DoD
Programs
• UK
Generic
Vehicle
Architecture
– Mandates
DDS
for
vehicle
comm.
– Mandates
DDS-­‐RTPS
for
interop.
• DISR
– Mandates
DDS
for
Pub-­‐Sub
API
– Mandates
DDS-­‐RTPS
for
Interop
• Army,
OSD
– UCS,
Unmanned
Vehicle
Control
– FACE
(Future
Airborne
Capability
Environment)
• US
Navy
Open
Architecture
– Mandates
DDS
for
Pub-­‐Sub
• SPAWAR
NESI
– Mandates
DDS
for
Pub-­‐Sub
SOA
©
2012
RTI
•
ALL
RIGHTS
RESERVED
9

10. DDS
Deployed
Applica9on
Examples
©
2012
RTI
•
ALL
RIGHTS
RESERVED
10

11. Everyday
Example:
Calendaring
Alterna)ve
Process
#1
(message-­‐centric):
1. Email:
“Mee9ng
Monday
at
10:00.”
2. Email:
“Here’s
dial-­‐in
info
for
mee9ng…”
3. Email:
“Mee9ng
moved
to
Tuesday”
4. You:
“Where
do
I
have
to
be?
When?”
5. You:
(siFing
through
email
messages…)
©
2012
RTI
•
ALL
RIGHTS
RESERVED
11

12. Example:
Calendaring
Alterna)ve
Process
#2:
1. Calendar:
(add
meeIng
Monday
at
10:00)
2. Calendar:
(add
dial-­‐in
info)
3. Calendar:
(move
meeIng
to
Tuesday)
4. You:
“Where
do
I
have
to
be?
When?”
5. You:
(check
calendar.
Contains
consolidated-­‐state)
The
difference
is
state!
The
infrastructure
consolidates
changes
and
maintains
it
©
2012
RTI
•
ALL
RIGHTS
RESERVED
12

13. Data-­‐Centric
Middleware
allows
applica9ons
to
be
integrated
to
the
Informa9on
Model
APP
APP
APP
APP
Standard
API
Data
Model
Standard
Mapping(*)
DDS
Global
Data
Space
No
custom
mappings
/
code
necessary
Direct
support
for
data-­‐centric
ac9ons:
create,
dispose,
read/take
Copyright
©
2010
RTI
-­‐
All
rights
Reserved.
.
13

14. Integra9ng
components
to
generic
middleware
technology
Comp
Comp
Comp
Comp
Custom
Integra9on
Data
Model
Custom
Mapping
Middleware
Ar)facts
Akin
to
implemen9ng
an
OO
design
on
a
Procedural
Language:
Requires
mapping
inheritance,
encapsula9on,
excep9ons,
…
Copyright
©
2010
RTI
-­‐
All
rights
Reserved.
.
14

15. Tradi9onal
data-­‐centric
technologies
not
suited
to
scalable
near
real-­‐9me
systems
Other
data-­‐centric
technologies:
– Databases:
SQL
– Web:
HTTP
(mostly)
• …assume
the
world
changes
slowly
• …use
network
resources
inefficiently
Not
scalable
• …are
highly
centralized
100
apps
=>
100x
load
Slow
A
few
updates/sec
App
App
Server
App
App
State
App
App
Unreliable
Failure
here
kills
many
apps
©
2012
RTI
•
ALL
RIGHTS
RESERVED
15

16. DDS
is
decentralized.
Can
be
deployed
without
servers/brokers
DDS:
• …allows
you
to
observe
frequent
changes
• …uses
network
resources
efficiently
• …is
peer-­‐to-­‐peer
and
decentralized
Fast
Scalable
Managed
Reliable
100,000’s
updates/sec
Load
indep.
#
apps
with
QoS
No
single
pt.
failure
App
App
App
App
App
App
DDS
Data-­‐Centric
Bus
©
2012
RTI
•
ALL
RIGHTS
RESERVED
16

17. The
data-­‐centric
communicaIons
model
©
2012
RTI
•
ALL
RIGHTS
RESERVED
17

18. DDS
Adressing:
Data-­‐Objects
in
the
Global
Data
Space
• Domain:
world
you’re
talking
about
Domain
• Topic:
group
of
similar
objects
(e.g.
Yellowstone
Park)
– Similar
structure
(“type”)
what
– Similar
way
they
change
when
Topic
over
9me
(“QoS”)
how
(e.g.
bears
in
the
park)
• Instance:
individual
object
Instance
Instance
Booboo
(e.g.
Yogi
the
bear)
• DataWriter:
source
of
observa9ons
about
a
set
of
data-­‐objects
(Topic)
• DataReader:
observer
of
a
set
of
data-­‐objects
Topic
(Topic)
Snow
Depth
Sensors
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
GPS
value
value
value
value
value
value
value
value
value
value
value
value
value
value
value
value
value
value
value
value
value
value
value

19. Data-­‐Centric
Qos-­‐Aware
Pub-­‐Sub
Model
Virtual,
decentralized
global
data
space
Source
Speed Power Phase
(Key)
WPT1 37.4 122.0 -12.20
WPT2 10.7 74.0 -12.23
WPTN 50.2 150.07 -11.98
Persistence
Recording
CRUD
opera9ons
Service
Service
©
2012
RTI
•
ALL
RIGHTS
RESERVED
19

20. ShapesDemo
Demo:
Publish-­‐Subscribe
©
2012
RTI
•
ALL
RIGHTS
RESERVED
20

21. Data-­‐Centric
example
Communica9ons
Model
Data
Domain
Data
Domain
New Writer
Par9cipant
Reader
Par9cipant
“Alarm”
Got new “Alarm”
subscriber data
!
Offered Requested
Listener
QoS Listener
QoS
• Par)cipants
scope
the
global
data
space
(domain)
• Topics
define
the
data-­‐objects
(collec9ons
of
subjects)
• DataWriters
publish
data
on
Topics
• DataReaders
subscribe
to
data
on
Topics
• QoS
Policies
are
used
configure
the
system
• Listeners
are
used
to
no9fy
the
applica9on
of
events
©
2012
RTI
•
ALL
RIGHTS
RESERVED
21

22. Quality
of
Service
(QoS)
• Aside
from
the
actual
data
(WHAT)
to
be
delivered,
users
ozen
need
to
specify
HOW
to
send
it
…
…
reliably
(or
“send
and
forget”)
…
how
much
data
(all
data
,
last
5
samples,
every
2
secs)
…
how
long
before
data
is
regarded
as
‘stale’
and
is
discarded
…
how
many
publishers
of
the
same
data
is
allowed
…
how
to
‘failover’
if
an
exis9ng
publisher
stops
sending
data
…
how
to
detect
“dead”
applica9ons
…
…
• These
op9ons
are
controlled
by
formally-­‐defined
Quality
of
Service
(QoS)
policies

23. Real-­‐Time
Quality
of
Service
Control
Collision
; 1 Hz; Subscriber
avoidance
Reliable application
ry
get histo
Best effort; 0.2 HZ;
Publisher
Radar
Track
get history Subscriber
Best e
ffo
• Publish reliably no hist rt; 0.01 Hz;
• 10 Hz
ory Airline
• Keep 10 minute history Subscriber
operations
(6000 samples) center
Backup
Publisher
Database,
real-time log
• Quality
of
Service
(QoS)
determined
per-­‐en6ty
• QoS
Contract:
Request
-­‐
Offered
• Publishing
and
subscribing
applica9ons
can
be
no9fied
when
QoS
contract
is
violated
– e.g.
Messages
lost
or
deadlines
missed
• High
availability
via
automa9c
failover

24. Real-­‐Time
Quality
of
Service
(QoS)
QoS Policy QoS Policy
DURABILITY USER DATA
User
QoS
Cache
HISTORY TOPIC DATA
LIFESPAN GROUP DATA
WRITER DATA LIFECYCLE PARTITION
Presenta)on
Resources
READER DATA LIFECYCLE PRESENTATION
ENTITY FACTORY DESTINATION ORDER
RESOURCE LIMITS OWNERSHIP
Availability
RELIABILITY OWNERSHIP STRENGTH
Delivery
TIME BASED FILTER LIVELINESS
DEADLINE LATENCY BUDGET
Transport
CONTENT FILTERS TRANSPORT PRIORITY

25. ShapesDemo
Qos
in
Ac9on
• Detec9ng
presence
• History
cache
• Dele9ng
objects
• Ownership
• Liveliness
• Filtering
• Durability
©
2009
Real-­‐Time
Innova9ons,
Inc.
25

26. OperaIonal
robustness
and
performance
©
2012
RTI
•
ALL
RIGHTS
RESERVED
26

27. Architecture
for
the
next-­‐genera9on
systems
• Exis9ng
technologies
are
reaching
robustness/performance/scalability
limits
• DDS
provides
a
fundamentally
new
DataBus
architecture
and
approach
– Powerful
data-­‐centric
model
– Ultra-­‐scalable
and
robust
– Fully
decentralized,
peer-­‐to-­‐peer,
“no
bo}lenecks”
architecture
– Superior
Wire
Protocol
– Standards-­‐based,
mul9-­‐plaborm
Brokers
as
choke-­‐points
Connext
DDS
Approach
Single-­‐lane
traffic
No
priori9za9on
©
2012
RTI
•
ALL
RIGHTS
RESERVED
27

28. Performance
Performance
results
are
vendor-­‐specific
These
results
are
for
RTI
Connext
DDS.
Average
Latency
(Microseconds)
400
Number
of
Subscribers
350
1
(1
per
CPU
and
NIC)
300
20
(1
per
CPU
and
NIC)
250
40
(1
per
CPU,
2
per
NIC)
200
150
• Reliable
mul9cast
100
• Fully
meshed,
reliable
50
0
Orders
of
magnitude
faster
than
IT
solu9ons
Throughput
(Messages
per
Seconds)
Fastest
DDS
solu)on
©
2012
RTI
•
ALL
RIGHTS
RESERVED
28

29. Scalability
Scalability
results
are
vendor-­‐specific
These
results
are
for
RTI
Connext
DDS.
• Scalable
600,000
Performance!!
• Millions of data
Per
Subscriber
(200
Bytes)
500,000
elements!
Messages
per
Second
• .5m updates/sec
400,000
(batched)!
• 10s µs latency!
300,000
• 1000s consumers
per update!
200,000
• Orders
of
magnitude
more
scalable
100,000
than
IT
solu9ons
0
1

~1000
subscribers,
<
15%
throughput
decrease
0
200
400
600
800
1,000
Number
of
Subscribers
©
2012
RTI
•
ALL
RIGHTS
RESERVED
29

30. Comparison
with
other
technologies
Comparison
results
are
vendor-­‐specific
These
results
compare
with
RTI
Connext
DDS.
Message
Length
(samples)
Adapted
from
Vanderbilt
presenta9on
at
July
2006
OMG
Workshop
on
RT
Systems
©
2012
RTI
•
ALL
RIGHTS
RESERVED
30

31. Common
use-­‐cases
/
paRerns
©
2012
RTI
•
ALL
RIGHTS
RESERVED
31

32. Common
Use
Cases
• Isola9ng
subsystems
• Detec9ng
presence
of
applica9ons
• Discovering
publishers
and
subscribers
• Keeping
a
“last-­‐value”
cache
• Monitoring
the
health
of
applica9ons
• Monitoring
the
health
of
data
• Reliable
data
delivery
• Building
a
highly-­‐available
system
• Managing
network
load
and
behavior
3/24/13
©
2012
RTI
•
COMPANY
CONFIDENTIAL
32

33. Isola9ng
Subsystems
• Concept
of
Domains
and
Domain
Par9cipants
N4
App
4
•
Container
for
N1
App
1
Pub/Sub
Pub/Sub
applica9ons
that
want
(A,B/C,D)
(D/C,E,F)
to
communicate
•
Applica9ons
can
join
N2
App
2
N4
App
5
or
leave
a
domain
in
any
order
Subscribe
Domain Publish
(C)
(C)
•
New
Applica9ons
are
“Auto-­‐Discovered”
N3
App
3
N5
App
6
•
An
applica9on
that
Pub/Sub
Subscribe
(E,F/A,C)
(B,C)
has
joined
a
domain
is
also
called
a
“Domain
Par9cipant”
Single
‘Domain’
System

34. Isola9ng
Subsystems
• Use
mul9ple
domains
tor
scalability,
modularity
and
isola9on
Node
1
-­‐
App
1
Node
4
-­‐
App
1
Pub/Sub
Domain
A Pub/Sub
Node
2
-­‐
App
1
Node
4
-­‐
App
2
Subscribe
Publish
Domain
B
Node
3
-­‐
App
1
Node
5
-­‐
App
1
Pub/Sub
Subscribe
Domain
C
Node
5
-­‐
App
2
Node
6
-­‐
App
1
Added
Func.
Pub/Sub
Pub/Sub
Mul)ple
Domain
System

35. Detec9ng
presence
of
applica9on
• DDS
has
a
buil9n
discovery
service
• It
provides
the
means
for
an
applica9on
to
discover
the
presence
of
other
par9cipants
on
the
Domain
– The
Topic
“DCPSPar9cipants”
can
be
read
as
a
regular
Topic
to
see
when
DomainPar9cipants
join
and
leave
the
network
• Applica9ons
can
also
include
meta-­‐data
that
is
sent
along
by
DDS
discovery

36. Discover
Publishers
and
Subscribers
• DDS
provides
a
mechanism
to
detect
en99es,
their
capabili9es
and
requirements
• An
applica9on
can
discover
all
the
other
en99es
in
the
system
that
match
the
requirements
– The
Topics
“DCPSPublica9ons”,
“DCPSSubscrip9ons”,
“DCPSTopics”,
and
“DCPSPar9cipants”
can
be
read
to
observe
the
other
en99es
in
the
domain
3/24/13
©
2012
RTI
•
COMPANY
CONFIDENTIAL
36

37. Publishing
data
that
outlives
it
source
• The
concept
of
durability
in
the
global
dataspace
– Vola9le
• No
durability
– Transient
local
• Durability
maintained
by
the
publishing
applica9on
– Transient
• Durability
maintained
by
a
3rd
party
applica9on
(persistence
service)
– Persistent
• Durability
of
data
is
maintained
by
a
3rd
party
and
saved
to
disk
• Durability
maintained
azer
restart
3/24/13
©
2012
RTI
•
COMPANY
CONFIDENTIAL
37

38. Keeping
a
“last
value”
cache
• A
last-­‐value
cache
is
already
built-­‐in
into
every
Writer
in
the
system
• A
late
joiner
will
automa9cally
ini9alize
to
the
last
value
• Last
value
cache
can
be
configure
with
history
depth
greater
than
1
• The
Persistence
Service
can
be
used
to
provide
a
last
value
cache
for
durable
data
3/24/13
©
2012
RTI
•
COMPANY
CONFIDENTIAL
38

39. Monitoring
the
health
of
applica9ons
• DDS
has
mechanisms
to
monitor
presence,
health
and
ac9vity
of
all
en99es
• Supports
a
concept
of
liveliness
– Automa9c
• Managed
by
the
infrastructure
– Manually
asserted
• Managed
by
the
applica9on
• What
does
it
mean
if
the
applica9on
no
longer
publishes
data?
– No
news,
good
news?
– Applica9on
has
crashed?
– Applica9on
is
disconnected?
3/24/13
©
2012
RTI
•
COMPANY
CONFIDENTIAL
39

40. Monitor
the
health
of
data-­‐objects
• Concept
of
deadline
– DDS
can
monitor
the
ac9vity
of
each
individual
data-­‐instance
in
the
system
– If
an
instance
is
not
updated
according
to
the
requirements
of
the
receiving
applica9on,
the
applica9on
is
no9fied
– Trigger
for
built-­‐in
failover
mechanism
• Concept
of
lifespan
– DDS
understands
if
a
data-­‐object
has
outlived
its
purpose
and
is
considered
‘stale’
data
3/24/13
©
2012
RTI
•
COMPANY
CONFIDENTIAL
40

41. Ensuring
a
reliable
data
delivery
• DDS
supports
a
transport
independent
efficient
reliability
protocol
– In-­‐order
delivery
– Reliable
delivery
– Detec9on
and
no9fica9on
of
data
loss
– Very
configurable
• Warning
thresholds
• Heartbeats,
gaps,
acks,
nacks,
blocking
or
non-­‐blocking
behaviour
3/24/13
©
2012
RTI
•
COMPANY
CONFIDENTIAL
41

42. Building
a
high-­‐available
system
• High
Availability
has
mul9ple
facets,
many
of
which
can
be
handled
by
DDS
– Detec9on
of
presence
• DISCOVERY
– Detec9on
of
health
and
ac9vity
• LIVELINESS,
DEADLINE,
LIFESPAN
– Survive
applica9on
and
system
failures
• DURABILITY
– Ability
to
handle
redundant
data
source
and
failover
• OWNERSHIP
– Reliable
data
transfer
• RELIABILITY
3/24/13
©
2012
RTI
•
COMPANY
CONFIDENTIAL
42

43. Manage
network
load
&
behaviour
• DDS
provides
mechanism
to
limit
the
data-­‐
rates
– Filter
by
9me
– Filter
by
content
– Par99ons
– Use
of
mul9cast
– Configured
reliability
level
– TransportPriority
QoS
policy
– LatencyBudget
QoS
policy
3/24/13
©
2012
RTI
•
COMPANY
CONFIDENTIAL
43

44. Conclusions
• DDS
is
a
mature
interna9onal
Standard
from
OMG
– Plaborm
Neutral:
Opera9ng
systems
and
Programming
Languages
– Deployed
worldwide
in
Military
systems
and
other
Demanding
real-­‐9me
applica9ons
• DDS
Is
mandated
by
DoD
for
Publish-­‐Subscribe
and
data-­‐distribu9on
applica9ons
• DDS
is
an
ideal
integra9on
plaborm
for
Intelligent
Systems
– Highly
Tunable
via
Quality
of
Service
(QoS)
– Rich
services
(persistence,
filtering,
high-­‐availability)
©
2012
RTI
•
ALL
RIGHTS
RESERVED
44

45. Find
out
more…
www.r9.com
dds.omg.org
community.r9.com
www.omg.org
demo.r9.com
www.youtube.com/real9meinnova9ons
blogs.r9.com
www.twi}er.com/RealTimeInnov
www.facebook.com/RTIsozware
www.slideshare.net/GerardoPardo
©
2012
RTI
•
ALL
RIGHTS
RESERVED
45

46. Thank You!
dds.omg.org
www.omg.org
©
2012
RTI
•
ALL
RIGHTS
RESERVED
46