1. Tuning and Troubleshooting
OpenSplice DDS Applications
OpenSplice DDS
Angelo Corsaro, PhD Hans van’t Hag.
Chief Technology Officer Product Manager
OMG DDS Sig Co-Chair OMG DDS Co-Author
PrismTech PrismTech
angelo.corsaro@prismtech.com hans.vanthag@prismtech.com

2. OpenSplice DDS
US OSUM 2013
http://www.prismtech.com/events/opensplice-user-meeting-usa-2013
Copyright
2011,
PrismTech
–
All
Rights
Reserved.

3. Part I -- Setting the Context
OpenSplice DDS

4. Kinds of Issues
When building distributed applications there are several kinds of issues you can
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
encounter. Some of the most common issues are often:
☐ Connectivity: “things” don’t seem to talk to each other
☐ Performance:the latency is too high or throughput is too low
OpenSplice DDS
☐ Scalability: your system does not scale as necessary
☐ Resource Utilization: your application is using way too much CPU, Memory,
and/or Network
This webcast explores common issues in DDS applications and demonstrates how
the tools provided by OpenSplice DDS can their diagnose and resolution

5. OpenSplice DDS
Connectivity

6. Connectivity Issues
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
☐ DDS provides dynamic discovery thus as there is network
connectivity applications that are intended to communicate
should discover each other and just work!
OpenSplice DDS
☐ What may be the cause of connectivity issues in DDS?
☐ The network is not working (did you plug the wire?)
☐ Your applications does not share the same domain (thus your mistake)
☐ The QoS of the DataWriter and the DataReader don’t match

7. OpenSplice DDS
QoS Mismatch?

8. Tip #8
Understand the QoS Model
DURABILITY LIVELINESS DEST. ORDER TIME-BASED FILTER
☐ DDS defines 22 QoS HISTORY OWENERSHIP PARTITION RESOURCE LIMITS
policies that can be
LIFESPAN OWN. STRENGTH PRESENTATION
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
RELIABILITY DW LIFECYCLE
applied to USER DATA DEADLINE DR LIFECYCLE
communication entities TOPIC DATA LATENCY BUDGET ENTITY FACTORY
to control their local as GROUP DATA TRANSPORT PRIO
OpenSplice DDS
well as end-to-end
behaviour RxO QoS Local QoS
Most of the QoS Policies that control an end-to-end property follow the so-
called Request vs. Offered (RxO) Model based on which the QoS requested
by the Consumer should not exceed the QoS Provided by the Producer. If
there is an RxO mismatch no communication will take place

9. OpenSplice DDS

10. OpenSplice DDS
Performance

11. Latency / Throughput
☐ DDS provides several QoS Policies to control latency and
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
throughput
☐ OpenSplice provides extensions such as Streams to efficiently
distribute small data (e.g. several millions samples per second)
OpenSplice DDS
☐ OpenSplice Tools such TimeBasedFilter Deadline
as the Tuner can help
find the right QoS [Inbound]
setting for achieving
Throughput LatencyBudget Latency
the desired level or [Outbound]
latency/throughput
TransportPriority

12. OpenSplice DDS
Scalability

13. Federated / Standalone Deployment
Application Application Application Application
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
OpenSplice DDS OpenSplice DDS OpenSplice DDS
Operating Operating
OpenSplice DDS
Systems Systems
Federated Deployment Standalone Deployment
☐ Performance
☐ Usability
☐ Scalability
☐ Simple Configuration
☐ Determinism
☐ Simple Deployment

14. Deployment Configuration
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
☐ The deployment mode can be changed by
a simple configuration parameter. No
With OpenSplice DDS
recompilation or re-linking! you can choose the
deployment
☐ <SingleProcess>true</SingleProcess> architecture that makes
OpenSplice DDS
the most sense for your
☐ The deployment options can be mixed at use case.
will (even within a single computing node)
☐ The same application can be deployed in “Simple when Sufficient.
Performant when Required”
federated and standalone mode (even on
the same system)

15. Deployment Configuration
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
☐ The deployment mode can be changed by
a simple configuration parameter. No
With OpenSplice DDS
recompilation or re-linking! you can choose the
deployment
☐ <SingleProcess>true</SingleProcess> architecture that makes
OpenSplice DDS
the most sense for your
☐ The deployment options can be mixed at use case.
will (even within a single computing node)
☐ The same application can be deployed in “Simple when Sufficient.
Performant when Required”
federated and standalone mode (even on
the same system)

16. Resource Utilization
OpenSplice DDS

17. Shared Memory Sizing
OpenSplice DDS

18. Shared Memory Size
☐ OpenSplice DDS shared memory size is defined in its
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
configuration file. The size defined by the default
configuration file is 10MBytes
Beware that different OS have different limitations w.r.t.
OpenSplice DDS
☐
the maximum shared memory segment that can be
allocated
☐ If you want to go beyond the OS limits you need to
change the configuration of your kernel

19. Linux
☐ The default value for the maximum shared memory segment is
32MBytes
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
☐ This default can be changed in several ways
(1) Adding this line to your /etc/rc.d/rc.local file:
OpenSplice DDS
echo “your_max_shared_memory_size” > /proc/sys/kernel/shmmax
(2) Changing the settings for the sys-limits (save changes on /etc/
sysctl.conf to maintain them across reboots):
$ sysctl -w kernel.shmmax=yourMaxValue

20. Windows
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
☐ The default maximum size for Shared Memory segments on Windows is
2GB
☐ To exend it, say to 3GB, add the /3GB the boot.ini as shown below:
OpenSplice DDS
[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)WINDOWS="Windows NT
Workstation Version 4.00"
/3GB

21. Understand Topic Keys
OpenSplice DDS

22. Topic Types & Keys
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
☐ Topic Types can define some of their attributes as keys
OpenSplice DDS
☐ Yet, even when a Topic type does not define a key the keylist
directive has to be provided -- just to tell the IDL compiler that
this is a topic

23. Topic Types & Keys
Keyful Keyless
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
enum TemperatureScale {
CELSIUS,
FAHRENHEIT, struct UniversalConstants {
KELVIN double PI;
}; double e; // Nepero Number
double g; // Gravitational Constant
double NA; // Avogadro Number
OpenSplice DDS
struct TempSensorType {
short id; double F; // Faraday Constant
float temp; double K; // Boltzman Constant
float hum; double c; // Light Speed
TemperatureScale scale; };
}; #pragma keylist UniversalConstants
#pragma keylist TempSensorType id

24. Topic Instances
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
enum TemperatureScale {
CELSIUS,
FAHRENHEIT, Writer Reader
KELVIN
};
OpenSplice DDS
2 26.0 70.0 CELSIUS 1 26.0 70.0 CELSIUS
struct TempSensorType {
short id; 1 26.0 70.0 CELSIUS 2 26.0 70.0 CELSIUS
float temp;
float hum;
TemperatureScale scale;
};
#pragma keylist TempSensorType

25. Topic Instances
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
enum TemperatureScale {
CELSIUS,
FAHRENHEIT, Writer Reader
KELVIN
};
OpenSplice DDS
struct TempSensorType { 2 26.0 70.0 CELSIUS
1 26.0 70.0 CELSIUS 2 26.0 70.0 CELSIUS
short id; 1 26.0 70.0 CELSIUS
float temp;
float hum;
TemperatureScale scale;
};
#pragma keylist TempSensorType id

26. Instance Management
☐ Each topic instance takes
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
resources (e.g. memory)
Writer Reader
☐ If you don’t choose carefully
your key, you may end-up 2 26.0 70.0 CELSIUS
OpenSplice DDS
1 26.0 70.0 CELSIUS 2 26.0 70.0 CELSIUS
creating an instance per
1 26.0 70.0 CELSIUS
sample...
☐ In addition, instances can be
explicitly disposed in order to
reclaim resources

27. Example
struct Counter {
short cid;
long count;
};
#pragma keylist Counter ???
OpenSplice DDS

28. Example
struct Counter {
short cid;
long count;
};
#pragma keylist Counter cid
OpenSplice DDS

29. Example
struct SpatialSensor {
short x;
short y;
float v;
};
OpenSplice DDS
#pragma keylist SpatialSensor ???

30. Example
struct SpatialSensor {
short x;
short y;
float v;
};
OpenSplice DDS
#pragma keylist SpatialSensor

31. Topic Instances Lifecycle
struct VehiclePosition {
string plate;
long x;
long y;
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
}; com.myco.VPos
#pragma keylist VehiclePosition
plate
OpenSplice DDS
struct VehiclePosition {
string plate;
long x;
long y;
};
#pragma keylist VehiclePosition
com.myco.VPos

32. Topic Instances Lifecycle
struct VehiclePosition {
string plate;
New
long x; “A01” 100 200
long y;
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
}; com.myco.VPos
#pragma keylist VehiclePosition
plate
OpenSplice DDS
struct VehiclePosition {
string plate;
long x;
long y; New
};
#pragma keylist VehiclePosition “A01” 100 200
com.myco.VPos

33. Topic Instances Lifecycle
struct VehiclePosition {
string plate; New
long x; “A01” 100 200
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
long y;
}; New
“B41” 57 31
#pragma keylist VehiclePosition
plate
com.myco.VPos
OpenSplice DDS
struct VehiclePosition {
string plate;
long x;
long y;
};
#pragma keylist VehiclePosition “B41” 57 31 “A01” 100 200
com.myco.VPos

34. Topic Instances Lifecycle
struct VehiclePosition {
string plate;
New
long x; “A01” 110 210 “A01” 100 200
long y;
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
}; New
#pragma keylist VehiclePosition “B41” 57 31
plate
com.myco.VPos
OpenSplice DDS
struct VehiclePosition {
string plate;
long x;
long y;
};
#pragma keylist VehiclePosition “A01” 110 210 “B41” 57 31 “A01” 100 200
com.myco.VPos

35. Topic Instances Lifecycle
New
“A01” 120 220 “A01” 110 210 “A01” 100 200
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
New
“B41” 57 31
com.myco.VPos
OpenSplice DDS
“A01” 120 220 “A01” 110 210 “B41” 57 31 “A01” 100 200
com.myco.VPos

36. Topic Instances Lifecycle
New
“A01” 120 220 “A01” 110 210 “A01” 100 200
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
New
“B41” 47 19 “B41” 57 31
com.myco.VPos
OpenSplice DDS
“B41” 47 19 “A01” 120 220 “A01” 110 210 “B41” 57 31 “A01” 100 200
com.myco.VPos

37. Topic Instances Lifecycle
New
“A01” 120 220 “A01” 110 210 “A01” 100 200
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
New
d
Dispose
“B41” - - “B41” 47 19 “B41” 57 31
com.myco.VPos
OpenSplice DDS
“B41” is ed
D47pos19 “A01” 120 220 “A01” 110 210 “B41” 57 31 “A01” 100 200
com.myco.VPos

38. Part II -- Demoes
OpenSplice DDS

39. Content
☐ Tools
☐ Configuration: OpenSplice Configurator
☐ Black-box System (regression) Testing: OpenSplice Tester
☐ White-box application Inspection & Tuning: OpenSplice Tuner
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
☐ Recording & Replay: OpenSplice RnR
☐ Shared-memory and Network monitoring: OpenSplice mmStat and ‘Wireshark’ ™
☐ Common Issues
☐ System Configuration (deployment architecture, networking-setup)
☐ System Deployment (resource usage/watermarks, statistics, reports, tracing)
☐ Tool Deployment (intrusiveness: local versus remote operation)
☐ Demo’s
☐ Tester: system browser (SHMEM vs. SP) and QoS conflict detection (Reliability)
☐ Tuner: Statistics monitoring & QoS Tuning (Latency Budget)
☐ Tester: 1-click capturing & real-time Charting (virtual attributes, scripting)
☐ RnR: scripted recording & replay with RnR-service and Tester
☐ RnR: graphical control with RnRManager

40. OpenSplice Tools

41. Total Control: OpenSplice Configurator™
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
☐ The reference tool for
configuring OpenSplice DDS
☐ Rich online guide to
configuration options
☐ Context help and parameter
validation
☐ 100% Java

42. OpenSplice Tester™
☐ “Black-Box” DDS system testing
☐ Automated testing of DDS systems
☐ Dynamic Discovery of DDS Entities
☐ Domain-specific scripting languages
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
☐ Batch execution of regression tests
☐ Debugging of distributed DDS systems
☐ System browser of DDS participants
☐ Connectivity & QoS conflict monitoring
☐ One-click definition of monitoring timeline
☐ Analysis/comparison of topic data
☐ Virtual topic-attributes to ease analysis
☐ Statistics monitoring
☐ 1-click spawning of Tuner to ‘attach’ to a remote process / federation
☐ Integrated IDE
☐ Syntax highlighting editor
☐ One-click relations between script/logs/timeline

43. Total Control: OpenSplice Tuner™
☐ “Whitebox” debug/tuning Tool
☐ Looking ‘inside’ a federation and/or
application
☐ Different perspectives
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
(participant, topic, partition)
☐ Monitoring & Tuning
☐ Inspect and Tune the app’s DDS entities
☐ Make snapshots of reader-caches
☐ Detect and resolve QoS Mismatch
☐ Inspect Statistics
☐ Reading & Writing
☐ Read/Write data for arbitrary topics
☐ Import & Export
☐ Inject Topic Definitions
☐ export and import XML-based reader/writer snapshots

44. OpenSplice RnR
☐ Dynamic recording of any topic-
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
data in a DDS system
☐ Selective replay with variable speed
☐ Distributed control by topic-based
API
☐ Seamless integration with
OpenSplice Tester (topic-based API) RnR
RnR RnR
Service
Service(s) Manager
☐ Dedicated RnR-Manager graphical
GUI for scenario-definition and data Any topic Record/Replay command
& status topics
Record/Replay command
& status topics
import/analysis
OpenSplice DDS

45. Wireshark
☐ WireShark Packet
Dissector
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
☐ Watch what goes on
the wire
☐ Inspect DDSI-RTPS and/or
RT-Networking packets
being exchanged
between applications
☐ Native RTnetworking
dissector support available in
source-distribution
☐ OpenSpliceDDSV6.2.1HDEx86.win32toolswireshark-pluginsospl (see README)

46. mmStat
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
mmstat -h
mmstat [-M|m] [-e] [-a] [-i interval] [-s sample_count] [URI]
mmstat [-t|T] [-i interval] [-s sample_count] [-l limit] [-f filter_expression] [URI]
Mode:
-m Show memory statistics (default mode)
-M Show memory statistics difference
-t Show meta object references
-T Show meta object references difference
-h Show this help
-e Extended mode, shows bar for allocated memory
-a Show pre-allocated memory as well.
-i interval Display interval (in milliseconds)
-s sample_count Stop after sample_count samples
-l limit Show only object count >= limit
-f filter_expr Show only meta objects which name passes the filter expression

47. Common Issues: Examples & Demo’s

48. 11
Common issues: System Configuration
☐ Selecting the proper deployment architecture
☐ Federated / shared-memory (for scalability and data/application lifecycle decoupling)
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
☐ Standalone / single-process (for simplicity yet at the cost of ‘network-scheduling determinism’)
☐ Selecting the proper networking architecture
☐ RTNative networking: ‘push’ based architecture using pre-configured (multicast) network-partitions
☐ Pro : zero discovery times (optional topology-discovery only needed for fast disconnect-reaction times)
☐ Con : push implies data might show-up at places where there’s no actual/current interest .. Yet tunable in OpenSplice
☐ DDSI networking: ‘pull’ based architecture using advertized locators
☐ Pro : selective distribution to ONLY the set of discovered participants, using unicast whenever just 1 destination
☐ Con : (standardized) discovery-protocol has scalability limitations (amount of heartbeats) .. Yet tunable in OpenSplice
☐ Configuring your domain
☐ Using “OpenSplice Configurator” to aid in correct configurations
☐ Assure system-wide consistent values
☐ Domain-ID’s and in case of RTNetworking the related port-numbers (channels and discovery)
☐ RTNetworking only: NetworkPartition definitions (names and related [multicast-] addresses)

49. Common issues: System Deployment
13
☐ How to Prevent Issues
☐ Using the Configurator to validate configurations [demo]
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
☐ Assuring system-wide consistent networking/durability setup
☐ Using ResourceLimits policies in your applications
☐ How to detect Issues
☐ Setting resource-usage ‘watermarks’ with the configurator [demo]
☐ Checking ospl_info.log and ospl_error.log files [demo]
☐ Checking QoS RxO conflicts with the Tester and/or Tuner [demo]
☐ Capturing behavior with Tester (and [auto-]creation of scripts to reproduce)
☐ Recording of information using RnR (manually and/or scripted) [demo]
☐ How to resolve issues
☐ Runtime Tuning of QoS policies with the Tuner [demo]
☐ Exploring reader/writer/service statistics with Tester & Tuner [demo]

50. Demo’s

51. Demo’s
☐ Environment
☐ Perf1/2/3 reference-system (shmem versus SP)
☐ Ishapes demo-application
☐ Configurator / Tester / Tuner / RnR tools
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
☐ Configurator
☐ Used configuration-files (SHMEM/RnR, SHMEM/Statistics, SP/Resource-limits
☐ Demo-config: perf1: RnR added, perf2: statistics added perf3: resource-limits
☐ Tuner/Tester ‘ishapes’ demos
☐ Tester: system browser (SHMEM vs. SP)
☐ Tester: QoS conflict detection (Reliability)
☐ Tuner: QoS Tuning (Latency Budget)
☐ Tuner: Status & Statistics (Writer/Reader, perf2: Network-queues, perf3: Network receiving)
☐ Tester: real-time Charting (virtual attributes, scripting)
☐ Recording & Replay
☐ Scripting with Tester
☐ Controlling with RnRManager

52. Demo setup
remote Tester-2
Remote Remote
Splice-
Tuner 1 Tuner 2
Hans
Office-net
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
Router
Switch Test-net
Tester-1/ Ishapes Ishapes Ishapes
RnRManager Federated Federated standalone
Shared Memory Shared Memory S N
O DDS E
A
soap RT-Net RnR / Dur. Soap RT-Net Durability P
libs T
W
Perf1 Perf2 Perf3
Switch Perf-net

53. Teaser: ‘Streams’
☐ Bundled Example
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
☐ Streaming 10 mln. Samples
☐ Throughput: up to 29 mln. Samples/s ! (shmem/i7)

54. Configuration
Copyright
2013,
PrismTech
–
All
Rights
Reserved.
☐ Perf1
☐ Ospl_shmem_native_rnr.xml
☐ Perf2
☐ Ospl_shmem_native_statistics.xml
☐ Perf3
☐ Ospl_sp_native_resource_limits.xml

55. QoS conflict detection
Copyright
2013,
PrismTech
–
All
Rights
Reserved.

56. QoS Tuning
Copyright
2013,
PrismTech
–
All
Rights
Reserved.

57. Statistics Monitoring
Copyright
2013,
PrismTech
–
All
Rights
Reserved.

58. Data-Capture
Copyright
2013,
PrismTech
–
All
Rights
Reserved.

59. Scripted RnR
Copyright
2013,
PrismTech
–
All
Rights
Reserved.

60. Graphical RnR
Copyright
2013,
PrismTech
–
All
Rights
Reserved.

61. OpenSplice DDS

62. References
Copyright
2011,
PrismTech
–
All
Rights
Reserved.
OpenSplice | DDS Escalier
¥ #1 OMG DDS Implementation ¥ Fastest growing JVM Language ¥ Scala API for OpenSplice DDS
¥ Open Source ¥ Open Source ¥ Open Source
¥ www.opensplice.org ¥ www.scala-lang.org ¥ github.com/kydos/escalier
OpenSplice DDS
¥ Simple C++ API for DDS ¥ DDS-PSM-Java for OpenSplice DDS ¥ DDS-based Advanced Distributed
¥ Open Source ¥ Open Source Algorithms Toolkit
¥ github.com/kydos/simd-cxx ¥ github.com/kydos/simd-java ¥ Open Source
¥ github.com/kydos/dada

63. :: Connect with Us ::
¥opensplice.com ¥forums.opensplice.org
¥@acorsaro
¥opensplice.org ¥opensplicedds@prismtech.com ¥@prismtech
OpenSplice DDS
¥ crc@prismtech.com
¥sales@prismtech.com
¥youtube.com/opensplicetube ¥slideshare.net/angelo.corsaro