Distributed Deployment Model Driven Development

Distributed Deployment

version 1.0
12/13/11

PathMATE Technical Notes

Pathfinder Solutions
Wrentham, MA, USA
www.pathfindermda.com
+1 508-568-0068

copyright 1995-2011 Pathfinder Solutions LLC, all rights reserved

Table Of Contents
1 Introduction .......................................................................................... 1

2 Overview............................................................................................... 1
2.1 Design Elements............................................................................ 1
2.1.1 Task...................................................................................... 1
2.1.2 Process.................................................................................. 2
2.1.3 Processor............................................................................... 2
2.2 Configurations and Connections ....................................................... 3
2.3 Modeling in the Multi-Process World ................................................. 6
2.3.1 NodeManagement ................................................................... 7
2.3.2 Routing Parameters ................................................................. 7
2.3.3 Intertask Contention and the PfdSyncList ................................... 7

3 Deploying to Distributed Topologies...................................................... 7
3.1 Structural Design Elements ............................................................. 8
3.2 Topology Identification and Allocation ............................................... 8
3.2.1 Task Identification................................................................... 8
3.2.2 Task Priorities......................................................................... 9
3.2.3 ProcessType Definition ............................................................10
3.3 Topology Definition – the Process Configuration Table ........................12
3.3.1 Single Process Single Deployment ............................................13
3.3.2 Single Process Multi Deployment (SPMD) ...................................13
3.3.3 Process Configuration Table .....................................................13

4 Messaging and Connectivity ................................................................ 15
4.1 Elements .....................................................................................15
4.2 Interprocessor and Endianness .......................................................16
4.2.1 Build-in Type Data Item Serialization ........................................18
4.2.2 User-Defined Type Serialization – Advanced Realized Types .........18
4.3 Communication Errors ...................................................................19
4.3.1 Multi-Task.............................................................................19
4.3.2 Model-Level Interface – SW:RegisterForErrorGroup.....................19
4.4 Generated Topology Summary Report .............................................20
4.5 Trace Debugging ..........................................................................20

A. Thread Proliferation in Hand Written Systems .................................... 22
PI-MDD, Asynchronous Domain Interactions, and The Event-Driven Paradigm.22

Table Of Figures

ii

Figure 1: Default Single-Task System Topology .................................................. 3
Figure 2: Example Two-Task System Topology ................................................... 3
Figure 3: Simplified Task Symbology ................................................................ 4
Figure 4: Multi-Task, Multi-Processor Topology ................................................... 5
Figure 5: Simplified Process Symbology ............................................................ 6
Figure 6: SPMD Deployment ............................................................................ 6
Figure 7: Default System Topology ................................................................... 8
Figure 8: One Process Two-Task System Topology .............................................. 9
Figure 9: Example Multi-Task, Multi-Processor Topology .....................................12
Figure 10: Example SPMD Deployment.............................................................14

iii

Technical Note: Distributed Deployment

1 Introduction
The PI-MDD method for constructing software for complex and high performance systems
separates the complexities of the problem space subject matters from the strategies and
details of the implementation platforms the system executes within. While programming
language and target operating system are key aspects of this implementation platform, they
represent one-time selections, often made by organizational inertia. Perhaps the most
demanding facet of this separation discipline is that the most complex and creative aspects of
platform revolve around deploying a complex software system across a distributed execution
unit topology with a range of processors, processes and tasks.

Modern software systems at nearly all levels of complexity execute with some degree of
parallel execution. Even the simplest systems apply separate threads of control to manage the
lowest level of communication input and output. Conceptually coherent “single” systems often
involve highly synchronized activities executing on separate processors. In PI-MDD one of the
most fundamental disciplines pushes nearly all aspects of this complexity from the modeling
space. So how is this clearly important, nearly ubiquitous, and non-trivial concern addressed
with PI-MDD and PathMATE? The distributed deployment of PI-MDD models to multiple
execution units is managed with an integrated set of model markings (properties), specialized
code generation rules/templates and a flexible set of implementation mechanisms. The PI-
MDD models remain independent of this target environment topology, and can be deployed to
a range of topologies to support unit testing, system testing, and alternative deployment
architectures.

2 Overview
2.1 Design Elements
2.1.1 Task
Task – a separate thread of execution control that can run interleaved with or in parallel with
other tasks in the same Process. Tasks within the same Process can address the same
memory space. Tasks are supported by specific OS-level mechanisms. Even on “bare boards”
without a multi-tasking OS, there is a main Task, and interrupt vectors are considered to run in
a second Interrupt Task.

Modeled Task – a Task that was explicitly identified as part of the deployment of model
elements through the TaskID marking. Modeled tasks are managed automatically by
PathMATE mechanisms, including start, stop, nonlocalDispatchers, inter-task incident queuing,
and task-local memory pools. One instance of the PfdTask class (SW_Task in C) manages
each Modeled Task that is executing modeled elements.

Each Modeled Task is designated to run on a specific ProcessType.

Note: When a realized domain is marked to run in a specific task, the resulting Task is still a
Modeled Task because it was created from marking a model element – the realized domain,
and have all the conveniences of modeled domains in this regard. For the purposes of this
document, a reference to a Task from here forward will mean a Modeled Task unless explicitly
designated otherwise.

1


Incident – An instance of a modeled Signal or of an IncidentHandle (callback) is an Incident.
Each Incident is designated to run within a specific Task, by the Task allocation of either its
destination class or target service/operation.

NonlocalDispatcher – For each domain service and class operation that can be invoked (from
PAL) from another task, a corresponding method is automatically generated to provide a
function that can be directly invoked at the code level from any Task. A nonlocalDispatcher
creates an IncidentHandle to its corresponding domain service or class operation and delivers
this incident.

Realized Task – a task that was not identified as part of the deployment of model elements,
and is controlled either by PathMATE mechanisms or realized code independent of the Modeled
Tasks. Realized tasks may be started by PathMATE to support specific mechanism execution –
eg communication receiver and sender tasks – or may be started and managed by completely
by realized code and therefore are unknown to PathMATE mechanisms. For the purposes of
this document, any reference to a Realized Task from here forward will be explicitly
designated.
2.1.2 Process
Process – Generally contained within an executable, a Process is a set of one or more Tasks
that can run interleaved with each other, or genuinely in parallel, controlled by a single
scheduler, and sharing a memory space. Separate Processes cannot address the same general
memory space. PathMATE supports the external identification of a process via IP address and
port. One or more Processes may run on a Processor. Some deployment environments
(RTOSs) will only support a single Process running on a single processor. For the purposes of
this technical note in this scenario where a Processor is executing a single executable with one
or more Tasks will be termed to have a single Process executing. One instance of the
PfdProcess class (SW_Process in C) manages each Process that is executing modeled
elements.

Process Identifier (PID) - A numerical value used to uniquely address each process instance
within an intercommunicating set of PathMATE-based Processes. This is specified in the
Process Configuration Table and can be used as part of a Routing Parameter.

Process Type - A “type” of executable built with a potentially unique set of PI-MDD elements,
and with a specific set of conditional compiler directives, deployed together as a Process.
Typically each different executable is a Process Type. One or more instances of a Process Type
may be running at any give time in a system, each with their own unique Process Identifier.
Model elements are deployed to a Process Type via the ProcessType marking.

Process Configuration Table – A table constructed at runtime within each PathMATE Process
that identifies (Process Type and Process ID) itself and the process instances that it can
connect to. This table is either loaded from a file at startup time or may be constructed from
interprocess (socket) messages received throughout system processing. The Process
Configuration Table in each process instance must have consistent subsets of information with
all other Process that it can connect to.
2.1.3 Processor
Processor – a distinct physical processor, that may run one of more Processes. For the
purposes of this technical note the Processor is a secondary concept, and the Process is used
to bound deployed elements that run on a specific Processor.

2


2.2 Configurations and Connections
2.2.1 Single Task
From an application perspective, the work of the system is done in the Tasks by PIM
processing – model actions executing. Each Task performs this work by dispatching Incidents
which may cause actions to run. The Incidents are held in an Incident Queue.

Figure 1: Default Single-Task System Topology

2.2.2 Multi Task
If two or more Tasks run within a Process, then a task-safe Inter-Task Incident Queue (ITIQ) is
used to queue Incidents between Tasks. PIM Processing may generate Incidents that are
queued locally, or on another Task’s ITIQ.

Figure 2: Example Two-Task System Topology

3


This internal Task structure is common for all Modeled Tasks. For simplicity from here forward
Tasks will be shown as a simple outline and name:

Figure 3: Simplified Task Symbology

2.2.3 Multi Process
When a System is deployed to two or more Processes, TCP/IP sockets are used to convey
messages between them. Incidents – service handles or events – are send between processes
and dispatched upon receipt to PIM tasks where they initiated PIM processing. Each incident to
be sent is converted to a PfdSockMessage through Serialization, where the data members of
the incident and it’s parameter values are encoded into the stream of bytes within the
PfdSockMessage.

In each Process a number of realized communication tasks are automatically created and
managed by the PfdProcess. One receiver task receives messages from any source, and a
sender task for each destination Process instance manages the connection with and sends
messages to that process. Each sender task has an Inter-Task Message Queue (ITMQ) where
each outbound PfdSockMessage is queued by local PIM processing for sending. The ITMQ is
implemented with a PfdSyncList containing instances of PfdSockMessageOut.

4


In the topology shown below there is a single instance of ProcessType MAIN and a single
instance of ProcessType DEVICE_CONTROL.

Figure 4: Multi-Task, Multi-Processor Topology

5


This internal Process structure is common for all Processes. For simplicity from here forward
Process will be shown as a simple outline and name, containing only their Modeled Tasks:

Figure 5: Simplified Process Symbology

2.2.4 Multi Process SPMD
In some systems there are can be more than one instance of a Process of a single
ProcessType. This general capability is termed Single Process Multi Deployment (SPMD).
Often each different instance is deployed to its own processor. Note the varying PIDs in the
diagram below.

Figure 6: SPMD Deployment

2.3 Modeling in the Multi-Process World
With the underlying asynchronous nature of PI-MDD model element interactions and the
explicit PI-MDD preference of asynchronous interactions at the domain level (and below),
proper PI-MDD models should generally be well formed for a range of deployment options.
Certainly this is a solid start, and leaves the Application Modeler and the Target Deployment

6


Designer on good joint footing to develop and effective deployment. But at some point there
may be aspects of the PI-MDD model that may need adjustments to facilitate proper
deployment.
2.3.1 NodeManagement
In systems that fully utilize complex multi-processing target environments, often times there is
explicit interaction between high-level application control (“mission-level” logic) and detailed
aspects of the platform and its resources. In this realm these interactions form a bona fide
Problem Space subject matter: NodeManagement. A Domain can be created to encapsulate
interactions with topology awareness, local processing resources and multi-process
communication mechanisms. This can alleviate application domains of the need to somehow
to intelligently and flexibly respond to details about topology and resources localize these
capabilities in one component.
2.3.2 Routing Parameters
To explicitly route the invocation of a domain service to specific process id/task id combination,
a domain service can specify a parameter as a <<Routing>> parameter. A parameter of type
DestinationHandle or Group<DestinationHandle> can have it’s Stereotype marking set to
Routing, which causes its nonlocalDispatcher to be generated with appropriate routing code,
using the parameter runtime value(s).

But where does a caller get the right values? Often times a class can have the proper routing
information instantiated at startup (via a static initializer, XML file or binary instance data) and
use these attributes. The SoftwareMechanisms domain services DestinationHandleLocal() and
DestinationHandleRemote() provide encoding of specified Process IDs and Task IDs into
DestinationHandles at runtime.

In systems with more complex or dynamic topologies, a NodeManagement domain can
maintain the appropriate body of routing data. It can publish services to provide this data at
the level of abstraction appropriate to preserve the application’s independence of specific
topologies. The caller can then go to NodeManagement to get timely and appropriate routing
information.
2.3.3 Intertask Contention and the PfdSyncList
A simple starting point for deploying a domain within a Process Type is to allocate the domain
– in its entirety – to execute within a single task. This way there is no contention between
tasks because domain share no resources that they need to explicitly protect.

However there are legitimate design contexts where it is advantageous to deploy a single
domain to multiple tasks in a single process. In this context a danger emerges when an
element of the domain accesses a shared assets from one task when the same asset can be
accessed from another task. In this case a protection mechanism is required. These shared
assets are class instance containers – both for an association (across from a many participant)
and for a class instance population. The ThreadSafe marking is supported on the Association
and the Class. Setting it to “T” will generate a PfdSyncList for the instance container, which
uses an internal PfdReentrantMutex to make access safe for intertask use.

3 Deploying to Distributed Topologies
Structural Design is the PI-MDD activity where the execution units for the deployment
environment are identified and model elements are allocated to them.

7


3.1 Structural Design Elements
The construction, interconnection and deployment of multi-processor systems from PI-MDD
models requires:
- The Model: While the Domain is the primary initial focus for deployment, elements
within a domain can also be separately allocated to their separate execution units,
including the Domain Service, Class, Class Operation, and State.
- The Topology: Tasks and process types are identified via markings ProcessType and
TaskID, applied to a range of model elements.
- Generated Code, Projects: While general mechanism layer code exist to support
Distributed Deployment, nothing specific to any topology exists until after
Transformation. Markings drive the generation of build folders for each ProcessType
and code tailored to the target topology.
- Run-Time Data: Each specific instance of a Process is identified via a Process ID in the
Process Configuration Table. While the actual image file code content for each instance
of a given ProcessType is identical (they all use the same executable file), individual
Process instances can be configured to behave differently via class instance and link
data. These can be captured in XML or binary instance data files which are deployed
with the correct process instance and/or working directory.

3.2 Topology Identification and Allocation
The identification of execution units starts with simple default allocations. If no topology
markings are specified at all, the default configuration is a single ProcessType named “MAIN”,
with a single Task named “SYS_TASK_ID_MAIN”.

Figure 7: Default System Topology

In this configuration the system project files are generated with compiler symbol settings that
hide the inter-task and inter-process code in the PathMATE Mechanisms layer. While realized
communications code can always be included in the system, it will not have any PathMATE
generated communications mechanisms.
3.2.1 Task Identification
Additional Tasks are identified by the “TaskID” marking, which can be applied to analysis
elements of type Domain, DomainService, Object, or ObjectService. The default allocation for
Domains is SYS_TASK_ID MAIN. The default TaskID for all other elements is the TaskID of
their containing element.

8


A TaskID can be set to any identifier name, however by convention all fixed task ids are in the
form SYS_TASK_ID_<name>. The task id SYS_TASK_ANY has special meaning, indicating all
actions in the marked element are executed locally within the calling task.

An additional special TaskID value, DYNAMIC, indicates a new task is started, or retrieved from
the task pool and the object/service is run in that task. Instances of DYNAMIC classes This
rich capability is described in the separate Tech Note “PathMATE Dynamic Tasks”.

Example: Deploy a system to a topology with two tasks: SYS_TASK_ID_MAIN and
SYS_TASK_ID_LOGGING. The following marking deploys the Instrumentation domain to the
‘LOGGING task, thereby causing the generation of a multi-task system:
Domain,*.Instrumentation,TaskID,SYS_TASK_ID_LOGGING

This one marking line causes the system to deploy in this configuration, with the
Instrumentation domain actions running in the “PIM processing” bubble in the
SYS_TASK_ID_LOGGING task, and the remainder of the system running in the “PIM
processing” bubble in the SYS_TASK_ID_MAIN task:

Figure 8: One Process Two-Task System Topology

An invocation to an Instrumentation domain service from PIM processing (a model action)
within the SYS_TASK_ID_MAIN task domain is generated as a call to that domain service’s
nonlocalDispatcher counterpart. This nonlocalDispatcher creates a PfdServiceHandle to the
target service on the SYS_TASK_ID_LOGGING task and delivers it – placing it on the ITIQ for
SYS_TASK_ID_LOGGING.
3.2.2 Task Priorities
The enumeration pfdos_priority_e defined in pfd_os.hpp specifies the task priorities that are
generally available:
 SYS_TASK_PRIORITY_HIGHEST
 SYS_TASK_PRIORITY_HIGHER
 SYS_TASK_PRIORITY_NORMAL

9


 SYS_TASK_PRIORITY_LOWER
 SYS_TASK_PRIORITY_LOWEST
Modeled Tasks
Any model element that can be marked with TaskID can also optionally be marked with
TaskPriority, specifying one of the above values. The default is
SYS_TASK_PRIORITY_NORMAL.
NOTE: all model elements that explicitly set a TaskID must all have the same TaskPriority.

Mechanisms Realized Tasks
The priorities of realized tasks started by PathMATE mechanisms are controlled by the following
compiler symbols.
task priority symbol description default value
PFD_MAIN_OOA_THREAD_PRIORITY OOA processing task SYS_TASK_PRIORITY_NORMAL
PFD_RECEIVER_THREAD_PRIORITY TCP receiver task SYS_TASK_PRIORITY_NORMAL
PFD_TRANSMIT_THREAD_PRIORITY TCP sender task SYS_TASK_PRIORITY_NORMAL
PFD_IE_THREAD_PRIORITY Spotlight connection task SYS_TASK_PRIORITY_LOWEST
PFD_INPUT_THREAD_PRIORITY Driver input task SYS_TASK_PRIORITY_NORMAL

The following definition pattern in pfd_os.hpp supports the external specification of a priority
for each type of realized task:
#ifndef PFD_TRANSMIT_THREAD_PRIORITY
#define PFD_TRANSMIT_THREAD_PRIORITY SYS_TASK_PRIORITY_NORMAL
#endif
In this manner the system Defines marking can be used to override default realized task
priorities.
3.2.3 Task Stack
By default, the stack size allocated to each Task is controlled by OS defaults. When the task is
started a 0 is passed into the OS call and it determines the size of the stack.
You can specify a default stack size for all Tasks via the DefaultStackSize system marking, eg:
System,SimpleOven,DefaultStackSize,500000

In addition to controlling size, the actual memory allocated for use as the task stack can be
allocated explicitly, allowing custom realized code to monitor it for overrun, etc. By marking
the system with a non-0 DefaultStackSize and defining the compile flag
PATH_ALLOCATE_EXPLICIT_STACK, a stack is explicitly allocated in pfd_start_task_param(). If
the Task being started is a modeled task, the instance of PfdTask that it is running has it’s
stack data members set with this info. (Platform support may vary; inspect pfd_os.cpp
pfd_start_task_param() to see if your platform is supported.)
3.2.4 ProcessType Definition
By default a single ProcessType is generated – MAIN. Build file generation produces a single
build folder called MAIN, and produces a single executable. By default all domains and domain
services are allocated to the ProcessType MAIN. Additional ProcessTypes can be defined by
allocating a domain or domain service via the ProcessType marking. For example
Domain,*.ExternalDeviceControl,ProcessType,DEVICE_CONTROL

Once more ProcessTypes are defined (in addition to MAIN) code is generated to communicate
between multiple processes. Each ProcessType generates a build folder with the ProcessType
name, and generates code that starts the domains and tasks configured for that ProcessType.

10


The code also automatically starts the PfdProcess’s realized communication tasks to handle
inter-process communications.

With the definition of ProcessTypes, PfdIncident::deliver() – used by nonlocalDispatchers – is
extended where needed to handle inter-process invocation scenarios, automatically using the
PfdProcess’s realized communication tasks.

Example: Deploy a system to a topology with two ProcessTypes. ProcessType MAIN has two
tasks - SYS_TASK_ID_MAIN and SYS_TASK_ID_LOGGING. ProcessType DEVICE_CONTROL
has two tasks - SYS_TASK_ID_MAIN and SYS_TASK_ID_REGISTER_IO. The following marking
deploys the DeviceControl and HardwareIF domains to the DEVICE_CONTROL ProcessType,
thereby causing the generation of a multi-process system:
Domain,*.DeviceControl,ProcessType,DEVICE_CONTROL
Domain,*.HardwareIF,ProcessType,DEVICE_CONTROL
Because all other domains by default are allocated to ProcessType MAIN, and we want
SoftwareMechanisms services to run locally wherever they are called, we allocate this domain
to ProcessType ANY, TaskID SYS_TASK_ANY:
Domain,*.SoftwareMechanisms,ProcessType,ANY
Domain,*.SoftwareMechanisms,TaskID,SYS_TASK_ANY
To retain the Instrumentation deployment configuration, and to run the HardwareIF domain in
its own task we add these markings:
Domain,*.Instrumentation,TaskID,SYS_TASK_ID_LOGGING
Domain,*.HardwareIF,TaskID,SYS_TASK_ID_REGISTER_IO

These 6 markings cause the system to deploy in this configuration:

11


Figure 9: Example Multi-Task, Multi-Processor Topology

To specify a default process type other than MAIN, the default ProcessType can be changed via
the system marking DefaultProcessType, for example:
System,*,DefaultProcessType,MY_PROCESS

3.3 Topology Definition – the Process Configuration Table

12


3.3.1 Single Process Single Deployment
Once an executable is built for each ProcessType, one or more instances of the executable may
be started. By default PathMATE assumes a simple configuration where a single instance of
each ProcessType is run. The PathMATE PfdProcess class has a Process Configuration Table to
keep track of the configuration of Process Instances that run in the system. Each Process
tracks the processes it can connect to with the following information:
Process ID ProcessType IP address TCP port

Table 1: Process Configuration Table Fields

A key stop in understanding this Process Configuration Table is in the generated
gc/sys/system_indices.hpp, where the ProcessTypes constants are defined:
/* define process types for the system */
enum
{
SYS_PROCESS_TYPE_MAIN = 0,
SYS_PROCESS_TYPE_DEVICE_CONTROL = 1,
SYS_PROCESS_TYPE__count = 2
};

The generated code includes a default Process Configuration Table that allows one instance of
each ProcessType. The example system depicted in Figure 9: Example Multi-Task, Multi-
Processor Topology has the following default Process Configuration Table:
0 0 127.0.0.1 5501
1 1 127.0.0.1 5502

Table 2: Default Process Configuration Table

3.3.2 Single Process Multi Deployment (SPMD)
Some system configurations call for more than one instance of a given ProcessType be
executing at the same time. Essentially this permits two or more entries in the Process
Configuration Table to specify a given ProcessType.

In cases where a single instance of a given ProcessType is configured, the system can continue
to use implicit routing of services to that process based only on ProcessType, as was done in
the Single Process Single Deployment configuration. However for cases where two or more
instances of a given ProcessType are configured, the system requires a way for the application
to control in which process instance a given services will run.

A Domain Service with a parameter marked as a Routing Parameter routes the service
invocation to the Process ID and Task specified by this parameter. In this manner a PathMATE
system can utilize multiple Process Instances of the same Process Type.
3.3.3 Process Configuration Table
To specify a custom target topology a Process Configuration Table is constructed in a comma-
separated file, specifying a row in the table for each Process instance. Each row must have a
unique Process ID and combination of IP address and TCP port.

This example table shows our system with 4 instances of the DEVICE_CONTROL process
running simultaneously, as depicted below (SimpleOven_with_4_DCs.txt):

13


0 0 43.2.0.1 6000
1 1 43.2.1.21 7001
2 1 43.2.1.31 7002
3 1 43.2.1.41 7003
4 1 43.2.1.51 7004

Table 3: Example SPMD Process Configuration Table

Figure 10: Example SPMD Deployment

The location of the Process Configuration Table file is provided to the process executable at run
time via the “-config” command line arguments, eg:
SimpleOven-DEVICE_CONTROL.exe -config SimpleOven_with_4_DCs.txt

A simple strategy to ensure each Process Instance has process configuration information
consistent with all other Process Instances is to build a single Process Configuration Table file
and provide it to all Process Instances when they start. However if some subsets of Process
Instances do not communicate with other subsets of Process Instances, each Process Instance
can have it’s own version of Process Configuration Table. Not all copies need to contain all
Process Instance entries. The following rules apply:
 When specified, each Process Instance (identified by Process ID) must specify the same
Process Type, IP Address and TCP port in all Process Configuration Table files
 A Process Instance can only send messages to a Process Instance it knows about via its
own Process Configuration Table file

14


3.3.3.1 Additional Fields – UDP Port and Spotlight Port
To support external realized code that may require UDP port information and to allow the
manual specification of Spotlight ports, the Process Configuration Table carries two additional
fields for this information:
Process ID ProcessType IP address TCP port UDP port Spotlight Port

Table 4: Additional Process Configuration Table Fields

The Spotlight port is used by the Spotlight instrumentation code. In addition, these static
methods are provided by the PfdProcess class for realized code-level access to the topology
information:
// PROCESS TABLE ACCESSORS

// Get IP address for specified process
static int getPidFromIpAddress(int ip_address);
// Get IP address for specified process
static int getIpAddress(DestinationHandle dest);
// Get TCP port for specified process
static int getTcpPort(DestinationHandle dest);
// Get UDP port for specified process
static int getUdpPort(DestinationHandle dest);
// Get Spotlight port for specified process
static int getSpotlightPort(DestinationHandle dest);
// Get Spotlight port for current process
static int getSpotlightPort();

3.3.3.2 Adding to the Process Configuration Table at Run Time
Immediately preceding the first message to carry application data, a connect message is sent
to the destination Process Instance. The connect message carries the sender’s PID, IP address
and TCP listener port. If the destination Process Instance did not already have an entry in its
Process Configuration Table for the sender Process Instance, one is added.

3.3.3.3 Minimal Process Configuration Table
To facilitate deployments to targets without files systems or otherwise without topology
configuration files, a Process Instance can discover and build their system topology table at run
time. To suppress the default process configuration, start the process executable with the “-
pid” and “-port” command line argument,s eg:
SimpleOven-DEVICE_CONTROL.exe –pid 4 –port 7004

Do not specify a process config table. Other processes connecting to it will add to its internal
process config table via the information provided in their connect messages.

Key limitation: a Process Instance with a Minimal Process Configuration Table cannot send
messages to other processes until those processes send a message to this Process Instance
first.

4 Messaging and Connectivity
4.1 Elements
The fundamental unit of messaging in a PI-MDD model is the IncidentHandle. This is a
PathMATE Mechanism that identifies a specific domain service, class operation or signal to be
invoked/generated, and carries the parameter values needed for any input parameters defined

15


by that service. An asynchronous mechanism, IncidentHandles cannot be constructed for a
service/operation with return values or output parameters. IncidentHandles are often referred
to by the term Incident. There are two subtypes of IncidentHandle – the Event and the
ServiceHandle. ServiceHandles handle domain service and class operation invocations, so they
are the type of IncidentHandle most commonly encountered.

In a PI-MDD model the invocation of a domain service or class operation is specified in PAL
with a construct that looks just like a synchronous function/procedure/method invocation from
common implementation languages. Depending on the marking of the model for topology, the
tasks/process context for the caller may be in a different execution unit (task/process) than
the action being invoked. This requires the resolution of the PAL invocation with a range of
possible implementations. The following term classify these resolutions:
- Local: the caller and the target service/operation are within the same task
- Inter-task: the caller and the target service/operation are within the same process,
but between different tasks
- Inter-process: the caller and the target service/operation are between different
processes (there is no distinction between node-local inter-process and inter-processor
inter-process communications)

incident type locality communication mechanism
Operation Local synchronous function/method invocation; local
Invocation Local IncidentHandle dispatch
Inter-task convert(*1); queue on target task ITIQ
convert(*1); send via socket; upon receipt queue on
Inter-Process target task ITIQ
IncidentHandle
CALL Local local dispatch
Inter-task queue on target task ITIQ
send via socket; upon receipt queue on target task
Inter-Process ITIQ
NOTES:
*1 - The operation invocation is converted automatically to an IncidentHandle.

Table 5: Incident Communication Mechanisms

4.2 Connection Initiation and Reconnection
When a Process starts up, the Process Configuration Table is established as outlined in section
3.3.3 Process Configuration Table. For each row in the Table a Connection is created to
maintain the current state of the messaging connection to each known remote process.
However at this time no actual connections to other Process instances are started. Instead
they are started on demand, when an action in the local Process causes a message (incident)
to be sent to a remote Process.

4.2.1 Connect-on-Initial-Send
When an outbound message is queued, the state of the Connection to the destination Process
is checked. If it is not connected at this time connection is initiated. This initial outbound
message is held in a queue until the connection is successfully established. Then the message,
along with any others that may have been queued up, are send to the destination. Once all

16


are sent, the connection is left up – in a usable state. Later as other messages are send to
this same destination, they can simply be sent using the established connection.
4.2.2 Retry Initial Connection Delay
If the initial attempt to establish a connection to a remote Process fails, reconnection will be
attempted every PATH_DELAY_BETWEEN_RECONNECT_ATTEMPTS_MS milliseconds. If not specified
by the user, this compiler symbol has a default value of 1000.
4.2.3 Reconnecting an Established Connection
If a connection fails that had already been established and is in use, a priority is put on trying
to quickly restore it. A limited number of reconnection attempts are tried immediately after
failure is detected, without any delay between them. The number is controlled by the compiler
symbol PATH_IMMEDIATE_RECONNECT_MAX_COUNT. If not specified by the user, this compiler
symbol has a default value of 5. If the connection cannot be reestablished within
PATH_IMMEDIATE_RECONNECT_MAX_COUNT iterations, reconnection attempts are continued every
PATH_DELAY_BETWEEN_RECONNECT_ATTEMPTS_MS milliseconds.
4.2.4 Inter-Send Delay
The default task priorities place sender tasks at a lower priority, and this is expected to allow
for PIM processing to continue even if sender tasks are fully occupied by outbound message
traffic. To facilitate the even distribution of message activity across all sending tasks and the
receiver task, a “sleep” (via pfd_sleep) of 0 milliseconds is used after each individual socket
send call to release the processor if needed. This sleep also happens between individual socket
message packet sends for large messages. (large as defined by your current TCP-IP stack
configuration).
If the user wishes to further throttle outbound message traffic,
PATH_DELAY_BETWEEN_MESSAGE_PACKET_SENDS_MS can be specified. If not specified by the
user, this compiler symbol has a default value of 0.

Alternatively if the user wants to eliminate the inter-packet “sleep” altogether,
PATH_DELAY_BETWEEN_MESSAGE_PACKET_SENDS_MS can be set to -1, and the pfd_sleep call is
skipped.

4.3 Outbound Message Queues
When a PIM processing Task delivers an Incident to be sent interprocess, it is serialized into a
PfdSockMessageOut and placed on the outbound message queue for the Connection
corresponding to its destination. The queue is a PfdSyncList, and is limited in length based on
the current socket connection state of the Connection.
PATH_ENTRY_DISCONNECTED_TX_QUEUE_LIMIT specifies the maximum number of messages
held by the queue when disconnected. If not specified by the user, this compiler symbol has a
default value of 128. PATH_ENTRY_CONNECTED_TX_QUEUE_LIMIT specifies the queue limit
when connected, and its default is 1024.

If PIM processing attempts to enqueue an outbound message when the outbound message
queue is at its capacity, the oldest message is discarded.

4.4 Interprocessor and Endianness
The inter-processor communication pattern is the same as the inter-process case with sockets
providing connectivity, but an additional concern for inter-processor communication is byte

17


ordering. In an effort to ensure consistency of communications and readability of all messages
between processors of both types of endian-ness, all messages are constructed with Network
Byte Ordering (big endian). The C++ industry standard approach for serialization with a
parent serializable class and a data factory for deserialization is implemented with the
PfdSerializable mechanism class. This perform the proper message construction and decoding
automatically for all PI-MDD modeled elements and built-in types.
4.4.1 Build-in Type Data Item Serialization
Incident parameter values are serialized and transmitted between Processes automatically
when the invocation of the target service/operation/event crosses Process boundaries.

4.4.1.1 Special Type Handling
To ensure complete encoding of a data item of a user defined type that is implemented as a 64
bit integer, the user defined type is marked with an ExternalType value of “long long” or
“unsigned long long”.

Real values are implemented with the double type, and are serialized with the IEEE Standard
for Floating-Point Arithmetic (IEEE 754).

An IncidentHandle passed as a parameter is serialized in its entirety – including all of its own
parameter values – and sent to the destination.

In the case of the built-in type Handle and user-defined pointer-based types derived from it,
only the pointer value will be transmitted. The expectation is Handle pointers are not be
dereferenced on anywhere except where they were created.

4.4.2 User-Defined Type Serialization – Advanced Realized Types
The user may define a class at the realized code level that can implement a model-level user-
defined type. If data items (parameters) of this type end up crossing process boundaries,
PathMATE serialization mechanisms may be applied to aid in this communication.

An Advanced Realized Type (ART) is created by inheriting from the PathMATE inheriting from
the PfdSerializable mechanism class from your realized class. The user then provides their
own class-specific implementations for the virtual serialization and deserialization methods
specified in PfdSerializable. In addition the model-level user-defined type is marked with
Serializeable = TRUE.

For C++ and Java, the underlying implementation class for the data type must inherit from the
PfdSerializeable class, and implement the virtual methods insertToBuffer and
extractFromBuffer. Network-safe serialization and de-serialization functions are provided in
msg_base.hcpp/hpp for all supported serializable scalar types.

In C, the following additional properties are required to specify the serialization functions:
- SerializeFunction=<name of serialize function>
- DeserializeFunction=<name of deserialize function>
These serialize and deserialize functions must match the following function pointer type
definitions defined in sw_msg_base.h:
typedef void (*sw_serialize_function_t)(message_buffer_t *msg_buf, int *msg_len,
void* target, bool_t is_ascii);

18


typedef void* (*sw_deserialize_function_t)(message_buffer_t *msg_buf, int
*msg_len, bool_t is_ascii);
Additional (existing) markings are also helpful in providing all the information needed to
conveniently apply these capabilities:
- ExternalType=<implementation type this user-defined type maps to>
- IncludeFile=<name of file defining above ExternalType>

4.5 Communication Errors
The Software Mechanisms Error Registrar is the central point for error handler callback
registration and error reporting. Errors are grouped, and error handlers are registered for
groups. If no user-defined error callback has been registered for a group and an error is
reported against the group, a default error handler is applied. In addition to pre-defined error
groups, the user can define their own error groups and error codes.

4.5.1 Multi-Task
The Error Registrar provides a process-wide point of error notification registration for
application-level domains (and realized code). Model-level callbacks can be registered for
notification of an error is received under a specified ErrorGroup.

4.5.2 Model-Level Interface – SW:RegisterForErrorGroup
The service SoftwareMechanisms:RegisterForErrorGroup(Integer error_group, IncidentHandle
error_handler) handles callback registration for any group of errors reported via
SoftwareMechanisms::ReportError() and from built-in mechanism-level error reporting. Errors
are grouped by type, and an error handlers is registered for a specific group. If no user-
defined error callback has been registered for a group and an error is reported against the
group, a default error handler is applied.

User-provided error handlers must conform to the ErrorHandle Incident Profile - publishing the
same parameters as the SoftwareMechanisms:DefaultErrorHandler service (Integer
error_group, Integer error_code).
In addition to pre-defined error groups, the user can define their own groups.
If a communication error happens the provided callback will loaded with values for
error_group and error_code, and then called. For interprocess communication errors, the
SW_ERROR_GROUP_COMMUNICATIONS is used. This group includes the following error
codes:
SW_ERROR_CODE_COMMUNICATIONS_ACCEPT_FAILED,
SW_ERROR_CODE_COMMUNICATIONS_ADDRESS_FAILURE,
SW_ERROR_CODE_COMMUNICATIONS_BIND_FAILED,
SW_ERROR_CODE_COMMUNICATIONS_CONNECT_FAILED,
SW_ERROR_CODE_COMMUNICATIONS_DISCONNECT,
SW_ERROR_CODE_COMMUNICATIONS_INCONSISTENT_CONNECTIONS,
SW_ERROR_CODE_COMMUNICATIONS_LISTEN_FAILED,
SW_ERROR_CODE_COMMUNICATIONS_NETWORK_INIT_FAILED,
SW_ERROR_CODE_COMMUNICATIONS_OTHER,
SW_ERROR_CODE_COMMUNICATIONS_RECEIVE_FAILED,
SW_ERROR_CODE_COMMUNICATIONS_SEND_FAILED,
SW_ERROR_CODE_COMMUNICATIONS_SOCKET_CREATE_FAILED,
SW_ERROR_CODE_COMMUNICATIONS_SOCKET_SHUTDOWN_FAILED,
SW_ERROR_CODE_COMMUNICATIONS_SOCKET_ERROR,
SW_ERROR_CODE_COMMUNICATIONS_TIMEOUT,
SW_ERROR_CODE_COMMUNICATIONS_CONNECT_MESSAGE_MISMATCH

19


See error_codes.hpp for the most complete list of pre-defined error codes.

4.6 Generated Topology Summary Report
Each time code is generated, TopologyReport.txt is generated into the _info subfolder of the
deployment folder. This is the system topology report emitting the ProcessType and TaskID of
all deployable elements. This report can be helpful for complex systems as a definitive
reference of the actual ProcessType and TaskID of each item, resolving potential ambiguities
with defaults and multiple properties files.

4.7 Trace Debugging
Compiler preprocessor definitions are used to control trace debugging – printing to the standard error stream
cerr.
 PATH_DEBUG_INTERPROCESS_HL – Enables trace debugging for summary-level interprocess
processing
 PATH_DEBUG_INTERPROCESS – Enables trace debugging for all levels of interprocess
processing (Automatically activates PATH_DEBUG_INTERPROCESS_HL).
 PATH_DEBUG_INTERPROCESS_MSG – Enables trace debugging for all interprocess message
sending and receipt
 PATH_DEBUG_TASK_TRACE – Enables trace debugging for intertask processing

4.8 Realized High-Speed Communications
In general each outbound socket message is created by taking an outbound incident and
serializing it into a PfdSockMessage, as identified in section 2.2.3 Multi Process. In some
specific cases a message is sent repeated times with very little or no changes to its contents
between transmissions. If this happens frequently or with very large incident parameter ART
payloads, the designer may discover the complete re-serialization of the incident is too slow to
meet message processing deadlines.

In these cases, it can be possible to avoid this repeated serialization overhead by writing
realized code that constructs a PfdSockMessage and sends it directly with PfdTopology::
enqueueOutboundRawMessage(). This realized code can create an instance of a
PfdSockMessageOut, send it, poll for send completion, then modify specific data values within
it, and resend this same instance all with relatively high efficiency. This process sidesteps the
safer and generally more convenient Incident-based approach, but also avoids that approach's
buffer allocation and full serialization processing.

Steps:
 Create a raw socket message PfdSockMessageOut instance, usually by manually
serializing an Incident in realized code
 Call deferDeletion() on the raw message to prevent it from being deleted after each
send.
 Save off a pointer to this raw message for repeated use
 Enqueue the raw message by calling PfdProcess::sendRawMessageInterProcess().
 Before updating specific data values in the raw message for a subsequent send, call
isComplete() on the message and ensure it returns TRUE
 Update specific data values in the raw message for the next send
 Send again with PfdProcess::sendRawMessageInterProcess().
 don't forget to put your toys away when play time is over: delete raw_message;

20


21


A. Thread Proliferation in Hand Written Systems
Software development for complex systems since the dawn of 3gls (FORTRAN, COBOL,
C, Pascal, Ada, C++, Java, etc) for high-performance systems is expressed nearly
universally as long synchronous sequences of subroutine/procedure/function/method
invocations. This serialization of processing is more a product of the single-
dimensionality of these programming languages than any inherent simplicity in the
problem spaces being addressed.
The advent of multitasking and multiprocessor systems has driven 3gl programmers to
break single-dimensional synchronous processing into separate sequences – threads.
This decomposition has met varying levels of success, and introduced specific system
partitioning techniques trying to take advantage of “apparent” parallelism offered by
OS-level tasks. The subsequent rise in available of multi-core systems (and their
“actual” parallelism) has further spurred the this move to use multi-threading.
As development organizations gain proficiency with OS-level multi threading and with
the absence of any other way to achieve parallelism (the fundamentally synchronous
nature of programming languages hasn’t changed) typical complex system architectures
have experienced a sharp rise in the number of OS-level threads employed. With this
rise comes the computational costs of these complex OS mechanisms, and the
development and maintenance costs of more complex designs. In many cases an
overabundance of threads has resulted in systems with lower performance even with
more capable processors and memory.
PI-MDD offers an alternative to some of this harmful thread proliferation.

PI-MDD, Asynchronous Domain Interactions, and The Event-Driven
Paradigm
The use of PI-MDD models breaks some of the constraints of synchronous sequences of
subroutine/procedure/function/method invocations. Domain Modeling pushes system
design from the very beginning to a base of primarily asynchronous interactions. The
direct support of the callback in the UML Action Language via the IncidentHandle, and
with state machines available within domains, there are a wide selection of
asynchronous elements to augment the basic synchronous function/method call. With
PathMATE automation the model-level function call (to a domain service or class
operation) can also be generated with an implementation that uses an asynchronous
IncidentHandle as needed. These asynchronous primitives break the simplistic lock that
synchronous programming languages put on basic behavioral expression.
The result of these asynchronous forms of expression is a PI-MDD system that has a
large number of simple entities operating in asynchronous independence from each
other, even when deployed to the same OS thread of control. OS threads are no longer
needed to work around blocking calls or to interweave long chains of operations.
Augmenting this fundamentally asynchronous form of behavioral expression, PI-MDD
components can also be deployed to separate OS threads as needed - when blocking
(or “nearly blocking” – long latency) calls to realized code function must be made.
Typically these are found with communication sends and receives, files operations, or
calls into legacy code still organized in long, synchronous call chains. Additional tasks

22


are also applied to manage priority so a few, higher priority elements are allocated to a
high priority task separate from the main task with ordinary priority.
The net result of this is the allocation of much larger fragments of processing to fewer
OS threads, while still realizing greater interleaving of processing through
fundamentally asynchronous behavioral expression. The need for inter-task
synchronization mechanisms is greatly reduced, and therefore run-time overhead and
general complexity is reduced.

23


B. Marking Summary
marking name applies to default value description

Domain, Service, Class,
TaskID Operation SYS_TASK_ID_MAIN Allocate action processing to a Task

ProcessType Domain, Service <DefaultProcessType> Allocate action processing to a Process Type

Indicate this DestinationHandle specifies
Routing Parameter <none> destination.

"T" generates a PfdSyncList for the
ThreadSafe Class, Association F instance container

Specifies stack size for all Tasks; 0 indicates
DefaultStackSize System 0 use OS default

Allows the default ProcessType name to be
DefaultProcessType System MAIN changed

Domain, Service, Class, Sets the priority of the task for this analysis
TaskPriority Operation SYS_TASK_PRIORITY_NORMAL element; must be a pfdos_priority_e literal

Allows specification of compiler symbol
Defines System, Domain <none> values in the markings

ExternalType User Defined Type void* Implementation type for an ART

Include file for the implementation type for an
IncludeFile User Defined Type <none> ART

TRUE indicates this is an ART, inheriting
Serializable User Defined Type FALSE from PfdSerializable

24


C. Compiler Symbol Summary
symbol name default value description
PFD_MAIN_OOA_THREAD_PRIORITY SYS_TASK_PRIORITY_NORMAL Default task priority for each OOA processing task
PFD_RECEIVER_THREAD_PRIORITY SYS_TASK_PRIORITY_NORMAL Default task priority for the TCP receiver task
PFD_TRANSMIT_THREAD_PRIORITY SYS_TASK_PRIORITY_NORMAL Default task priority for each TCP sender task
Default task priority for the Spotlight connection
PFD_IE_THREAD_PRIORITY SYS_TASK_PRIORITY_LOWEST task
PFD_INPUT_THREAD_PRIORITY SYS_TASK_PRIORITY_NORMAL Default task priority for the Driver input task
Define this to cause a stack to be allocated
PATH_ALLOCATE_EXPLICIT_STACK <not defined> explicitly, separate from the start task OS call
Time between sender task attempts to reconnect
PATH_DELAY_BETWEEN_RECONNECT_ATTEMPTS_MS 100 with destination process, in milliseconds
Max number of times the sender task attempts to
reconnect with destination process without waiting
PATH_IMMEDIATE_RECONNECT_MAX_COUNT 5 between
Time between interprocess message packat sends
(socket send calls), in milliseconds; 0 means
PATH_DELAY_BETWEEN_MESSAGE_PACKET_SENDS_MS 0 pfd_sleep(0); -1 means no sleep at all
Max number of pending outbound messages
queues for a single destination while not
PATH_ENTRY_DISCONNECTED_TX_QUEUE_LIMIT 128 connected to that destination
Max number of pending outbound messages
queues for a single destination while connected to
PATH_ENTRY_CONNECTED_TX_QUEUE_LIMIT 1024 that destination
Define this to turn on high level "printf" debugging
PATH_DEBUG_INTERPROCESS_HL <not defined> of interprocess mechanisms
Define this to turn on full "printf" debugging of
PATH_DEBUG_INTERPROCESS <not defined> interprocess mechanisms
Define this to turn on "printf" debugging of
PATH_DEBUG_INTERPROCESS_MSG <not defined> interprocess message sending
Define this to turn on "printf" debugging of PfdTask
PATH_DEBUG_TASK_TRACE <not defined> mechanics

25

Distributed Deployment Model Driven Development

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (8)

Destaque

Destaque (7)

Semelhante a Distributed Deployment Model Driven Development

Semelhante a Distributed Deployment Model Driven Development (20)

Último

Último (20)

Distributed Deployment Model Driven Development