2. Introduction
• The architecture is used to communicate
data among a set of processors without the
need for a global memory.
• Each PE has its own local memory and
communicates with other PEs using
message.
3. Message Passing Systems
104 MESSAGE PASSING ARCHITECTURE
P1 M1 P2 M2 Pn Mn
Link 1 Link 2
Link n
Interconnection Network
Figure 5.1 Message passing systems.
networks have received considerable attention over the years. As shown in Chapter 2,
two important factors must be considered in designing message passing interconnec-
tion networks: link bandwidth and the network latency. The link bandwidth is defined
as the number of bits that can be transmitted per unit of time (bits/s). Network latency
is defined as the time to complete a message transfer through the network.
4. Message Passing Systems
104 MESSAGE PASSING ARCHITECTURE
P1 M1 P2 M2 Pn Mn
Link 1 Link 2
Link n
Interconnection Network
Figure 5.1 Message passing systems.
networks have received considerable attention over the years. As shown in Chapter 2,
two important factors must be considered in designing message passing interconnec-
external channel
tion networks: link bandwidth and the network latency. The link bandwidth is defined
as the number of bits that can be transmitted per unit of time (bits/s). Network latency
is defined as the time to complete a message transfer through the network.
5. MP Network
• Normally, a static-type network
• hypercubes
• nearest-neighbor 2D & 3D mesh
• Two important factors must be considered:
• link bandwidth - the number of bits that can
be transmitted per unit of times (bits/s)
• network latency - the time to complete a
message transfer through the network
6. Executing Program
• The program is divided into concurrent
processes; each is executed on a separate
processor.
• If the no. of processes is larger than the
number of processors, then more than one
process will have to be executed on a
processor in a time-shared fashion.
104 MESSAGE PASSING ARCHITECTURE
P1 M1 P2 M2 Pn Mn
Link 1 Link 2
Link n
Interconnection Network
Figure 5.1 Message passing systems.
networks have received considerable attention over the years. As shown in Chapter 2,
two important factors must be considered in designing message passing interconnec-
tion networks: link bandwidth and the network latency. The link bandwidth is defined
as the number of bits that can be transmitted per unit of time (bits/s). Network latency
7. Process Communication
• Processes running on a given processor
use what is called internal channels to
exchange messages among themselves.
• Processes running on different processors
use the external channels to exchange
messages.
8. Data Exchanged
• Data exchanged among processors cannot
be shared; it is rather copied (using send/
receive messages).
• An important advantage of this form of
data exchange is the elimination of the
need for synchronization constructs, such
as semaphores, which results in
performance improvement.
9. More advantages
• A message passing scheme offers flexibility
in accommodating a large number of
processors in addition to being readily
scalable.
• A given node can execute more than one
process, each at a given time.
10. 5.2 ROUTING IN MESSAGE PASSING NETWORKS 105
Outside World
P1
m1
P2
m4
P3
m2 m3
P4
Figure 5.2 An example of a message passing system.
Three types of granularity can be distinguished. These are:
1. Coarse granularity: Each process holds a large number of sequential instruc-
11. Process Granularity
• It’s a parameter which describes the size of
a process in a message passing system.
computation time
Process Granularity =
communication time
12. Granularity Types
• There are 3 types of granularity:
• Coarse - each process holds a large number
of sequential instructions and takes a
substantial amount of time to execute.
• Medium - a middle ground where
communication overhead is reduced
• Fine - each process contains a few sequential
instructions
• Use mostly medium or coarse granularity
13. Routing in MP Networks
• Involves the identification of a set of
permissible paths that may be used by a
message to reach its destination, and a
function (η) that selects one path from the
set of permissible paths.
14. Routing Techniques
• Adaptive
• the path taken by the message depends on
network conditions for a given source and
destination pair
• Deterministic (oblivious)
• determines the path using only the source
and destination regardless of the network
conditions - simple but bandwidth inefficient
15. Routing for Broadcasting
& Multicasting
• Communication operations: unicast and
collective
• Unicast - communicate to only a single
destination
• Collective - several routing operations
are defined, broadcast & multicast are the
most widely used.
16. Broadcast Multicast
one-to-all operation one-to-many operation
distribute data during
uses in large-scale
computation of a
multiprocessors, inc.
distributed memory
parallel search algorithm
program
routing algorithms must be deadlock-free
little time and short path is desirable
17. of these resources in a strict monotonic order. This restricted way for using network
resources prevents the occurrence of circular wait, and hence prevents the occur-
rence of deadlock. The channel dependency graph (CDG) is a technique used
to develop a deadlock-free routing algorithm. A CDG is a directed graph
D ¼ G(C, E ), where the vertex set C consists of all the unidirectional channels in
the network and the set of edges E includes all the pairs of connected channels,
000
110 111 001 010 100
011 101 110 011 101 110
100 101
111 010 100 111 100 111 001
010 011 001
110 101 011 010
000 001
Figure 5.3 Hypercube broadcast tree-based communication.
18. Routing Potential Problems
• Deadlock
• When two messages each hold the resources
required by the other in order to move, both
message will be blocked.
• Livelock
• A situation in which a message keeps going around
the network and never reaches its destination.
• Starvation
• A node is said to suffer from starvation if it has a
message to inject into the network but is never
allowed to do so.
19. Message Passing vs
Shared Memory
• Shared memory systems may be easier to
program, but the difficult to scale up to a large
number of processors.
• If scalability to larger and larger systems was to
continue, systems had to use message passing
techniques.
• However, a number of problems associated with
message passing systems: communication
overhead and difficulty of programming.
