In this deck from ATPESC 2019, Ken Raffenetti from Argonne presents an overview of HPC interconnects. "The Argonne Training Program on Extreme-Scale Computing (ATPESC) provides intensive, two-week training on the key skills, approaches, and tools to design, implement, and execute computational science and engineering applications on current high-end computing systems and the leadership-class computing systems of the future." Watch the video: https://wp.me/p3RLHQ-luc Learn more: https://extremecomputingtraining.anl.gov/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. Interconnects
Ken Raffenetti
Software Development Specialist
Programming Models and Runtime Systems Group
Mathematics and Computer Science Division
Argonne National Laboratory

2. U.S. DOE Potential System Architecture Targets
ATPESC (07/29/2019)
System attributes 2010 2018-2019 2021-2022
System peak 2 Peta 150-200 Petaflop/sec 1 Exaflop/sec
Power 6 MW 15 MW 20 MW
System memory 0.3 PB 5 PB 32-64 PB
Node performance 125 GF 3 TF 30 TF 10 TF 100 TF
Node memory BW 25 GB/s 0.1TB/sec 1 TB/sec 0.4TB/sec 4 TB/sec
Node concurrency 12 O(100) O(1,000) O(1,000) O(10,000)
System size (nodes) 18,700 50,000 5,000 100,000 10,000
Total Node
Interconnect BW
1.5 GB/s 20 GB/sec 200GB/sec
MTTI days O(1day) O(1 day)
Past
production
Current generation
(e.g., CORAL)
Exascale
Goals
[Includes modifications to the DOE Exascale report]

3. General Trends in System Architecture
§ Number of nodes is increasing, but at a moderate pace
§ Number of cores/threads on a node is increasing rapidly
§ Each core is not increasing in speed (clock frequency)
§ What does this mean for networks?
– More cores will drive the network
– More sharing of the network infrastructure
– The aggregate amount of communication from each node will
increase moderately, but will be divided into many smaller messages
– A single core may not be able fully saturate the NIC
ATPESC (07/29/2019)

4. A Simplified Network Architecture
§ Hardware components
– Processing cores and
memory subsystem
– I/O bus or links
– Network
adapters/switches
§ Software components
– Communication stack
§ Balanced approach
required to maximize
user-perceived network
performance
ATPESC (07/29/2019)
P0
Core0 Core1
Core2 Core3
P1
Core0 Core1
Core2 Core3
Memory
MemoryI
/
O
B
u
s
Network Adapter
Network
Switch
Processing
Bottlenecks
I/O Interface
Bottlenecks
Network
End-host
Bottlenecks
Network
Fabric
Bottlenecks

5. Agenda
ATPESC (07/29/2019)
Network Adapters
Network Topologies
Network/Processor/Memory
Interactions

6. Bottlenecks on Traditional Network Adapters
§ Network speeds plateaued at
around 1Gbps
– Features provided were limited
– Commodity networks were not
considered scalable enough for
very large-scale systems
ATPESC (07/29/2019)
P0
Core0 Core1
Core2 Core3
P1
Core0 Core1
Core2 Core3
Memory
MemoryI
/
O
B
u
s
Network Adapter
Network
Switch
Network
Bottlenecks
Ethernet (1979 - ) 10 Mbit/sec
Fast Ethernet (1993 -) 100 Mbit/sec
Gigabit Ethernet (1995 -) 1000 Mbit /sec
ATM (1995 -) 155/622/1024 Mbit/sec
Myrinet (1993 -) 1 Gbit/sec
Fibre Channel (1994 -) 1 Gbit/sec

7. End-host Network Interface Speeds
§ HPC network technologies provide high bandwidth links
– InfiniBand EDR gives 100 Gbps per network link
• Will continue to increase (HDR 200 Gbps, etc.)
– Multiple network links becoming a common place
• ORNL Summit and LLNL Sierra machines, Japanese Post T2K machine
• Torus style or other multi-dimensional networks
§ End-host peak network bandwidth is “mostly” no longer
considered a major limitation
§ Network latency still an issue
– That’s a harder problem to solve – limited by physics, not technology
• There is some room to improve it in current technology (trimming the fat)
• Significant effort in making systems denser so as to reduce network latency
§ Other important metrics: message rate, congestion, …
ATPESC (07/29/2019)

8. Simple Network Architecture (past systems)
§ Processor, memory,
network are all
decoupled
ATPESC (07/29/2019)
P0
Core0 Core1
Core2 Core3
P1
Core0 Core1
Core2 Core3
Memory
MemoryI
/
O
B
u
s
Network Adapter
Network
Switch

9. Integrated Memory Controllers (current systems)
§ In the past 10 years or so, memory
controllers have been integrated on
to the processor
§ Primary purpose was scalable
memory bandwidth (NUMA)
§ Also helps network communication
– Data transfer to/from network requires
coordination with caches
§ Several network I/O technologies
exist
– PCIe, HTX, NVLink
– Expected to provide higher bandwidth
than what network links will have
ATPESC (07/29/2019)
P1
Core0 Core1
Core2 Core3
Memory
Memory
I
/
O
B
u
s
Network Adapter
Network
Switch
Memory
Controller
P0
Core0 Core1
Core2 Core3
Memory
Controller

10. Integrated Networks (future systems?)
ATPESC (07/29/2019)
Network
Switch
P0
Core0 Core1
Core2 Core3
Memory
Controller
Network
Interface
Off-chip Memory
§ Several vendors are considering
processor-integrated network
adapters
§ May improve network bandwidth
– Unclear if the I/O bus would be a
bottleneck
§ Improves network latencies
– Control messages between the
processor, network, and memory are
now on-chip
§ Improves network functionality
– Communication is a first-class citizen
and better integrated with processor
features
– E.g., network atomic operations can be
atomic with respect to processor
atomics
In-package
Memory
P1
Core0 Core1
Core2 Core3
Memory
Controller
Network
Interface
Off-chip Memory
In-package
Memory

11. Processing Bottlenecks in Traditional Protocols
§ Ex: TCP/IP, UDP/IP
§ Generic architecture for all networks
§ Host processor handles almost all
aspects of communication
– Data buffering (copies on sender and
receiver)
– Data integrity (checksum)
– Routing aspects (IP routing)
§ Signaling between different layers
– Hardware interrupt on packet arrival or
transmission
– Software signals between different
layers to handle protocol processing in
different priority levels
ATPESC (07/29/2019)
P0
Core0 Core1
Core2 Core3
P1
Core0 Core1
Core2 Core3
Memory
MemoryI
/
O
B
u
s
Network Adapter
Network
Switch
Processing
Bottlenecks

12. Network Protocol Stacks: The Offload Era
§ Modern networks are spending more and more network real-estate on
offloading various communication features on hardware
§ Network and transport layers are hardware offloaded for most modern
HPC networks
– Reliability (retransmissions, CRC checks), packetization
– OS-based memory registration, and user-level data transmission
ATPESC (07/29/2019)

13. Comparing Offloaded Network Stacks with
Traditional Network Stacks
ATPESC (07/29/2019)
Application Layer
MPI, PGAS, File Systems
Transport Layer
Low-level interface
Reliable/unreliable protocols
Link Layer
Flow-control, Error Detection
Physical Layer
Hardware Offload
Copper or Optical
HTTP, FTP, MPI,
File Systems
Routing
Physical Layer
Link Layer
Network Layer
Transport Layer
Application Layer
Traditional Ethernet
Sockets Interface
TCP, UDP
Flow-control and
Error Detection
Copper, Optical or Wireless
Network Layer Routing
Management tools
DNS management tools

14. Current State for Network APIs
§ A large number of network vendor specific APIs
– InfiniBand verbs, Intel PSM2, IBM PAMI, Cray Gemini/DMAPP, …
§ Recent efforts to standardize these low-level communication
APIs
– Open Fabrics Interface (OFI)
• Effort from Intel, CISCO, etc., to provide a unified low-level communication
layer that exposes features provided by each network
– Unified Communication X (UCX)
• Effort from Mellanox, IBM, ORNL, etc., to provide a unified low-level
communication layer that allows for efficient MPI and PGAS communication
– Portals 4
• Effort from Sandia National Laboratory to provide a network hardware
capability centric API
ATPESC (07/29/2019)

15. User-level Communication: Memory Registration
ATPESC (07/29/2019)
1. Registration Request
• Send virtual address and length
2. Kernel handles virtual->physical
mapping and pins region into
physical memory
• Process cannot map memory
that it does not own (security !)
3. Network adapter caches the
virtual to physical mapping and
issues a handle
4. Handle is returned to application
Before we do any communication:
All memory used for communication must
be registered
1
3
4
Process
Kernel
Network
2

16. User-level Communication: OS Bypass
ATPESC (07/29/2019)
Process
Kernel
Network
User-level APIs allow direct interaction with
network adapters § Contrast with traditional network
APIs that trap down to the kernel
§ Eliminates heavyweight context
switch
§ Memory registration caches allow
for fast buffer re-use, further
reducing dependence on the
kernel

17. Send/Receive Communication
ATPESC (07/29/2019)
Network Interface
Memory Memory
Network Interface
Send Recv
Memory
Segment
Send entry contains information about the
send buffer (multiple non-contiguous
segments)
Processor Processor
Send Recv
Memory
Segment
Receive entry contains information on the receive
buffer (multiple non-contiguous segments);
Incoming messages have to be matched to a
receive entry to know where to place
Hardware ACK
Memory
Segment
Memory
Segment
Memory
Segment

18. PUT/GET Communication
ATPESC (07/29/2019)
Network Interface
Memory Memory
Network Interface
Send Recv
Memory
Segment
Send entry contains information about the
send buffer (multiple segments) and the
receive buffer (single segment)
Processor Processor
Send Recv
Memory
Segment
Hardware ACK
Memory
Segment
Memory
Segment

19. Atomic Operations
ATPESC (07/29/2019)
Network Interface
Memory Memory
Network Interface
Send Recv
Memory
Segment
Send entry contains information about the
send buffer and the receive buffer
Processor Processor
Send Recv
Source
Memory
Segment
OP
Destination
Memory
Segment

20. Network Protocol Stacks: Specialization
§ Increasing network specialization is the focus today
– The next generation of networks plan to have further support for
noncontiguous data movement, and multiple contexts for multithreaded
architectures
§ Some networks, such as the Tofu network, Cray Aries and InfiniBand, are
also offloading some MPI and PGAS features on to hardware
– E.g., PUT/GET communication has hardware support
– Increasing number of atomic operations being offloaded to hardware
• Compare-and-swap, fetch-and-add, swap
– Collective operations (NIC and switch support)
• SHARP collectives
– Hardware tag matching for MPI send/recv
• Cray Seastar, Bull BXI, Mellanox Infiniband (ConnectX-5 and later)
ATPESC (07/29/2019)

21. Agenda
ATPESC (07/29/2019)
Network Adapters
Network Topologies
Network/Processor/Memory
Interactions

22. Traditional Network Topologies: Crossbar
§ A network topology describes how different network
adapters and switches are interconnected with each other
§ The ideal network topology (for performance) is a crossbar
– Alltoall connection between network adapters
– Typically done on a single network ASIC
– Current network crossbar ASICs go up to 64 ports; too expensive to
scale to higher port counts
– All communication is nonblocking
ATPESC (07/29/2019)

23. Traditional Network Topologies: Fat-tree
§ The most common topology for small and medium scale
systems is a fat-tree
– Nonblocking fat-tree switches available in abundance
• Allows for pseudo nonblocking communication
• Between all pairs of processes, there exists a completely nonblocking
path, but not all paths are nonblocking
– More scalable than crossbars, but the number of network links still
increases super-linearly with node count
• Can get very expensive with scale
ATPESC (07/29/2019)

24. Network Topology Trends
§ Modern topologies are moving towards
more “scalability” (with respect to cost, not
performance)
§ Blue Gene, Cray XE/XK, and K
supercomputers use a torus-network; Cray
XC uses dragonfly
– Linear increase in the number of
links/routers with system size
– Any communication that is more than one
hop away has a possibility of interference –
congestion is not just possible, but common
– Even when there is no congestion, such
topologies increase the network diameter
causing performance loss
§ Take-away: topological locality is important
and its not going to get better
ATPESC (07/29/2019)

25. Network Congestion Behavior: IBM BG/P
0
500
1000
1500
2000
2500
3000
3500
1
2
4
8
16
32
64
128
256
512
1K
2K
4K
8K
16K
32K
64K
128K
256K
512K
1M
Bandwidth(Mbps)
Message Size (bytes)
P2-P5
P3-P4
No overlap
ATPESC (07/29/2019)
P0 P1 P2 P3 P4 P5 P6 P7

26. 2D Nearest Neighbor: Process Mapping (XYZ)
ATPESC (07/29/2019)
X-Axis
Z-Axis
Y-Axis

27. Nearest Neighbor Performance: IBM BG/P
ATPESC (07/29/2019)
0
100
200
300
400
500
600
700
800
900
2 4 8 16 32 64 128 256 512 1K
ExecutionTime(us)
Grid Partition (bytes)
System Size : 16K Cores
XYZT
TXYZ
ZYXT
TZYX
0
500
1000
1500
2000
2500
2 4 8 16 32 64 128 256 512 1K
ExecutionTime(us)
Grid Partition (bytes)
System Size : 128K Cores
XYZT
TXYZ
ZYXT
TZYX
2D Halo Exchange

28. Agenda
ATPESC (07/29/2019)
Network Adapters
Network Topologies
Network/Processor/Memory
Interactions

29. Network Interactions with Memory/Cache
§ Most network interfaces understand and work with the cache
coherence protocols available on modern systems
– Users do not have to ensure that data is flushed from cache before
communication
– Network and memory controller hardware understand what state the
data is in and communicate appropriately
ATPESC (07/29/2019)

30. Send-side Network Communication
ATPESC (07/29/2019)
North BridgeCPU
NIC
FSB Memory Bus
I/OBus
Appln.
Buffer
L3 $ Memory
Appln.
Buffer

31. Receive-side Network Communication
ATPESC (07/29/2019)
North BridgeCPU
NIC
FSB Memory Bus
I/OBus
Appln.
Buffer
L3 $ Memory
Appln.
Buffer

32. Network/Processor Interoperation Trends
§ Direct cache injection
– Most current networks inject data into memory
• If data is in cache, they flush cache and then inject to memory
– Some networks are investigating direct cache injection
• Data can be injected directly into the last-level cache
• Can be tricky since it can cause cache pollution if the incoming data is not
used immediately
§ Atomic operations
– Current network atomic operations are only atomic with respect to
other network operations and not with respect to processor atomics
• E.g., network fetch-and-add and processor fetch-and-add might corrupt
each other’s data
– With network/processor integration, this is expected to be fixed
ATPESC (07/29/2019)

33. Network Interactions with Accelerators
§ PCI Express peer-to-peer capabilities enables network
adapters to directly access third-party devices
– Coordination between network adapter and accelerator (GPUs,
FPGAs, …)
– Data does not need to be copied into to/from buffers when going over
the network
– GPUDirect RDMA for NVIDIA GPUs is one example
ATPESC (07/29/2019)

34. Summary
§ These are interesting times for all components in the overall
system architecture: processor, memory, interconnect
– And interesting times for computational science on these systems
§ Interconnect technology is rapidly advancing
– More hardware integration is the key to removing bottlenecks and
improve functionality
• Processor/memory/network integration is already in progress and will
continue for the foreseeable future
– Offload technologies continue to evolve as we move more
functionality to the network hardware
– Network topologies are becoming more “shared” (cost saving)
ATPESC (07/29/2019)

35. Thank You!
Email: raffenet@mcs.anl.gov