In this deckfrom the 2017 DDN User Group meeting at ISC, Judit Planas, Postdoctoral Researcher at Ecole Polytechnique Federale de Lausanne (EPFL) presents I/O Challenges in Brain Tissue Simulation (IME Neuromapp). Learn more: https://insidehpc.com/2017/08/video-io-challenges-brain-tissue-simulation/ and http://ddn.com Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. Judit Planas | DDN User Group | ISC 2017 49
I/O Challenges in
Brain Tissue Simulation
Judit Planas | DDN User Group | ISC 2017

2. Judit Planas | DDN User Group | ISC 2017 50
Outline
 BBP introduction
 I/O challenges in brain simulation
 Why?
 Examples
 Case study: ReportingLib mini-app
 Description
 Evaluation
 Conclusions & future work
Outline

3. Judit Planas | DDN User Group | ISC 2017 51
The Blue Brain Project
BBP Introduction
 The Blue Brain Project is a Swiss initiative that targets the digital reconstruction and
simulation of the brain, hosted in Geneva (Switzerland)
 BBP is a multi-disciplinary team that brings together people from a wide variety of
backgrounds, like neuroscientists, computer engineers, physicists, mathematicians or
chemists

4. Judit Planas | DDN User Group | ISC 2017 52
BBP’s Value Proposition for Neuroscience
BBP BiologicalValue
 Help scientists understand how the brain works internally
 Recently, BBP scientists have been able to reproduce the electrical behavior
of a neocortex fragment by means of a computer reconstruction [1]
 Brain volume: 1/3 mm3
 30’000 neurons
 40 million synapses (connections between neurons)
 This model has revealed novel insights into the functioning of the neocortex
 Supercomputer-based simulation of the brain provides a new tool to study the interaction
within the different brain regions
 Understanding the brain not only will help the diagnosis and treatment of brain diseases, but
also will contribute to the development of neurorobotics and neuromorphic computing
[1] Henry Markram et al. Reconstruction and Simulation of Neocortical Microcircuitry. Cell, Vol. 163, Issue 2, pp 456 - 492

5. Judit Planas | DDN User Group | ISC 2017 53
I/O Challenges in Brain Simulation
I/O Challenges Introduction
Visualization
Scripting Analysis
Visualization
Scripting
Analysis
 Simulating the brain is a complex workflow where many tools are involved, depending on the
use case:
 Producers: Neuron/CoreNeuron are the common brain simulators used daily at BBP to generate
simulation reports (results)
 Consumers:A large number of different applications are used to analyze simulation results, ranging
from sophisticated visualization frameworks to custom scripts that extract different characteristics
of the results
 Each tool accesses data in a particular way, making it difficult to find an efficient layout to fit
all use cases
Configuration Simulation Analysis

6. Judit Planas | DDN User Group | ISC 2017 54
Data Layout
Producing Results
 ReportingLib is the tool in charge of writing simulation reports
 Reports are written in binary format using MPI I/O collective calls
 In-house data layout, organized per time step, then per GID (neuron ID)
 The process of writing the reports can represent up to 30% of total execution time
GID 1 GID 2 GID 3 GID 4 GID 5 GID 6 GID 7 …
Time step 0
Time step 1
Time step 2
...
I/O Challenges Introduction

7. Judit Planas | DDN User Group | ISC 2017 55
Consuming Results: Use Case 1
 Current data layout fits some
of the use cases
 Example: simulation
visualization with in-vitro Ca
concentration:
 Read pattern: all GIDs, by
timeframe (row-wise)
 > 200’000 neurons
simulated, 5% reported
I/O Challenges Examples

8. Judit Planas | DDN User Group | ISC 2017 56
Consuming Results: Use Case 2
 But current data layout is not
optimal for other use cases
 Example: voltage series of
selected neurons of different
morphology types (source [1]):
 Read pattern: for the selected
GIDs, read all time steps
(column-wise)
I/O Challenges Examples

9. Judit Planas | DDN User Group | ISC 2017 57
 Neuromapp is a mini-app framework that mimics the key parts of our scientific workflows:
 Computation kernels (CPU-intensive)
 Spike exchange (network communication)
 Writing simulation reports (I/O-intensive)
 Reproduces exact behavior of original production code
 Easy to run on different platforms, from small clusters to supercomputers
 Software/hardware technologies evolve fast, mini-apps are easy-to-use prototypes
 Simple code, easy to share and understand by external collaborators
Neuromapp: Mini-App Framework
ReportingLib Mini-App Description
✳ The work developed with Neuromapp has been published this year at ISC:
[2]Timothée Ewart et al. Neuromapp: a Mini-Application Framework to Improve Neural Simulators. ISC 2017.

10. Judit Planas | DDN User Group | ISC 2017 58
 Motivation:The current production code is too complex to explore I/O alternatives
 I/O benchmarks are useful to measure hardware capabilities, but the throughput observed by
specific applications may differ
 Specially when using unpredictable or uncommon access patterns
ReportingLib Mini-App
ReportingLib Mini-App Description
MPI collective write
Initialize random output data
Repeat for each
simulation step
Sleep to simulate computing phase
Repeat for each
I/O phase
Overwrite output data

11. Judit Planas | DDN User Group | ISC 2017 59
DDN IME Data Flow
IME server IME server IME server IME server IME server
Application
IME Client
Parallel Filesystem
1. Application issues
fragmented,
misaligned IO
2. IME clients send
fragments to IME
servers
3. Fragments to IME
servers accessible via
DHT to all clients
4. Fragments to be
flushed from IME are
assembled into PFS
stripes
5. PFS receives
complete PFS stripe
ApplicationApplicationApplication
IME ClientIME ClientIME Client
ReportingLib Mini-App Evaluation
✳ Source: DDN IME presentation by Steve Crusan, Copyright DDN Inc.

12. Judit Planas | DDN User Group | ISC 2017 60
ReportingLib Mini-App: Experimental Results
0
20000
40000
60000
80000
100000
120000
140000
1 2 4 8 1 2 4 8 1 2 4 8 1 2 4 8
1 2 4 8
AggregatedBW[MB/s]
#Ranks/node (top)
#Nodes (bottom)
DDN IME System
100 KB - nrsci 100 KB - bench 650 KB - nrsci 650 KB - bench 1024 KB - nrsci 1024 KB - bench
 Final file size: 100 GB
 Block size: 100 KB, 650 KB, 1 MB
 > 100 reporting steps
 2 styles: ReportingLib, IOR-like
 Client node configuration:
 Intel Xeon E5-1620 (8 cores)
 64 GB DRAM
 100 Gb/s Infiniband
 Lustre file system
 IME system (4 servers):
 2 x Intel Xeon E5-2650 (16 cores)
 64 GB DRAM
 12 x Intel SSDs, 256 GB
 Using MPI I/O interface
ReportingLib Mini-App Evaluation
0
20000
40000
60000
80000
100000
120000
140000
1 2 4 8 1 2 4 8 1 2 4 8 1 2 4 8
1 2 4 8
AggregatedBW[MB/s]
#Ranks/node (top)
#Nodes (bottom)
DDN IME System (CSCS)
100 KB - nrsci 650 KB - nrsci 1024 KB - nrsci
100 KB - bench 650 KB - bench 1024 KB - bench

13. Judit Planas | DDN User Group | ISC 2017 61
IME nrsci IME bench Lustre nrsci Lustre bench
0
100000
200000
300000
400000
500000
600000
1 2 4 8 16 24 32 42 84
AggregatedBW[MB/s]
#Nodes (8 ranks/node)
A-Star System
ReportingLib Mini-App: Scaling Results
 Final file size: 42 GB
 Block size: 650 KB
 100 reporting steps
 2 styles: ReportingLib, IOR-like
 Client node configuration:
 2 x Intel Xeon E5-2690 (24 cores)
 128 GB DRAM
 Infiniband EDR
 Lustre file system
 IME system:
 2 x Intel Xeon E5-2695 (28 cores)
 256 GB DRAM
 24 x NVMe SSDs, 800 GB
 Using MPI I/O interface
ReportingLib Mini-App Evaluation
0
100
200
300
400
500
600
700
1 2 4 8 16 24 32 42 84
AggregatedBW[MB/s]
#Nodes (8 ranks/node)
✳ Performance numbers provided by DDN (J.T. Acquaviva)

14. Judit Planas | DDN User Group | ISC 2017 62
Application
System/HW
ReportingLib Mini-App: Discussion
 Results prove the need of having representative I/O load:
 Bandwidth measured by IOR-like benchmark differs from what our mini-app observes
 Why the mini-app’s reported bandwidth is higher than theoretical peak?
 Inherent properties of our use case: alternating computation and I/O
 IME capability of caching & overlapping: using computation time to flush data to disk
 IME advantages:
 IME runs out of the box for us (MPI I/O support)
 IME perf is faster than regular MPI I/O (MVAPICH2 v2.2b)
ReportingLib Mini-App Evaluation
Configuration Max. BW FS Access
MPI I/O IOR-like
(bench)
1.6 GB/s Lustre, direct access
IME IOR-like (bench) 8.8 GB/s Lustre, IME servers
IME RepLib (nrsci) 128 GB/s Lustre, IME servers
Sim I/O Sim I/O Sim
I/O
…
I/O …
t

15. Judit Planas | DDN User Group | ISC 2017 63
Conclusions
 With the increasing computing power over the last years, I/O has been stressed and
represents now one of the major bottlenecks
 Brain simulation exposes a number of challenges for I/O and storage systems
 Neuromapp is our prototyping framework:
 It represents our common use cases: from computation through network comm. to I/O
 Easy to use and share to evaluate new software/hardware technologies
 DDN IME fills the gap between fast computing power and not-so-fast system I/O
 Easy to adopt for our use case (direct MPI I/O support)
 Increased bandwidth from the application point of view
 Further exploration:
 Scale to larger DDN IME system
 Evaluate read performance following real use case read patterns
Conclusions

16. Judit Planas | DDN User Group | ISC 2017 64
Acknowledgements
 We would like to thank the collaboration of the following teams:
 The BBP HPC team for all the discussions, help and feedback received
 DDN IME team for all the support provided
 CSCS system administrators to facilitate the access to their systems
Acknowledgements