Introduction to multi-language implementation for HPC

1. Harmonic Stack for
Speed
2018/3/6
Yung-Yu Chen
https://www.linkedin.com/in/yungyuc/

2. The World Is Changing
❖ Computing helps human beings to change the world in
an unprecedented pace
❖ What are these?
30 cm in diameter, 1978
3 sizes, 3.5” smallest

3. Python as Your Computing Platform
❖ Python programming language is the way to go
❖ One language to rule them all
❖ Interactive analysis and visualization
❖ Data preparation and processing
❖ Workflow automation
❖ Distributed processing and parallel computing
❖ Join PyCon TW 2018, 1-2 June: https://tw.pycon.org/2018/

4. Interactive
❖ http://jupyter.org: run Python everywhere and code it
through browser
❖ https://matplotlib.org: de facto 2D plotting library
❖ https://www.vtk.org: versatile 3D visualization toolbox
❖ https://www.paraview.org: if you only want a
frontend

5. Data Manipulation
❖ http://pandas.pydata.org: de facto tool for data
analytics
❖ http://www.pytables.org: HDF5 hierarchical data
access
❖ http://unidata.github.io/netcdf4-python/: netCDF, yet
another data storage based on HDF5

6. Workflow Automation
❖ https://docs.python.org/3/library/argparse.html:
standard command-line argument processing
❖ https://github.com/saltstack/salt: cloud-oriented
automation for management and configuration
❖ AWS, GCE, Azure all offer SDK for Python

7. Concurrency
❖ https://docs.python.org/3/library/asyncio.html
❖ support native asynchronous constructs
❖ https://docs.python.org/3/library/multiprocessing.html
❖ parallel computing and distributed processing using multiple
processes
❖ http://zeromq.org/bindings:python
❖ socket communication
❖ http://pythonhosted.org/mpi4py/
❖ use MPI in Python

9. The Only Thing That Doesn’t Change, Is Change
❖ Way too many options for computing
❖ What an engineer should learn?
❖ Yes, I know, Python. But it’s all the tools with Python,
and not Python alone, right?
❖ Can we find something so fundamental that doesn’t
change?

10. Machine Code
❖ Computer can only execute machine code
❖ But it looks hard.
❖ And it is really hard.
_main:
0000000100001c80 pushq %rbp
0000000100001c81 movq %rsp, %rbp
0000000100001c84 subq $0x60, %rsp
0000000100001c88 leaq 0x2cd(%rip), %rax
0000000100001c8f movl $0x0, -0x4(%rbp)
0000000100001c96 movl %edi, -0x8(%rbp)
0000000100001c99 movq %rsi, -0x10(%rbp)
0000000100001c9d movq %rax, %rdi
0000000100001ca0 callq 0x100001ee8
...

11. C++: Compromise
❖ First, there was the C programming language
❖ It allows us to write a little bit more friendly code
❖ A compiler takes the code and turn it into assembly,
and it is then turned into machine code
❖ Some geniuses want to make something super powerful
but looks like C
❖ It produces the fastest possible machine code from
machine-friendly, convoluted concepts

12. Slow for Fast, Fast Is Slow
❖ It takes a lot of time to write code that computers run
fast
❖ C++ -> machine code
❖ Quickly written code takes additional time from
computers to interpret
❖ Python
❖ Ease of use and performance are competing. You cannot
get both.

13. As Easy as Python, as Fast as C++
❖ Create a new tool that serves both purposes?
❖ You need to re-create two whole eco-systems that
develop for 20-30 years
❖ Doesn’t sound very realistic, does it?
❖ How about using C++ for what needs speed, and
Python for everything else? Yes if we have a way to
bridge them!

14. pybind11
❖ C++, born in 1985 (development since 1979)
❖ ISO standard: C++98, 03, 11, 14, 17
❖ Python, born in 1991
❖ Community maintained
❖ C++11/14/17(/20) is referred to as modern C++ and usually just “C+
+11”
❖ C++ is notoriously hard, but modern C++ starts to make sense
❖ pybind11 is a tool allowing us to use exclusively modern C++ with
Python

15. Wrapping Tools for Python Abound
❖ So many software packages provide Python API
❖ pybind11, boost.python, Cython, ctypes, swig, etc.
❖ Python C API

16. Multi-Language Systems
❖ Recall the purpose: quickly make fast software
❖ Know very well the system fundamentals
❖ It’s always the key to a good software system
❖ In C++: core data structure and performance hotspots
❖ In Python: workflow and customization

17. Things to Learn
❖ Master your problem domain: mathematics and
mechanics
❖ Computer architecture
❖ Numerical analysis
❖ Practice Python and then C++ for 10 years

18. Mathematics and Mechanics
❖ Calculus
❖ Linear algebra
❖ Vector analysis
❖ Thermodynamics
❖ Continuum mechanics: solids and fluids
❖ Materials and chemistry

19. Computer Architecture
❖ Abstract description for how silicon is made for
computing
❖ What runs fast and what runs slow, and why
❖ How do you code to make the best use of the hardware

20. Numerical Analysis
❖ Turn mathematics into code; the mathematics describes
the physics
❖ Interpolation
❖ Equation solving
❖ Linear systems
❖ Optimization

21. How to Learn Python
❖ Python should be the first programming language to
learn because it’s easy to use
❖ Edit a text file and run it
❖ Open a browser and run it
print("hello, world") $ python3 hello.py
hello, world

22. Step 1: Install Runtime
❖ https://www.anaconda.com/download/
❖ Anaconda contains Python and a lot of useful software
packages.
❖ When one starts, it’s even difficult to find and install
software, but Anaconda solves the problem
❖ Use command line as much as possible
❖ If you can, use Linux (Ubuntu) or mac osx, which give you a
powerful command-line shell. Windows 10 professional has
one too, but not as popular as the Unixes.

23. Step 2: Interact
Programmers try and error and then CPR:
copy-paste-replace
Highly unlikely to get things right the
first time

24. Step 3: Review and Rewrite
❖ Continuously change the code
❖ Automatic testing is necessary
❖ No one can fully understand 10,000 lines of
code and make sure it’s right
❖ Architect the system
❖ Develop features piece by piece
❖ Connect components
❖ Reorganize structure to make it bigger
solvcon
!"" __init__.py
!"" anchor.py
!"" batch.py
!"" batch_torque.py
!"" block.py
!"" boundcond.py
!"" case.py
!"" case_core.py
!"" cloud.py
!"" cmdutil.py
!"" command.py
!"" conf.py
!"" connection.py
!"" dependency.py
!"" domain.py
!"" exception.py
!"" gendata.py
!"" helper.py
!"" hook.py
!"" io
# !"" __init__.py
# $"" ...
!"" mpy.py
!"" mthread.py
!"" parcel
# !"" __init__.py
# $"" ...
!"" rpc.py
!"" scuda.py
!"" solver.py
!"" testing.py
$"" ...

25. How to Learn C++
❖ Never starts with C++
❖ Benchmark first
❖ Use C++ when you need to control from byte to byte
❖ Keep in mind the differences between Python and C++
❖ Python function calls are 10x-100x slower than C++
❖ Python determines everything in runtime; C++
determines many things in compile time.

26. Compilation
Source file A
Construct 1
Construct 2
Source file B
Construct 1
Construct 2
Binary
Construct A.1
Construct B.2
Construct B.1
Construct A.2
…
Compile Runtime
input
data data
output
data data
Binary
Construct A.1
Construct B.2
Construct B.1
Construct A.2
}Pythondoesn’thaveit

27. Learn by Doing
❖ Incentive is utmost important: Program to solve your
problems
❖ After your 500,000 lines of code, you’ll get it
❖ If you don’t code enough, reading books doesn’t help

28. Your Job
❖ Make mobile app run faster? No problem
❖ Scale web page higher? You nail it
❖ Increase video game frame rate? Yes
❖ Crunch more data? Absolutely
❖ Make scientific discoveries? That’s what you do
❖ If you can quickly make computers run fast, jobs find
you