SlideShare uma empresa Scribd logo
1 de 59
Python for High Performance and Scientific
Computing

EuroPython 2011 (23.06.2011, Florence, Italy)
Andreas Schreiber <Andreas.Schreiber@dlr.de>
German Aerospace Center (DLR), Cologne
http://www.dlr.de/sc


                                     EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Abstract



“Python is an accepted high-level scripting language with a growing community in
academia and industry. It is used in a lot of scientific applications in many different
scientific fields and in more and more industries, for example, in engineering or life
science). In all fields, the use of Python for high-performance and parallel computing is
increasing. Several organizations and companies are providing tools or support for
Python development. This includes libraries for scientific computing, parallel
computing, and MPI. Python is also used on many core architectures and GPUs, for
which specific Python interpreters are being developed. A related topic is the
performance of the various interpreter and compiler implementations for Python.
The talk gives an overview of Python’s use in HPC and Scientific Computing and gives
information on many topics, such as Python on massively parallel systems, GPU
programming with Python, scientific libraries in Python, and Python interpreter
performance issues. The talk will include examples for scientific codes and
applications from many domains..“
EuroPython 2011:
http://ep2011.europython.eu/conference/talks/python-for-high-performance-and-
scientific-computing
                                                                                                                                            Folie 2
                                              EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Outline


!   "HPC! and Python
!   "Tools and Libraries
!   "Debugging and Profiling
!   "Example Applications
!   "Python and HPC communities




                                                                                                                                Folie 3
                                  EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Related Talks at EuroPython


!   "Debugging and Profiling Techniques (Giovanni Bajo)
!   "Exploit your GPU Power with PyCUDA (and friends) (Stefano Brilli)
!   "Experiences making CPU-bound tasks run much faster (Ian Ozsvald)
!   "High-Performance Computing on Gamer PCs (Yann Le Du)
!   "Python(x,y): Diving into Scientific Python (Vincent Noel)
!   "and more!




                                                                                                                                        Folie 4
                                          EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Folie 5
EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
DLR
German Aerospace Center




!   "Research Institution
!    "Space Agency
!     "Project Management Agency




                                                                                                                                 Folie 6
                                   EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Locations and employees


6900 employees across                                    Stade ! ! Hamburg
33 institutes and facilities at                                                                           ! Neustrelitz
                                              Bremen !                        ! Trauen
      !" 15 sites.
                                                                                                  Berlin !
                                                  Braunschweig !

Offices in Brussels,                                                       ! Goettingen
Paris and Washington.
                                           ! Cologne
                                           ! Bonn


                                  Lampoldshausen !

                                             Stuttgart !
                                                      Augsburg !
                                                                                        ! Oberpfaffenhofen
                                                                  Weilheim !




                                                                                                                                Folie 7
                                  EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Software Development at DLR
More than 1000 employees develop!

           Mission-critical software
           (e.g., real-time software)

           Web-based software for
           internet and intranet

           Supporting software
           (e.g., data management)

           High-Performance and
           Scientific Computing Software
                                                                                                                        Folie 8
                          EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
HPC! and Python




                                                                                                                Folie 9
                  EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Overview of Applications
Meteorology, Astrophysics, Particle physics, Biology,
Genetics, Quantum chemistry, Fluid dynamics, Finance,
Oil exploration, !




                                                                       Images from: http://www.isgtw.org

                                                                                                                        Folie 10
                            EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
HPC Systems


!   "Current top systems have > 500,000 cores and > 8,000 TFlops
!   "See TOP500 list of supercomputers
    (http://www.top500.org)
!   "Top 3 systems (June 2011) are:
     1." K computer, Japan
     2." Tianhe-1A, China
     3." Jaguar, Unites States




                                                                                                                                      Folie 11
                                         EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
#1: K computer, SPARC64 VIIIfx 2.0GHz, Tofu interc. (RIKEN
Advanced Institute for Computational Science (AICS), Japan)




                                                                                                                            Folie 12
                                EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
#2: Tianhe-1A - NUDT TH MPP, X5670 2.93Ghz 6C, NVIDIA GPU,
FT-1000 8C (National Supercomputing Center in Tianjin, China)




                                                                                                                            Folie 13
                                EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
#3: Jaguar - Cray XT5-HE Opteron 6-core 2.6 GHz (DOE/SC/Oak
Ridge National Laboratory, United States)




                                                                                                                           Folie 14
                               EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Applications
Python is used in!

!   "Computational Fluid Dynamics (CFD)
!   "Plasma simulation
!   "Bio molecular simulation
!   "Artificial intelligence
!   "Natural language processing
!   "Data mining
!   "Scientific visualization
!   "Robotics
!   "Computer games
!   "System administration
!   "Education
!   "!


                                                                                                                                  Folie 15
                                      EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Python for Scientists and Engineers
Reasons for Python in Research and Industry

Observations:                                                                                       “I want to design
!   "Scientists and engineers don’t want to write software                                               planes,
                                                                                                      not software!”
    but just solve their problems
!   "If they have to write code, it must be as easy as
    possible
Why Python is perfect?
!   "Very easy to learn and easy to use
    ( = steep learning curve)
!   "Allows rapid development
    ( = short development time)
!   "Inherent great maintainability


                                                                                                                                      Folie 16
                                          EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
“
!"#$#%&##'&%()%*#%(+)%&)$(&%
),%-#)-.#%+")%.)/#%01(")23%
(")&#%+")%"4(#%*$456#(&7%
428%&59#2:&(&;%%

                                                                                                           Folie 17
               EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 10.07.2008
                                                                                                            23.06.2011
“
<,%9(=&%>))8%#2)?>"%,)$%%
@))>.#%428%ABCB7%9(=&%%
>))8%#2)?>"%,)$%'#7%*4*1;%%


                                                                                                           Folie 18
               EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 10.07.2008
                                                                                                            23.06.2011
“
01(")2%"4&%("#%5.#42#&(7%%
')&(D&59#2:&(D%)$%#2>92##$%%
,$9#28.1%&12(4E%428%%
&#'42:5&;%            Paul F. Dubois!



                                                                                                                Folie 19
                    EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 10.07.2008
                                                                                                                 23.06.2011
Tools and Libraries for HPC and SC
with Python




                      EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Scientific Tools and Libraries
General Tools

Very general scientific computing   High Performance Computing
! "NumPy                            Parallel Computing
! "SciPy                            !  PETSc
                                    !  PyMPI
Visualization                       !  Pypar
!  Matplotlib                       ! "mpi4py
!  VisIt                            GPGPU Computing
!  MayaVi                           ! "PyCUDA
! "Chaco                            ! "PyOpenCL
! "VTK




                                                                                                                                Folie 21
                                    EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Scientific Tools and Libraries
Domain Specific Tools
AI               Molecular & Atomic                       Electromagnetics
!    "pyem       Modeling                                 !  PyFemax
!    "ffnet      !    PyMOL
!    "pymorph    !    Biskit                              Astronomy
!    "Monte      !   "GPAW                                !  AstroLib
!    hcluster                                             !  PySolar
                 Geo sciences
Biology          ! "GIS Python                            Dynamic Systems
! "Brian         !  PyClimate                             !  Simpy
! "SloppyCell    !  ClimPy                                !  PyDSTool
! "NIPY          ! "CDAT
! "PySAT                                                  Finite Elements
                                                          !   SfePy
                                                                                                                             Folie 22
                                 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Scientific Tools and Libraries
Special Topics

Wrapping other languages
! "weave (C/C++)
! "f2py (Fortran)
!  Cython
!  Ctypes (C)
! "SWIG (C/C++)
!  RPy / RSPython (R)
!  MatPy (Matlab)
!  Jython (Java)
!  IronPython (.NET)




                                                                                                                        Folie 23
                            EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
NumPy


!   "Website: http://numpy.scipy.org/

!   "Offers capabilities similar to MATLAB within Python


!   "N-dimensional homogeneous arrays (ndarray)
!   "Universal functions (ufunc)
      !   "basic math, linear algebra, FFT, PRNGs
!   "Simple data file I/O
      !   "text, raw binary, native binary
!   "Tools for integrating with C/C++/Fortran
!   "Heavy lifting done by optimized C/Fortran libraries
      !   "ATLAS or MKL, UMFPACK, FFTW, etc...


                                                                                                                                         Folie 24
                                             EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
NumPy
Arrays

!   "A NumPy array is an N-dimensional homogeneous collection of “items”
     of the same “kind”. The kind can be any arbitrary structure and is
     specified using the data-type.
!   "Example: Multidimensional NumPy array

              >>> a = array([[ 0, 1, 2, 3],
                             [10,11,12,13]])
              >>> a
              array([[ 0, 1, 2, 3],
                     [10,11,12,13]])

              >>> a[1,3]
              13
              >>> a[1,3] = -1
              >>> a
              array([[ 0, 1, 2, 3],
                     [10,11,12,-1]])

                                                                                                                                     Folie 25
                                         EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
SciPy
Scientific Tools for Python

!   "Website: http://www.scipy.org
!   "Large library of scientific algorithms
!   "Extends NumPy with many tools for science and engineering
!   "Computationally intensive routines implemented in C and Fortran




                                                                                                                                          Folie 26
                                              EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
SciPy
Packages

!   "Special Functions
!   "Signal Processing
!   "Fourier Transforms
!   "Optimization
!   "Numerical Integration
!   "Linear Algebra
!    Input/Output
!   "Statistics
!   "Fast Execution
!   "Clustering Algorithms
!   "Sparse Matrices




                                                                                                                         Folie 27
                             EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Parallel Programming


Threading
! "Useful for certain concurrency issues, not usable for parallel computing
   due to Global Interpreter Lock (GIL)
subprocess
!   "Relatively low level control for spawning and managing processes
!   "multiprocessing - multiple Python instances (processes)
!   "basic, clean multiple process parallelism
MPI
! "mpi4py exposes your full local MPI API within Python
! "As scalable as your local MPI
GPU (OpenCL & CUDA)
! "PyOpenCL and PyCUDA provide low and high level abstraction for
   highly parallel computations on GPUs

                                                                                                                                      Folie 28
                                          EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
mpi4py


!   "Website: http://mpi4py.scipy.org/
!   "Wraps native MPI implementations
      !   "Prefers MPI2, but can work with MPI1
!   "Works best with NumPy data types, but can pass around any serializable
    object
!   "Provides all MPI2 features
!   "Well maintained
!   "Distributed with Enthought Python Distribution (EPD)
!   "Requires NumPy
!   "Portable and scalable




                                                                                                                                     Folie 29
                                         EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
How mpi4py works...


!   "mpi4py jobs must be launched with mpirun
!   "Each rank launches its own independent python interpreter
!   "Each interpreter only has access to files and libraries available locally to
    it, unless distributed to the ranks
!   "Communication is handled by MPI layer
!   "Any function outside of an if block specifying a rank is assumed to be
     global
!   "Any limitations of your local MPI are present in mpi4py




                                                                                                                                       Folie 30
                                           EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
from mpi4py import MPI
Example:        import numpy as np
Calculating "   import random
with mpi4py     comm = MPI.COMM_WORLD
                rank = comm.Get_rank()
                mpisize = comm.Get_size()
                nsamples = int(12e6/mpisize)

                inside = 0
                random.seed(rank)
                for i in range(nsamples):
                    x = random.random()
                    y = random.random()
                    if (x*x)+(y*y)<1:
                      inside += 1

                mypi = (4.0 * inside)/nsamples
                pi = comm.reduce(mypi, op=MPI.SUM, root=0)

                if rank==0:
                    print (1.0 / mpisize)*pi
                                                                                                                          Folie 31
                              EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
GPGPU
General-purpose computing on graphics processing units

Architecture
! "Many cores per node
! "Good for stream processing (independent vertices, many of them in
   parallel)
CUDA
!   "NVIDIA’s platform for C programming on GPGPUs
!   "Python binding: PyCUDA
OpenCL
! "Framework for writing programs that execute
   across heterogeneous platforms consisting of
   CPUs, GPUs, and other processors
!   "Open standard
!   "Python binding: PyOpenCL

                                                                                                                                  Folie 32
                                      EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
PyCUDA


!   "Website: http://mathema.tician.de/software/pycuda
!   "PyCUDA lets you access NVIDIA‘s CUDA parallel computation API
    from Python
!   "Integration with NumPy
!   "All CUDA errors are automatically
    translated into Python exceptions




                                                                                                                                     Folie 33
                                         EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
PyOpenCL


!   "Website: http://mathema.tician.de/software/pyopencl
!   "Convenient access to full OpenCL API from Python
!   "OpenCL errors translated into Python exceptions
!   "Object cleanup tied to object lifetime (follows Resource Acquisition Is
    Initialization)
!   "NumPy ndarrays interact easily with OpenCL buffers




                                                                                                                                      Folie 34
                                          EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
OpenCL Computing Model


Buffers
! "used for holding data, shipping data between devices


Kernels
! "functions which will operate on buffers in parallel


Queue
! "control sequence of operations: transferring buffers, running kernels,
   waiting on events, etc.




                                                                                                                                     Folie 35
                                         EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
" with PyOpenCL
import pyopencl as cl
import pyopencl.clrandom
import numpy as np

nsamples = int(12e6)

# set up context and queue
ctx = cl.create_some_context()
queue = cl.CommandQueue(ctx)

# create array of random values in OpenCL
xy = pyopencl.clrandom.rand(ctx,queue,(nsamples,2),np.float32)

# square values in OpenCL
xy = xy**2

# 'get' method on xy is used to get array from OpenCL into ndarray
print 4.0*np.sum(np.sum(xy.get(),1)<1)/nsamples
                                                                                                                                     Folie 36
                                         EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Copperhead: Data-Parallelism Embedded in Python
Slides by Michael Garland (NVIDIA Research)




    http://bit.ly/jK7nsM



                                                                                                                     Folie 37
                         EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Debugging and Profiling




                                                                                                               Folie 38
                   EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Debugging Python and C


!   "Python comes with gdb-like
    command line debugger pdb
!   "Helps in debugging serial
     Python applications
!   "C-extensions can be debugged
     with standard debuggers
!   "Parallel debugging
      !   Totalview can be used for
          debugging C-functions
      !   "Custom interpreter
          provides easy initial
          breakpoint



                                                                                                                                  Folie 39
                                      EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Profiling Mixed Python-C code


!   "Finding performance bottlenecks is critical to scalability on HPC
    platforms
!   "Number of profiling tools available:
      !    gprof, CrayPAT, scalasca - C, Fortran
             ! "can be used for C-extensions
      !   "import profile – Python


!   "TAU Performance System,
    http://www.cs.uoregon.edu/research/tau/home.php
      ! "Excurse: Slides by Sameer Shende (University of Oregon)




                                                                                                                                        Folie 40
                                            EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
TAU Performance System
Slides by Sameer Shende (University of Oregon)




http://slidesha.re/lsDZSj



                                                                                                                        Folie 41
                            EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Example Applications

                                                                       Stand #




                                                                                                  Folie 42
      EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Example:
GIST Maxwell’s Equation Solver (GMES)

!   "Object-Oriented Implementation of the Finite-Difference Time-Domain
    Method in Parallel Computing Environment
!   "Slides by Kyungwon Chun et al. (GIST - Gwagnju Institute of Science
    and Technology, South Korea)!




     http://bit.ly/mCs8W9

                                                                                                                                    Folie 43
                                        EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Example: FlowSimulator – A Python-controlled
framework to unify massive parallel CFD workflows




                                                                                                                         Folie 44
                             EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
What does CFD mean?


!   "CFD = Computational fluid dynamics means to
       ! " solve the compressible flow equations to
       ! " predict an aircrafts behavior and to
       ! " optimize its features.


!   "Numerical approach:
       ! " Spatial discretization
       ! " Simplified models for turbulent effects
       ! " Temporal discretization (to support
           unsteady flows)




                                                                                                                                           Folie 45
                                               EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Evolution of CFD
Classical chain
                                                                                                             Z




!   "Simple chain
                                                                                                             Y   X




       ! " Geometry
       ! " Mesh generator
       ! " Pre-processor
       ! " Flow solver
       ! " Post-processor
       ! " Visualization


!   "Tools
      !      " stand-alone mesher
      !      " stand-alone solver
               (with pre-/postprocessor)
      !      " shell scripts (e.g. different
               solver parameters)


                                                                                                                                           Folie 46
                                               EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Evolution of CFD
Solver-adaptation-loop
                                                                                                         Z




!   "Chain with solver-adaptation-loop
                                                                                                         Y   X




       ! " several consecutive solver
           steps
       ! " adaptation of mesh in between


!   "Tools:
       !      " stand-alone mesher
       !      " stand-alone solver
       !      " stand-alone adaptation
       !      " shell script control


!   "Solver/adaptation as Python module




                                                                                                                                       Folie 47
                                           EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Evolution of CFD
Multi-disciplinary simulations

!   "Multi-disciplinary chain
                                                                                                              Z



                                                                                                              Y   X




       ! " output of solver is input for
               ! " structural mechanics
                    (deformation)
               ! " flight mechanics (control
                    surface movement)
       ! " output of structural/flight
            mechanics modifies mesh

!   "Tools:
       !      " solver as Python module
       !      " stand-alone CSM
       !      " Python-based flight mechanics




                                                                                                                                            Folie 48
                                                EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Structure and Flight Mechanics




                                                                                                                      Folie 49
                          EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
FlowSimulator Environment
  Single Python script controls the whole workflow


                               FSControl
    Out of scope




                                      Flight
                   CFD   CSM                                                        CAA
                                    mechanics




Import                                                                                                             Export
                          FSDataManager


                                                                                                                                  Folie 50
                                      EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
FSControl
!   "Pure Python library

!   "Set of simple Python classes for different modules/tools:
      !   "simple interface: Import, Run, Export
      !   "data stored in FSDataManager


FSDataManager (FSDM)
!   "C++ library, wrapped to Python
      !   "FSDataManager is a real Python module!

!   "Parallelization context
      !   "all data structures “distributed”
      !   "facilitated transfers (Gather, Distribute, Send, !)



                                                                                                                                           Folie 51
                                               EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Python Framework for Coupled Fusion Simulations
Slides by Samantha Foley et al. (ORNL – Oak Ridge
National Laboratory, United States)




http://slidesha.re/iLuegW



                                                                                                                      Folie 52
                          EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Python & HPC Communities




               EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Where the Python and HPC Communities Meet!


!   "SciPy
     !   "Python for Scientific Computing
         Conference (since 2008)
!   "EuroSciPy
     !   "Annual European Conference for
         Scientists using Python (since 2008)
!   "SCxx
      ! "The International Conference for High
         Performance Computing, Networking,
         Storage, and Analysis (annual
         conference since 1988)
      ! "Tutorial and BoF session on Python
         (since 2009)


                                                                                                                                    Folie 54
                                        EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
The SC Conference Series (SCxx)
Tutorials (est. 2009)

!   "Python for Scientific and High Performance Computing
!   "Given by William R. Scullin (ANL – Argonne National Lab, United States)
    et al.
!   "Content
      !      "Python language and interpreter basics
      !      "Popular modules and packages for scientific applications
      !      "How to improve performance in Python programs
      !      "How to visualize and share data using Python
      !      "Where to find documentation and resources




                                                                                                                                        Folie 55
                                            EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
The SC Conference Series (SCxx)
Birds-of-a-Feather Sessions, est. 2009




                                                                                                                         Folie 56
                             EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Credits
          Credits
          Achim Basermann (DLR)
          Kyungwon Chun (GIST)
          Samantha Foley (ORNL)
          Michael Garland (NVIDIA)
          Michael Meinel (DLR)
          Tr a v i s O l i p h a n t ( E n t h o u g h t )
          William R. Scullin (ANL)
          Sameer Shende (U Oregon)




                                                                                                                  Folie 57
                      EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Questions?

Summary
•" Python used in HPC applications
•" Good tools available (SciPy etc.)
•" Wrapping of C/C++/Fortran is common

                Andreas Schreiber
                A n d r e a s . S c h r e i b e r @ d l r. d e
                http://www.dlr.de/sc                                                                                  Folie 58
                          EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
Contact


  python@dlr.de
     @onyame

          Andreas Schreiber
          A n d r e a s . S c h r e i b e r @ d l r. d e
          http://www.dlr.de/sc                                                                                  Folie 59
                    EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011

Mais conteúdo relacionado

Semelhante a Python for High Performance and Scientific Computing

IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)PyData
 
Python Master Thesis Projects in UK.
Python Master Thesis Projects in UK.Python Master Thesis Projects in UK.
Python Master Thesis Projects in UK.Phdtopiccom
 
Python for High Performance and Scientific Computing
Python for High Performance and Scientific ComputingPython for High Performance and Scientific Computing
Python for High Performance and Scientific ComputingAndreas Schreiber
 
Building a scientific workbench in Pharo
Building a scientific workbench in PharoBuilding a scientific workbench in Pharo
Building a scientific workbench in PharoESUG
 
用 Python 玩 LHC 公開數據
用 Python 玩 LHC 公開數據用 Python 玩 LHC 公開數據
用 Python 玩 LHC 公開數據Yuan CHAO
 
Python Intro For Managers
Python Intro For ManagersPython Intro For Managers
Python Intro For ManagersAtul Shridhar
 
¿Es posible construir el Airbus de la Supercomputación en Europa?
¿Es posible construir el Airbus de la Supercomputación en Europa?¿Es posible construir el Airbus de la Supercomputación en Europa?
¿Es posible construir el Airbus de la Supercomputación en Europa?AMETIC
 
Webinar: Mastering Python - An Excellent tool for Web Scraping and Data Anal...
Webinar:  Mastering Python - An Excellent tool for Web Scraping and Data Anal...Webinar:  Mastering Python - An Excellent tool for Web Scraping and Data Anal...
Webinar: Mastering Python - An Excellent tool for Web Scraping and Data Anal...Edureka!
 
Why Python Should Be Your First Programming Language
Why Python Should Be Your First Programming LanguageWhy Python Should Be Your First Programming Language
Why Python Should Be Your First Programming LanguageEdureka!
 
Python 101 For The Net Developer
Python 101 For The Net DeveloperPython 101 For The Net Developer
Python 101 For The Net DeveloperSarah Dutkiewicz
 
'Scikit-project': How open source is empowering open science – and vice versa
'Scikit-project': How open source is empowering open science – and vice versa'Scikit-project': How open source is empowering open science – and vice versa
'Scikit-project': How open source is empowering open science – and vice versaNathan Shammah
 
Programming for data science in python
Programming for data science in pythonProgramming for data science in python
Programming for data science in pythonUmmeSalmaM1
 

Semelhante a Python for High Performance and Scientific Computing (20)

IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
 
Python Master Thesis Projects in UK.
Python Master Thesis Projects in UK.Python Master Thesis Projects in UK.
Python Master Thesis Projects in UK.
 
Sci computing using python
Sci computing using pythonSci computing using python
Sci computing using python
 
Python for High Performance and Scientific Computing
Python for High Performance and Scientific ComputingPython for High Performance and Scientific Computing
Python for High Performance and Scientific Computing
 
Python
Python Python
Python
 
05 python.pdf
05 python.pdf05 python.pdf
05 python.pdf
 
Building a scientific workbench in Pharo
Building a scientific workbench in PharoBuilding a scientific workbench in Pharo
Building a scientific workbench in Pharo
 
Python final ppt
Python final pptPython final ppt
Python final ppt
 
Pythonfinalppt 170822121204
Pythonfinalppt 170822121204Pythonfinalppt 170822121204
Pythonfinalppt 170822121204
 
用 Python 玩 LHC 公開數據
用 Python 玩 LHC 公開數據用 Python 玩 LHC 公開數據
用 Python 玩 LHC 公開數據
 
Python Intro For Managers
Python Intro For ManagersPython Intro For Managers
Python Intro For Managers
 
¿Es posible construir el Airbus de la Supercomputación en Europa?
¿Es posible construir el Airbus de la Supercomputación en Europa?¿Es posible construir el Airbus de la Supercomputación en Europa?
¿Es posible construir el Airbus de la Supercomputación en Europa?
 
Webinar: Mastering Python - An Excellent tool for Web Scraping and Data Anal...
Webinar:  Mastering Python - An Excellent tool for Web Scraping and Data Anal...Webinar:  Mastering Python - An Excellent tool for Web Scraping and Data Anal...
Webinar: Mastering Python - An Excellent tool for Web Scraping and Data Anal...
 
Python
PythonPython
Python
 
Why Python Should Be Your First Programming Language
Why Python Should Be Your First Programming LanguageWhy Python Should Be Your First Programming Language
Why Python Should Be Your First Programming Language
 
Python 101 For The Net Developer
Python 101 For The Net DeveloperPython 101 For The Net Developer
Python 101 For The Net Developer
 
'Scikit-project': How open source is empowering open science – and vice versa
'Scikit-project': How open source is empowering open science – and vice versa'Scikit-project': How open source is empowering open science – and vice versa
'Scikit-project': How open source is empowering open science – and vice versa
 
Programming for data science in python
Programming for data science in pythonProgramming for data science in python
Programming for data science in python
 
python into.pptx
python into.pptxpython into.pptx
python into.pptx
 
Computer Science in Saarland
Computer Science in Saarland Computer Science in Saarland
Computer Science in Saarland
 

Mais de Andreas Schreiber

Provenance-based Security Audits and its Application to COVID-19 Contact Trac...
Provenance-based Security Audits and its Application to COVID-19 Contact Trac...Provenance-based Security Audits and its Application to COVID-19 Contact Trac...
Provenance-based Security Audits and its Application to COVID-19 Contact Trac...Andreas Schreiber
 
Visualization of Software Architectures in Virtual Reality and Augmented Reality
Visualization of Software Architectures in Virtual Reality and Augmented RealityVisualization of Software Architectures in Virtual Reality and Augmented Reality
Visualization of Software Architectures in Virtual Reality and Augmented RealityAndreas Schreiber
 
Provenance as a building block for an open science infrastructure
Provenance as a building block for an open science infrastructureProvenance as a building block for an open science infrastructure
Provenance as a building block for an open science infrastructureAndreas Schreiber
 
Raising Awareness about Open Source Licensing at the German Aerospace Center
Raising Awareness about Open Source Licensing at the German Aerospace CenterRaising Awareness about Open Source Licensing at the German Aerospace Center
Raising Awareness about Open Source Licensing at the German Aerospace CenterAndreas Schreiber
 
Open Source Licensing for Rocket Scientists
Open Source Licensing for Rocket ScientistsOpen Source Licensing for Rocket Scientists
Open Source Licensing for Rocket ScientistsAndreas Schreiber
 
Interactive Visualization of Software Components with Virtual Reality Headsets
Interactive Visualization of Software Components with Virtual Reality HeadsetsInteractive Visualization of Software Components with Virtual Reality Headsets
Interactive Visualization of Software Components with Virtual Reality HeadsetsAndreas Schreiber
 
Provenance for Reproducible Data Science
Provenance for Reproducible Data ScienceProvenance for Reproducible Data Science
Provenance for Reproducible Data ScienceAndreas Schreiber
 
Visualizing Provenance using Comics
Visualizing Provenance using ComicsVisualizing Provenance using Comics
Visualizing Provenance using ComicsAndreas Schreiber
 
Nachvollziehbarkeit mit Hinblick auf Privacy-Verletzungen
Nachvollziehbarkeit mit Hinblick auf Privacy-VerletzungenNachvollziehbarkeit mit Hinblick auf Privacy-Verletzungen
Nachvollziehbarkeit mit Hinblick auf Privacy-VerletzungenAndreas Schreiber
 
Reproducible Science with Python
Reproducible Science with PythonReproducible Science with Python
Reproducible Science with PythonAndreas Schreiber
 
A Provenance Model for Quantified Self Data
A Provenance Model for Quantified Self DataA Provenance Model for Quantified Self Data
A Provenance Model for Quantified Self DataAndreas Schreiber
 
Tracking after Stroke: Doctors, Dogs and All The Rest
Tracking after Stroke: Doctors, Dogs and All The RestTracking after Stroke: Doctors, Dogs and All The Rest
Tracking after Stroke: Doctors, Dogs and All The RestAndreas Schreiber
 
High Throughput Processing of Space Debris Data
High Throughput Processing of Space Debris DataHigh Throughput Processing of Space Debris Data
High Throughput Processing of Space Debris DataAndreas Schreiber
 
Bericht von der QS15 Conference & Exposition
Bericht von der QS15 Conference & ExpositionBericht von der QS15 Conference & Exposition
Bericht von der QS15 Conference & ExpositionAndreas Schreiber
 
Telemedizin: Gesundheit, messbar für jedermann
Telemedizin: Gesundheit, messbar für jedermannTelemedizin: Gesundheit, messbar für jedermann
Telemedizin: Gesundheit, messbar für jedermannAndreas Schreiber
 
Quantified Self mit Wearable Devices und Smartphone-Sensoren
Quantified Self mit Wearable Devices und Smartphone-SensorenQuantified Self mit Wearable Devices und Smartphone-Sensoren
Quantified Self mit Wearable Devices und Smartphone-SensorenAndreas Schreiber
 

Mais de Andreas Schreiber (20)

Provenance-based Security Audits and its Application to COVID-19 Contact Trac...
Provenance-based Security Audits and its Application to COVID-19 Contact Trac...Provenance-based Security Audits and its Application to COVID-19 Contact Trac...
Provenance-based Security Audits and its Application to COVID-19 Contact Trac...
 
Visualization of Software Architectures in Virtual Reality and Augmented Reality
Visualization of Software Architectures in Virtual Reality and Augmented RealityVisualization of Software Architectures in Virtual Reality and Augmented Reality
Visualization of Software Architectures in Virtual Reality and Augmented Reality
 
Provenance as a building block for an open science infrastructure
Provenance as a building block for an open science infrastructureProvenance as a building block for an open science infrastructure
Provenance as a building block for an open science infrastructure
 
Raising Awareness about Open Source Licensing at the German Aerospace Center
Raising Awareness about Open Source Licensing at the German Aerospace CenterRaising Awareness about Open Source Licensing at the German Aerospace Center
Raising Awareness about Open Source Licensing at the German Aerospace Center
 
Open Source Licensing for Rocket Scientists
Open Source Licensing for Rocket ScientistsOpen Source Licensing for Rocket Scientists
Open Source Licensing for Rocket Scientists
 
Interactive Visualization of Software Components with Virtual Reality Headsets
Interactive Visualization of Software Components with Virtual Reality HeadsetsInteractive Visualization of Software Components with Virtual Reality Headsets
Interactive Visualization of Software Components with Virtual Reality Headsets
 
Provenance for Reproducible Data Science
Provenance for Reproducible Data ScienceProvenance for Reproducible Data Science
Provenance for Reproducible Data Science
 
Visualizing Provenance using Comics
Visualizing Provenance using ComicsVisualizing Provenance using Comics
Visualizing Provenance using Comics
 
Quantified Self Comics
Quantified Self ComicsQuantified Self Comics
Quantified Self Comics
 
Nachvollziehbarkeit mit Hinblick auf Privacy-Verletzungen
Nachvollziehbarkeit mit Hinblick auf Privacy-VerletzungenNachvollziehbarkeit mit Hinblick auf Privacy-Verletzungen
Nachvollziehbarkeit mit Hinblick auf Privacy-Verletzungen
 
Reproducible Science with Python
Reproducible Science with PythonReproducible Science with Python
Reproducible Science with Python
 
Python at Warp Speed
Python at Warp SpeedPython at Warp Speed
Python at Warp Speed
 
A Provenance Model for Quantified Self Data
A Provenance Model for Quantified Self DataA Provenance Model for Quantified Self Data
A Provenance Model for Quantified Self Data
 
Open Source im DLR
Open Source im DLROpen Source im DLR
Open Source im DLR
 
Tracking after Stroke: Doctors, Dogs and All The Rest
Tracking after Stroke: Doctors, Dogs and All The RestTracking after Stroke: Doctors, Dogs and All The Rest
Tracking after Stroke: Doctors, Dogs and All The Rest
 
High Throughput Processing of Space Debris Data
High Throughput Processing of Space Debris DataHigh Throughput Processing of Space Debris Data
High Throughput Processing of Space Debris Data
 
Bericht von der QS15 Conference & Exposition
Bericht von der QS15 Conference & ExpositionBericht von der QS15 Conference & Exposition
Bericht von der QS15 Conference & Exposition
 
Telemedizin: Gesundheit, messbar für jedermann
Telemedizin: Gesundheit, messbar für jedermannTelemedizin: Gesundheit, messbar für jedermann
Telemedizin: Gesundheit, messbar für jedermann
 
Big Python
Big PythonBig Python
Big Python
 
Quantified Self mit Wearable Devices und Smartphone-Sensoren
Quantified Self mit Wearable Devices und Smartphone-SensorenQuantified Self mit Wearable Devices und Smartphone-Sensoren
Quantified Self mit Wearable Devices und Smartphone-Sensoren
 

Último

Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 

Último (20)

Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 

Python for High Performance and Scientific Computing

  • 1. Python for High Performance and Scientific Computing EuroPython 2011 (23.06.2011, Florence, Italy) Andreas Schreiber <Andreas.Schreiber@dlr.de> German Aerospace Center (DLR), Cologne http://www.dlr.de/sc EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 2. Abstract “Python is an accepted high-level scripting language with a growing community in academia and industry. It is used in a lot of scientific applications in many different scientific fields and in more and more industries, for example, in engineering or life science). In all fields, the use of Python for high-performance and parallel computing is increasing. Several organizations and companies are providing tools or support for Python development. This includes libraries for scientific computing, parallel computing, and MPI. Python is also used on many core architectures and GPUs, for which specific Python interpreters are being developed. A related topic is the performance of the various interpreter and compiler implementations for Python. The talk gives an overview of Python’s use in HPC and Scientific Computing and gives information on many topics, such as Python on massively parallel systems, GPU programming with Python, scientific libraries in Python, and Python interpreter performance issues. The talk will include examples for scientific codes and applications from many domains..“ EuroPython 2011: http://ep2011.europython.eu/conference/talks/python-for-high-performance-and- scientific-computing Folie 2 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 3. Outline ! "HPC! and Python ! "Tools and Libraries ! "Debugging and Profiling ! "Example Applications ! "Python and HPC communities Folie 3 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 4. Related Talks at EuroPython ! "Debugging and Profiling Techniques (Giovanni Bajo) ! "Exploit your GPU Power with PyCUDA (and friends) (Stefano Brilli) ! "Experiences making CPU-bound tasks run much faster (Ian Ozsvald) ! "High-Performance Computing on Gamer PCs (Yann Le Du) ! "Python(x,y): Diving into Scientific Python (Vincent Noel) ! "and more! Folie 4 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 5. Folie 5 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 6. DLR German Aerospace Center ! "Research Institution ! "Space Agency ! "Project Management Agency Folie 6 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 7. Locations and employees 6900 employees across Stade ! ! Hamburg 33 institutes and facilities at ! Neustrelitz Bremen ! ! Trauen !" 15 sites. Berlin ! Braunschweig ! Offices in Brussels, ! Goettingen Paris and Washington. ! Cologne ! Bonn Lampoldshausen ! Stuttgart ! Augsburg ! ! Oberpfaffenhofen Weilheim ! Folie 7 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 8. Software Development at DLR More than 1000 employees develop! Mission-critical software (e.g., real-time software) Web-based software for internet and intranet Supporting software (e.g., data management) High-Performance and Scientific Computing Software Folie 8 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 9. HPC! and Python Folie 9 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 10. Overview of Applications Meteorology, Astrophysics, Particle physics, Biology, Genetics, Quantum chemistry, Fluid dynamics, Finance, Oil exploration, ! Images from: http://www.isgtw.org Folie 10 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 11. HPC Systems ! "Current top systems have > 500,000 cores and > 8,000 TFlops ! "See TOP500 list of supercomputers (http://www.top500.org) ! "Top 3 systems (June 2011) are: 1." K computer, Japan 2." Tianhe-1A, China 3." Jaguar, Unites States Folie 11 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 12. #1: K computer, SPARC64 VIIIfx 2.0GHz, Tofu interc. (RIKEN Advanced Institute for Computational Science (AICS), Japan) Folie 12 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 13. #2: Tianhe-1A - NUDT TH MPP, X5670 2.93Ghz 6C, NVIDIA GPU, FT-1000 8C (National Supercomputing Center in Tianjin, China) Folie 13 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 14. #3: Jaguar - Cray XT5-HE Opteron 6-core 2.6 GHz (DOE/SC/Oak Ridge National Laboratory, United States) Folie 14 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 15. Applications Python is used in! ! "Computational Fluid Dynamics (CFD) ! "Plasma simulation ! "Bio molecular simulation ! "Artificial intelligence ! "Natural language processing ! "Data mining ! "Scientific visualization ! "Robotics ! "Computer games ! "System administration ! "Education ! "! Folie 15 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 16. Python for Scientists and Engineers Reasons for Python in Research and Industry Observations: “I want to design ! "Scientists and engineers don’t want to write software planes, not software!” but just solve their problems ! "If they have to write code, it must be as easy as possible Why Python is perfect? ! "Very easy to learn and easy to use ( = steep learning curve) ! "Allows rapid development ( = short development time) ! "Inherent great maintainability Folie 16 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 17. “ !"#$#%&##'&%()%*#%(+)%&)$(&% ),%-#)-.#%+")%.)/#%01(")23% (")&#%+")%"4(#%*$456#(&7% 428%&59#2:&(&;%% Folie 17 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 10.07.2008 23.06.2011
  • 18. “ <,%9(=&%>))8%#2)?>"%,)$%% @))>.#%428%ABCB7%9(=&%% >))8%#2)?>"%,)$%'#7%*4*1;%% Folie 18 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 10.07.2008 23.06.2011
  • 19. “ 01(")2%"4&%("#%5.#42#&(7%% ')&(D&59#2:&(D%)$%#2>92##$%% ,$9#28.1%&12(4E%428%% &#'42:5&;% Paul F. Dubois! Folie 19 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 10.07.2008 23.06.2011
  • 20. Tools and Libraries for HPC and SC with Python EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 21. Scientific Tools and Libraries General Tools Very general scientific computing High Performance Computing ! "NumPy Parallel Computing ! "SciPy ! PETSc ! PyMPI Visualization ! Pypar ! Matplotlib ! "mpi4py ! VisIt GPGPU Computing ! MayaVi ! "PyCUDA ! "Chaco ! "PyOpenCL ! "VTK Folie 21 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 22. Scientific Tools and Libraries Domain Specific Tools AI Molecular & Atomic Electromagnetics ! "pyem Modeling ! PyFemax ! "ffnet ! PyMOL ! "pymorph ! Biskit Astronomy ! "Monte ! "GPAW ! AstroLib ! hcluster ! PySolar Geo sciences Biology ! "GIS Python Dynamic Systems ! "Brian ! PyClimate ! Simpy ! "SloppyCell ! ClimPy ! PyDSTool ! "NIPY ! "CDAT ! "PySAT Finite Elements ! SfePy Folie 22 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 23. Scientific Tools and Libraries Special Topics Wrapping other languages ! "weave (C/C++) ! "f2py (Fortran) ! Cython ! Ctypes (C) ! "SWIG (C/C++) ! RPy / RSPython (R) ! MatPy (Matlab) ! Jython (Java) ! IronPython (.NET) Folie 23 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 24. NumPy ! "Website: http://numpy.scipy.org/ ! "Offers capabilities similar to MATLAB within Python ! "N-dimensional homogeneous arrays (ndarray) ! "Universal functions (ufunc) ! "basic math, linear algebra, FFT, PRNGs ! "Simple data file I/O ! "text, raw binary, native binary ! "Tools for integrating with C/C++/Fortran ! "Heavy lifting done by optimized C/Fortran libraries ! "ATLAS or MKL, UMFPACK, FFTW, etc... Folie 24 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 25. NumPy Arrays ! "A NumPy array is an N-dimensional homogeneous collection of “items” of the same “kind”. The kind can be any arbitrary structure and is specified using the data-type. ! "Example: Multidimensional NumPy array >>> a = array([[ 0, 1, 2, 3], [10,11,12,13]]) >>> a array([[ 0, 1, 2, 3], [10,11,12,13]]) >>> a[1,3] 13 >>> a[1,3] = -1 >>> a array([[ 0, 1, 2, 3], [10,11,12,-1]]) Folie 25 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 26. SciPy Scientific Tools for Python ! "Website: http://www.scipy.org ! "Large library of scientific algorithms ! "Extends NumPy with many tools for science and engineering ! "Computationally intensive routines implemented in C and Fortran Folie 26 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 27. SciPy Packages ! "Special Functions ! "Signal Processing ! "Fourier Transforms ! "Optimization ! "Numerical Integration ! "Linear Algebra ! Input/Output ! "Statistics ! "Fast Execution ! "Clustering Algorithms ! "Sparse Matrices Folie 27 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 28. Parallel Programming Threading ! "Useful for certain concurrency issues, not usable for parallel computing due to Global Interpreter Lock (GIL) subprocess ! "Relatively low level control for spawning and managing processes ! "multiprocessing - multiple Python instances (processes) ! "basic, clean multiple process parallelism MPI ! "mpi4py exposes your full local MPI API within Python ! "As scalable as your local MPI GPU (OpenCL & CUDA) ! "PyOpenCL and PyCUDA provide low and high level abstraction for highly parallel computations on GPUs Folie 28 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 29. mpi4py ! "Website: http://mpi4py.scipy.org/ ! "Wraps native MPI implementations ! "Prefers MPI2, but can work with MPI1 ! "Works best with NumPy data types, but can pass around any serializable object ! "Provides all MPI2 features ! "Well maintained ! "Distributed with Enthought Python Distribution (EPD) ! "Requires NumPy ! "Portable and scalable Folie 29 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 30. How mpi4py works... ! "mpi4py jobs must be launched with mpirun ! "Each rank launches its own independent python interpreter ! "Each interpreter only has access to files and libraries available locally to it, unless distributed to the ranks ! "Communication is handled by MPI layer ! "Any function outside of an if block specifying a rank is assumed to be global ! "Any limitations of your local MPI are present in mpi4py Folie 30 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 31. from mpi4py import MPI Example: import numpy as np Calculating " import random with mpi4py comm = MPI.COMM_WORLD rank = comm.Get_rank() mpisize = comm.Get_size() nsamples = int(12e6/mpisize) inside = 0 random.seed(rank) for i in range(nsamples): x = random.random() y = random.random() if (x*x)+(y*y)<1: inside += 1 mypi = (4.0 * inside)/nsamples pi = comm.reduce(mypi, op=MPI.SUM, root=0) if rank==0: print (1.0 / mpisize)*pi Folie 31 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 32. GPGPU General-purpose computing on graphics processing units Architecture ! "Many cores per node ! "Good for stream processing (independent vertices, many of them in parallel) CUDA ! "NVIDIA’s platform for C programming on GPGPUs ! "Python binding: PyCUDA OpenCL ! "Framework for writing programs that execute across heterogeneous platforms consisting of CPUs, GPUs, and other processors ! "Open standard ! "Python binding: PyOpenCL Folie 32 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 33. PyCUDA ! "Website: http://mathema.tician.de/software/pycuda ! "PyCUDA lets you access NVIDIA‘s CUDA parallel computation API from Python ! "Integration with NumPy ! "All CUDA errors are automatically translated into Python exceptions Folie 33 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 34. PyOpenCL ! "Website: http://mathema.tician.de/software/pyopencl ! "Convenient access to full OpenCL API from Python ! "OpenCL errors translated into Python exceptions ! "Object cleanup tied to object lifetime (follows Resource Acquisition Is Initialization) ! "NumPy ndarrays interact easily with OpenCL buffers Folie 34 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 35. OpenCL Computing Model Buffers ! "used for holding data, shipping data between devices Kernels ! "functions which will operate on buffers in parallel Queue ! "control sequence of operations: transferring buffers, running kernels, waiting on events, etc. Folie 35 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 36. " with PyOpenCL import pyopencl as cl import pyopencl.clrandom import numpy as np nsamples = int(12e6) # set up context and queue ctx = cl.create_some_context() queue = cl.CommandQueue(ctx) # create array of random values in OpenCL xy = pyopencl.clrandom.rand(ctx,queue,(nsamples,2),np.float32) # square values in OpenCL xy = xy**2 # 'get' method on xy is used to get array from OpenCL into ndarray print 4.0*np.sum(np.sum(xy.get(),1)<1)/nsamples Folie 36 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 37. Copperhead: Data-Parallelism Embedded in Python Slides by Michael Garland (NVIDIA Research) http://bit.ly/jK7nsM Folie 37 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 38. Debugging and Profiling Folie 38 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 39. Debugging Python and C ! "Python comes with gdb-like command line debugger pdb ! "Helps in debugging serial Python applications ! "C-extensions can be debugged with standard debuggers ! "Parallel debugging ! Totalview can be used for debugging C-functions ! "Custom interpreter provides easy initial breakpoint Folie 39 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 40. Profiling Mixed Python-C code ! "Finding performance bottlenecks is critical to scalability on HPC platforms ! "Number of profiling tools available: ! gprof, CrayPAT, scalasca - C, Fortran ! "can be used for C-extensions ! "import profile – Python ! "TAU Performance System, http://www.cs.uoregon.edu/research/tau/home.php ! "Excurse: Slides by Sameer Shende (University of Oregon) Folie 40 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 41. TAU Performance System Slides by Sameer Shende (University of Oregon) http://slidesha.re/lsDZSj Folie 41 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 42. Example Applications Stand # Folie 42 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 43. Example: GIST Maxwell’s Equation Solver (GMES) ! "Object-Oriented Implementation of the Finite-Difference Time-Domain Method in Parallel Computing Environment ! "Slides by Kyungwon Chun et al. (GIST - Gwagnju Institute of Science and Technology, South Korea)! http://bit.ly/mCs8W9 Folie 43 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 44. Example: FlowSimulator – A Python-controlled framework to unify massive parallel CFD workflows Folie 44 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 45. What does CFD mean? ! "CFD = Computational fluid dynamics means to ! " solve the compressible flow equations to ! " predict an aircrafts behavior and to ! " optimize its features. ! "Numerical approach: ! " Spatial discretization ! " Simplified models for turbulent effects ! " Temporal discretization (to support unsteady flows) Folie 45 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 46. Evolution of CFD Classical chain Z ! "Simple chain Y X ! " Geometry ! " Mesh generator ! " Pre-processor ! " Flow solver ! " Post-processor ! " Visualization ! "Tools ! " stand-alone mesher ! " stand-alone solver (with pre-/postprocessor) ! " shell scripts (e.g. different solver parameters) Folie 46 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 47. Evolution of CFD Solver-adaptation-loop Z ! "Chain with solver-adaptation-loop Y X ! " several consecutive solver steps ! " adaptation of mesh in between ! "Tools: ! " stand-alone mesher ! " stand-alone solver ! " stand-alone adaptation ! " shell script control ! "Solver/adaptation as Python module Folie 47 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 48. Evolution of CFD Multi-disciplinary simulations ! "Multi-disciplinary chain Z Y X ! " output of solver is input for ! " structural mechanics (deformation) ! " flight mechanics (control surface movement) ! " output of structural/flight mechanics modifies mesh ! "Tools: ! " solver as Python module ! " stand-alone CSM ! " Python-based flight mechanics Folie 48 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 49. Structure and Flight Mechanics Folie 49 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 50. FlowSimulator Environment Single Python script controls the whole workflow FSControl Out of scope Flight CFD CSM CAA mechanics Import Export FSDataManager Folie 50 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 51. FSControl ! "Pure Python library ! "Set of simple Python classes for different modules/tools: ! "simple interface: Import, Run, Export ! "data stored in FSDataManager FSDataManager (FSDM) ! "C++ library, wrapped to Python ! "FSDataManager is a real Python module! ! "Parallelization context ! "all data structures “distributed” ! "facilitated transfers (Gather, Distribute, Send, !) Folie 51 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 52. Python Framework for Coupled Fusion Simulations Slides by Samantha Foley et al. (ORNL – Oak Ridge National Laboratory, United States) http://slidesha.re/iLuegW Folie 52 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 53. Python & HPC Communities EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 54. Where the Python and HPC Communities Meet! ! "SciPy ! "Python for Scientific Computing Conference (since 2008) ! "EuroSciPy ! "Annual European Conference for Scientists using Python (since 2008) ! "SCxx ! "The International Conference for High Performance Computing, Networking, Storage, and Analysis (annual conference since 1988) ! "Tutorial and BoF session on Python (since 2009) Folie 54 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 55. The SC Conference Series (SCxx) Tutorials (est. 2009) ! "Python for Scientific and High Performance Computing ! "Given by William R. Scullin (ANL – Argonne National Lab, United States) et al. ! "Content ! "Python language and interpreter basics ! "Popular modules and packages for scientific applications ! "How to improve performance in Python programs ! "How to visualize and share data using Python ! "Where to find documentation and resources Folie 55 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 56. The SC Conference Series (SCxx) Birds-of-a-Feather Sessions, est. 2009 Folie 56 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 57. Credits Credits Achim Basermann (DLR) Kyungwon Chun (GIST) Samantha Foley (ORNL) Michael Garland (NVIDIA) Michael Meinel (DLR) Tr a v i s O l i p h a n t ( E n t h o u g h t ) William R. Scullin (ANL) Sameer Shende (U Oregon) Folie 57 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 58. Questions? Summary •" Python used in HPC applications •" Good tools available (SciPy etc.) •" Wrapping of C/C++/Fortran is common Andreas Schreiber A n d r e a s . S c h r e i b e r @ d l r. d e http://www.dlr.de/sc Folie 58 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011
  • 59. Contact python@dlr.de @onyame Andreas Schreiber A n d r e a s . S c h r e i b e r @ d l r. d e http://www.dlr.de/sc Folie 59 EuroPython 2011 > Andreas Schreiber > Python for High Performance and Scientific Computing > 23.06.2011