Talk for the plant science community on how the numerical relativity built a community around shared software.

1. Education & Community
Building (in Astrophysics)
Gabrielle Allen
University of Illinois

2. Gravitational Wave Physics
Instruments (LIGO)
Models & Simulation
Theory
Scientific Discovery!
Gmn = 8p Tmn
Colliding black holes & neutron
stars, supernovae collapse,
gamma-ray bursts, big bang, …

3. Gravitational Wave Physics
Instruments
Models & Simulation
Theory
Scientific Discovery!
Gmn = 8p Tmn
Colliding black holes & neutron
stars, supernovae collapse,
gamma-ray bursts, big bang, …

4. Complex Problems
• Multiscale, multiphysics, data-
driven
• E.g. To model neutron stars
need general relativity,
magneto-hydrodyamics,
neutrino & radiation transport,
complex equations of state,
chemical reactions all
integrated together
• Need simple but effective
interfaces that can be
implemented in software
Schnetter et al, PetaScale Computing:
Algorithms and Applications, 2007

5. Cactus Framework (1997 - )
www.CactusCode.org
• Open source component framework for HPC
• Modular system with high level abstractions
– Components (“thorns”) defined by parameters,
variables, methods
– Cactus “flesh” glues components together
– Cactus Computational Toolkit: general thorns
• Multiple application areas develop toolkits
– Numerical relativity, CFD, coastal science,
petroleum, quantum gravity, cosmology, …

6. Key Features
 Cactus framework provides scheduling, application APIs
for parallel operations
 Driver thorn provides load balancing, parallelization
 Application thorns deal only with local part of parallel
mesh
 Different thorns (with same interface) can be used to
provide the same functionality, easily swapped.

7. Thorn (Components)
• Configuration files (CCL files) define interface
of thorns with the Flesh and other thorns
– Implementation name and inheritance relations
– Variables
– Runtime parameters
– Scheduled methods and storage, synchronization
– Any configuration details
• Public (global) and private (internal)

8. Cactus Thorns
8
Core “Flesh”
Plug-In “Thorns”
(components)
driver
input/output
interpolation
Elliptic solvers
coordinates
boundary conditions
black holes
equations of state
remote steering
wave evolvers
multigrid
parameters
grid variables
error handling
scheduling
extensible APIs
make system
ANSI C
Coastal science
Your Physics !!
Quantum gravity
Maxwells equations
Neutron stars

9. Example scheduling tree

10. Building a Computational
Numerical Relativity Community
• Cactus came from the relativity community (USA GC)
• European project with 10 sites developed community
open code base
• Each group had different expertise
• Cactus allowed developing shared interfaces/standards
• Easy to add a component, share components
• Supports both collaboration and competition
EU Network for Gravitational Wave Sources: 2001

11. Today: Einstein Toolkit
• Software
– Around 200 Cactus components
– 3100 files, 1M LOC
– Tools for analysis and visualization
• Shared interfaces, tests, variables
• Einstein Toolkit Consortium
– Einstein Toolkit Maintainers and Members
– Development plan with 6 month release schedule
– Weekly open call, planning/bug fixes
• Examples and tutorials
– Complete production codes for black holes, neutron stars,
cosmology, gravitational waves
• Community support: active mail list and ticket system
http://www.einsteintoolkit.o
rg

12. Einstein Toolkit Consortium
Over 100 members, 56 groups, 20 countries

13. Community Building

14. Distributed & Inclusive
Hayley McPherson, PhD student, U. Melbourne
One new module added – new
code for relativistic cosmology -
-- single, isolated grad student

15. www.eu-network.org
Sources of Gravitational Radiation
EU Astrophysics
Network Overview
Ed Seidel
Albert Einstein Institute
Principal Network Coordinator
2001

16. Ed Seidel
Albert Einstein Institute
www.eu-network.org
Sources of Gravitational Radiation
Scientific and technological reasons
for carrying out research in our field
 Exploring Einstein’s General Relativity
Want to develop theoretical lab to probe this fundamental theory
 Fundamental theory of Physics (Gravity)
 Among most complex equations of physics
 Predict black holes, gravity waves, want much more
 Exciting new field: Gravitational Wave Astronomy
 LIGO, VIRGO, GEO, LISA, … ~ $1 Billion worldwide!
 Fundamentally new information about Universe
 A last major test of Einstein’s theory: do GWs exist?
A century later, both of these developments
happening at the same time: very exciting
coincidence!
 But, the community needed to carry out theoretical
work is lacking…
Not enough people or training

17. Ed Seidel
Albert Einstein Institute
www.eu-network.org
Sources of Gravitational Radiation
This EU Network
Astrophysics
10 EU
Institutions, 3
years, €1.5M
Continue these
problems
Entire Community
becoming Grid
enabled
 NSF Black Hole
Grand Challenge
 8 US
Institutions, 5
years, $4M
 Solve problem of
colliding black
holes (try…)
NASA Neutron Star
Grand Challenge
5 US Institutions,
3years, $1.4M
Solve problem of
colliding neutron
stars (try…)
NSF ASC
Project
5 US
Institutions, 3
years, $2.2M
Develop
“Collaboratory
”
EU GridLab
10 EU
Institutions, 3
years, €5M
Develop Grid
Tech
For these
projects
Grand Challenge Collaboratories
Building Communities to Solve These Problems

18. Ed Seidel
Albert Einstein Institute
www.eu-network.org
Sources of Gravitational Radiation
Network research objectives
 The formation of a close alliance among the different
expert groups to solve urgent problems required for
GWA, too large and complex for any single group.
 The development & training of a young community of
researchers for this emerging research area of GWA.
 The development of a community simulation code for
relativistic astrophysics
 The application of these numerical and approximation
tools to a set of core astrophysics problems
 Good focus problems for training a community

19. Ed Seidel
Albert Einstein Institute
www.eu-network.org
Sources of Gravitational Radiation
Theoretical Tools
Einstein Eqs
Pert. theory
Post-Newtonian
TheoryEnablers
Focus
Physics
Areas
Outreach
/Collab
GridLab
ZIB/Garching
LBL/NCSA/ANL NCSA/WashU
Cactus
Community
NSF KDI
Computational Tools
Cactus
Grid
Viz
KDI
3D BH Collisions Other (strange stars,
Core collapse, etc)
3D NS processes and
mergers
World
General View of Network

20. Ed Seidel
Albert Einstein Institute
www.eu-network.org
Sources of Gravitational Radiation
Our Team
7 Focus areas, links created, strengthened by Network
Valencia
Ports
SOTON
Meudon
Jena
Palma
Trieste
AUTH
Rome
WashU
Potsdam
Jena
Cactus Dev/Training
CS Efforts
Worldwide
PotsdamPorts
SOTON
Valencia
Rome
Trieste
AUTH
BH Data/Evolutions
Potsdam
Nonlinear GR Hydro
And Applications
Valencia
Trieste
Meudon
Potsdam
Characteristic Codes
SOTON
Meudon
Palma
Rome
AUTH
Jena
SOTON
Ports
Palma
Stellar Pert Theory
Valencia
Jena
AUTH
Palma
Perturbative Time Evolutions
Valencia
JenaJena
Post-Newtonian Schemes
Ports
Palma
Rome
Valencia

21. Ed Seidel
Albert Einstein Institute
www.eu-network.org
Sources of Gravitational Radiation
Project View: Code Leverage
NS Studies
Meudon BBH Meudon BNS
Hydra
Projects extremely
Collaborative
Build on each other
Share Modules or Build
them together
Cactus an important
collaborative technology
EE’s
MOL
Horizon Finders
Wave Extractors
Lazarus
Elliptics
Parallel I/O
Gauges
AMR
EE’s
MOL
Horizon Finders
Wave Extractors
Lazarus
Elliptics
Parallel I/O
Gauges
AMR
EE’s Community
Workshop
this spring/fall
improves EE’s,
all simulations
Vacuum BH Studies

22. Ed Seidel
Albert Einstein Institute
www.eu-network.org
Sources of Gravitational Radiation
Developing a Community:
People Leverage
 Actual Network funding rather small
 10-12 positions (some positions converted to 2 for shorter
times)
– 6-7 postdocs, 4-5 predocs
 Very Strong Leverage! 6:1 ratio of paid/volunteer
effort
 10 coordinators
 65 others officially working on the project at least part-time
 Other sites in Europe joining in
– Is it possible to get additional funding to help them?
 Other projects in US and EU adding value
 Attending this meeting
 ~65!!
 This project has seeded a much larger effort
 Now
 Even more later

23. Ed Seidel
Albert Einstein Institute
www.eu-network.org
Sources of Gravitational Radiation
Community Code: Advantages
 Sharing of Expertise: No single group can do
 Sharing of Code among projects
 Infrastructure
 BH routines apply directly to NS work, etc…
 But free to keep routines in group as long as desired
 Better Code
 Open source encourages people to be more careful in coding!
 Encourages documentation
 Encourages deeper thinking about how it interfaces to another
code
 More trusted code, results
 When code becomes open, and people can run it for themselves,
they will begin to believe the results
 Improvements propagate quickly though community
 Not well accepted yet, but we are starting a good trend…

24. Ed Seidel
Albert Einstein Institute
www.eu-network.org
Sources of Gravitational Radiation
Work Plan
Cactus
Nonlinear BH
simulations
Hydrodynamics
Post-Newtonian
theory
Stellar
perturbation
theory
Perturbative time
evolutions
Characteristic
Methods
Training complete,
AMR tested
12 months 24 months 36 months
Remote/distributed
Simulations, input
from projects below
Addressing Efficacy
Of Characteristic
Hydro
Full community
Code release, with
documentation
Merger simulations
with studies of waves
Continually refined as techniques developed
Completed binary
NS Initial
Data Module
Simulations refined as
techniques developed
Accretion
Code
PN GW, reaction
expressions
Post-N.Hydro Code,
Post-N Initial Data
Module
Simulations
continue
Pert Eqs. for rotating
NS, nonlinear results
Modes computed, valuable
input to num. simulations
Close limit module,
time evolution for
rotating NSs
Perturbative evolutions,
Input to num. simulations
Time

25. Ed Seidel
Albert Einstein Institute
www.eu-network.org
Sources of Gravitational Radiation
The training programme
 Feedback on talks, practice talks, etc
 Review Talk topics selected by YR’s
 Schools planned
 Training sessions, extensive online tutorials
 High performance computing,
 Cactus (our code framework)
 Visualization tools,
 Code maintenance systems (CVS), etc
 Developing “soft skills”
 Collaboration/Communication built in to this project
 Helping to prepare talks, documentation
 YRs have to ask first questions at meetings

26. Lessons
• Define simple test problems within the context of
the motivating challenge problems.
• Strong and persistent coordination for both
technical and strategic goals.
• Focus around interfaces (of different types),
science domain interfaces were critical.
• Focus on students (a 20 year activity), organize &
track exchanges, summer schools/training.
• Partner with computer scientists, look for
opportunities to share code/tools with other
domains.
• Advisory committee to help keep all on track.

27. Crops in Silico
Single
Plant
Weather
Climate
Crops
Gene
Regulatory
Photosynthesis
Field Region EarthPlantLeafCellMolecule
Lengthscale
Timescale
Decades
Year
Week
Hydrology
Capillary
Fluid
Flow
Soil
Minute
ms
Roots

28. Model Interfaces
• What is the set of challenge science
problems?
• What are the key abstract models needed
to solve these?
• What are the core input and output
variables as well as parameters for each
abstract model?
• What models/tools do we already have to
build from?