Cyberinfrastructure in Louisiana: From Black Holes to Hurricanes. Presentation at Cyberinfrastructure Days, Notre Dame, April 29-30, 2010. http://ci.nd.edu/

1. Cyberinfrastructure in Louisiana: From Black Holes to Hurricanes Gabrielle Allen Department of Computer Science Center for Computation & Technology Louisiana State University http://www.cct.lsu.edu/~gallen

2. Complex Problem SolvingComputation as Third Pillar of Scientific Investigation Real world applications Communities No single group, university, or state can do these problems Must integrate CS, Math, Bio, Sensors, Engineering, more... Data everywhere Supercomputers generate petabytes 2

3. 3 LIGO: Laser Interferometric Gravitational Wave Observatory Ties together theory, computation, and experiment Gravitational Wave Astronomy

4. 4 Louisiana Coastal Area Rich dynamic environment for modeling: coupled models, multi-scale, realtime data (sensors, satellites) Models, Data, Grids for … Hurricane forecasts Emergency preparedness Wetland reconstruction Ecological studies and fish populations Oilspillbehaviour Levee design Rescue Hypoxia “Dead Zone” Algae blooms 24/7/365 shipping forecasts

5. Hurricane Gustav

6. 6 Highly scalable algorithms (Petascale, exascale computing, heterogeneous architectures) Storage & networks: provenance, metadata Visualization (Large data, interactive, remote, AMR) Software engineering (code generation, verification & validation, interfaces, interoperability, etc) Workflows, Grid & distributed computing Education Research Issues

7. PITAC Report Summary (June 2005): 7 “Computational science -- the use of advanced computing capabilities to understand and solve complex problems -- is critical to scientific leadership, economic competitiveness, and national security. It is one of the most important technical fields of the 21st century because it is essential to advances throughout society.” “Universities must significantly change organizational structures: multidisciplinary & collaborative research are needed [for US] to remain competitive in global science” Complex problems: Innovations will occur at boundaries

8. Louisiana Response: Center for Computation & Technology (CCT) 8 State commitment for IT in 2002 $25M/year for Vision 20/20, $9M to LSU University commitment to build new programs Opportunity to build new world class program in interdisciplinary research & education Ed Seidel recruited to LSU, created & implemented vision for state-wide collaboration Center for Computation & Technology (2003) Louisiana Optical Network Initiative (2005) CyberTools (2007), LONI Institute (2007) Multidisciplinary Hiring Initiatives (2007)

10. Growth of CCT September 2003 May 2008

11. CCT Focus Areas: Faculty Led Areas chosen to bring together interdisciplinary researchers towards common theme: Faculty, Research Staff, Postdocs, Students System Science & Engineering / Core Computing Science / Coast to Cosmos / Material World / Cultural Computing

12. Recruiting Faculty Joint appointments: 50% CCT, 50% home department Tenure resides in department, but CCT has input in P&T Reduced teaching load to accommodate higher level service Also associate faculty from campus

13. Recruiting Faculty Joint appointments: 50% CCT, 50% home department Tenure resides in department, but CCT has input in P&T Reduced teaching load to accommodate higher level service Also associate faculty from campus Currently recruiting faculty in digital media and arts

14. External Funding: Increasingly able to respond to large solicitations

16. Hardware

17. Operations

18. Policies16 16 CyberinfrastructureDevelopment

19. CI in Louisiana TeraGrid LONI Institute: People and Collaborations Cybertools: Tools and Services LA Tech LONI: ~100TF IBM, Dell Supercomputers LSU LONI: 40 Gbpsnetwork SUBR UNO National Lambda Rail Tulane UL-L 17

20. 18 State initiative ($50M) to support research (2004): 40 Gbps optical network + NLR Connects stateuniversities, health science centers Compute resources: ~100 Tflops across state Data resources ~500TB with NSF PetaShare LONI customers: MS universities, K12, hospitals, LPB Louisiana Optical Network (LONI)

21. LONI Compute Resources

22. Support: HPC@LSU Partnership between LSU CCT and LSU ITS to support and maintain LSU HPC resources and provide user training HPC@LSU also supports/maintains LONI, through support from Louisiana BoR HPC@LSU also supports TeraGrid, through support from NSF 20

23. NSF EPSCOR RIICyberTools Project: Enabler and Driver 21 $12M from NSF/BOR, 2007-2010, 9 research institutions in Louisiana

24. Cybertools Infrastructure 22 WP1: Scheduling and Data Services Infrastructure deployment, high availability, scheduling, data archiving and retrieval, metadata WP2: Information Services and Portals Information services (infrastructure, apps, experiments), application interfaces for scientists, portals for information gathering WP3: Visualization services Data/viz integration, HD streaming viz, advanced viz facilities, integration with application toolkits WP4: Application toolkits Toolkits to support similation codes (CFD, MD, other), application managers, SAGA interfaces

25. Cybertools Infrastructure 23 WP1: Scheduling and Data Services Infrastructure deployment, high availability, scheduling, data archiving and retrieval, metadata WP2: Information Services and Portals Information services (infrastructure, apps, experiments), application interfaces for scientists, portals for information gathering WP3: Visualization services Data/viz integration, HD streaming viz, advanced viz facilities, integration with application toolkits WP4: Application toolkits Toolkits to support similation codes (CFD, MD, other), application managers, SAGA interfaces Each Workpackage has state-wide collaboration meetings bi-weekly Graduate students have interdisciplinary committees Working with LONI staff to support state software Cyberinfrastructure software developed in collaboration with science drivers

28. Can already reserve:

29. Time on supercomputers (advance reservation), and

30. Dedicated paths on GMPLS-based networks with simple topologies

31. Uses Paxos Commit to atomically reserve multiple resources, while providing a highly-available service

32. Used to coordinate bookings across EnLIGHTened and G-lambda testbedsin largest demonstration of its kind to date

33. Used for setting up the network for Thomas Sterling’s HPC Class which goes out live in HDCredit: Jon MacLaren 25 HARC: Highly Available Resource Co-scheduler

34. Vish

35. Above left: co-located and distributed users collaboratively manipulate a 3D visualization in an AccessGrid meeting using viz tangibles interaction devices Right: an “interaction tray” is used together with RFID-tagged “tangible menus” to access and manipulate scientific visualizations 27 Credit: Brygg Ulmer Viz Tangibles

36. Simple API for Grid Applications

37. Cactus Framework

40. Structured, multi-block, unstructured

41. Uses PETSc, Trilinos, MUMPS, HYPRE30

42. LONI Institute $15M 5-year project (2007-2012) $7M BoR, $8M from universities Create bold new inter-university superstructure New faculty (12), staff (6), students (36); train others. Focus on CS, Bio, Materials, but all disciplines impacted Promote research at interfaces for innovation Draw on, enhance strengths of all universities Solve complex problems through collaboration & computation Much stronger recruiting opportunities for all institutions Statewide interdisciplinary education & research program Create University-Industry Research Centers (UIRCs) Transform our state Such committed cooperation between sites extraordinary 31

43. NSF TeraGrid 32

44. Motivation:Gamma-Ray Burst Grand Challenge Most energetic events in the universe Mechanism still a riddle;grand challenge in astrophysics Modelling requires expertise in many fields of physics(general relativity, magneto-hydrodynamics, neutrinos, ...) Requires petascale computing ACM, doi:10.1145/1341811.1341831

45. Einstein Toolkithttp://www.einsteintoolkit.org Based on Cactus Framework Over 130 open, community developed Cactus modules Building a consortium of users Governance and software development Members 40 listed on web page 10 different groups US, Japan, Mexico, Spain, Germany, Canada 300 science publications, 50 student theses

46. Einstein Toolkit Research/Funding Relativistic astrophysics (black holes, supernova, Gamma Ray bursts) (PIF, ++) Large scale computing (PetaApps, PRAC, Blue Waters) Verification, performance (Alpaca, HPCOPS) Data and metadata (PIF) Visualization, interaction, high speed networks (CyberTools, EAGER) Other apps: Coastal, CFD (CyberTools)

48. 8 node renderer at LSU

49. Remote LONI nodes as data servers

50. 10243 spatial resolution (1GByte/timestep)

51. 20 timesteps cached remotely

53. References Louisiana: A Model for Advancing Regional e-Research through Cyberinfrastructure, D. S. Katz, G. Allen, R. Cortez, C. Cruz-Neira, R. Gottumukkala, Z. D. Green- wood, L. Guice, S. Jha, R. Kolluru, T. Kosar, L. Leger, H. Liu, C. McMahon, J. Nabrzyski, B. Rodriguez-Milla, E. Seidel, G. Speyrer, M. Stubblefield, B. Voss, S. Whittenburg, Phil. Trans. R. Soc. A 28, June 2009, vol. 367, no. 1897, 2459-2469, (2009). Computational Science, Infrastructure and Interdisciplinary Research on University Campuses: Experiences and Lessons from the Center for Computation & Technology, G. Allen, D. S. Katz,, CCT Technical Report Series, CCT-TR-2010-1, 2010 (Full version soon) Einstein Toolkit Consortium www.einsteintoolkit.org Cactus Framework www.cactuscode.org