Presentation on using Apache Airavata to build science gateways at the International Workshop on Science Gateways (IWSG) 2014 in Dublin, Ireland.

1. Developing Computational Science
Gateways using Apache Airavata
Marlon Pierce
Science Gateway Group, Indiana University
marpierc@iu.edu

2. 1. We Are Hiring
Apply: http://goo.gl/LbiXNO

3. http://www.catonmat.net/blog/what-would-john-mccarthy-say-if-he-saw-you-programming/
2. dev-subscribe@airavata.apache.org

4. 3. We’ll get to this one later….
http://www.bayoubelles.com/2011/08/night-before-big-day-was-not-what-was.html

5. Airavata’s Philosophy
• There a lots of ways to build Web interfaces for
Science Gateways.
– By Hand: PHP, Twitter Bootstrap, AngularJS, …
– Turnkey Frameworks: Liferay, Drupal, Plone, Joomla, …
– Science Gateway Frameworks: the SDSC Workbench,
HUBzero, gUse/WS-PGrade
• Gateway developers should concentrate on
building interfaces that serve their community.
• And outsource the general purpose services to
Airavata.

6. Airavata Role and Goals: Improve sustainability by
converging on a single set of hosted infrastructure services

7. Apache Airavata Components
Component Description
Airavata API
Server
Apache Thrift-generated server skeletons of the API
and data models; directs traffic to appropriate
components
Registry Insert and access application, host machine,
workflow, and provenance data.
Orchestrator Handles experiment execution request validation,
scheduling, and decision making; selects GFAC
instances that can fulfill a given request
GFAC Manages the execution and monitoring of an
individual application.
Workflow
Interpreter
Execute the workflow on one or more resources.
Messaging
System
WS-Notification and WS-Eventing compliant
publish/subscribe messaging system for workflow
events

8. Airavata Components

9. Airavata’s Future Directions: SciGaP
• SciGaP: Airavata as a multi-tenanted
Gateway Platform as a Service
• Goal: We run Airavata so you don’t have
to.
– Scalable support
• Challenges:
–Centralize system state
–Make Airavata more cloud friendly, elastic
http://scigap.org

10. Some Contribution Opportunities
Component Research Opportunities
Registry Better support for Thrift-generated objects; NoSQL and
other backend data stores; fault tolerance
Orchestrator Pluggable scheduling; load balancing and elasticity
GFAC ZooKeeper-like strategies for configuring and managing.
Messenger Investigate AMQP, Kafka, and other newer messaging
systems
Workflow
Interpreter
Alternative workflow processing engines.
Overall Message-based rather than direct CPI calls.
Airavata components expose Component Programming Interfaces (CPIs) that
allow you to switch out implementations. GFAC is also designed to be
pluggable.

11. Airavata and the Apache
Software Foundation
Experiences with Open Governance

12. Cyberinfrastructure: How open is
open source?
• What’s missing?
– Open source licensing
– Open Standards
– Open Code (GitHub,
SourceForge, Google
Code, etc)
We also need Open Governance

13. Open Source Software and Governance
• Open source projects need diversity,
governance.
– Sustainability
• Incentives for projects to diversify
their developer base.
• Govern
– Software releases
– Contributions
– Credit sharing.
– Members are added
– Project direction decisions.
– IP, legal issues
• Our approach: Apache Software
Foundation
Collaborate
Compete

14. “Apache” Means “Open”
Join the Airavata developer or architecture mailing
lists, get involved, submit patches, contribute.
Use Give Back

15. More Information
• Contact Us:
– marpierc@iu.edu, smarru@iu.edu
– Join dev@airavata.apache.org,
users@airavata.apache.org,
architecture@airavata.apache.org
• Websites:
– Apache Airavata: http://airavata.apache.org
– SciGaP Project: http://scigap.org
– Science Gateway Institute: http://sciencegateways.org

16. Airavata Audiences
Audience Description
Gateway
Developers
Use the Airavata API through and SDK
in their favorite programming
language.
Airavata
Developers
Want to change Airavata components,
experiment with different
implementations.
Middleware
Developers
Want to extend Airavata to talk to
their middleware clients.
Resource Providers Want to configure Airavata to work
with their middleware.

17. Apache Contributions Aren’t Just
Software
• Apache committers and PMC members aren’t just
code writers.
• Successful communities also include
– Important users
– Project evangelists
– Content providers: documentation, tutorials
– Testers, requirements providers, architects, and
constructive complainers
• Using Jira and mailing lists
– Anything else that needs doing.

18. Getting Involved, Contributing Back
• Airavata is open source,
open community software.
• Open Community: you can
contribute back
– Patches, suggestions, wiki
documentation, etc
• We reward contributors
– Committers: write access
to master Git repo
– Project Management
Committee members: full,
binding voting rights

19. Airavata Audiences
Audience Description
Gateway
Developers
Use the Airavata API through and SDK
in their favorite programming
language.
Airavata
Developers
Want to change Airavata components,
experiment with different
implementations.
Middleware
Developers
Want to extend Airavata to talk to
their middleware clients.
Resource Providers Want to configure Airavata to work
with their middleware.

20. The Apache Software Foundation
• Apache software powers
65% of web sites worldwide
• 501(c)3 non-profit
foundation
• Reasons for creating ASF
– Create legal entity
– Protect contributors from
liability
– Protect Apache assets
• Membership: individual
• Apache Incubator
• Governance and Staffing
– Board of Directors
– Project Management
Committees
– ASF Members
– Committers
– Contributors
• Funding
– All-volunteer
staffing/development
resources
– Donations
– Corporate investment

21. Where Is Airavata 1.0?
• Airavata 1.0 will be
the stable version of
the API.
• Version 1.0
determined by
developer
community vote.
• Semantic versioning

22. Airavata Thrift Definitions
airavata/airavata-api/thrift-interface-descriptions
Thrift File Contents
airavataAPI Definitions of services; top level file for other
Thrift files.
airavataErrors Definitions of exceptions thrown by the API
services.
airavataDataModel Top level container for data models.
workspaceModel Definitions of the structs (Project, User,
Gateway, Group); includes experimentModel
experimentModel Definitions of data structures that constitute the
experimentModel.
applicationCatalogAPI App catalog service definitions.
applicationCatalogDataModel App catalog data model definitions.

23. Airavata API and Apache Thrift
• We use Apache Thrift to define the API.
• Advantages of Thrift
– Supports well-defined, typed messages.
– Custom defined errors, exceptions
– Generators for many different programming
languages.
– Some shielding from API versioning problems.
• Downsides of Thrift
– No message headers, so everything must be explicitly
defined in the API.

24. A Few Observations on Successful
Gateways
• Support familiar community applications.
• Make HPC systems easy for new user
communities who need HPC.
• Keep it simple.
• Have champions who build and support the
community.
• Have a lot of common features.

25. Airavata API and Apache Thrift
• We use Apache Thrift to define the API.
– TCP/IP typically
– Not WS, not REST
• Advantages of Thrift
– Supports well-defined, typed messages.
– Custom defined errors, exceptions
– Generators for many different programming languages.
– Some shielding from API versioning problems.
• Downsides of Thrift
– No message headers, so everything must be explicitly
defined in the API.

26. Airavata API Thrift Definitions
airavata/airavata-api/thrift-interface-descriptions
Thrift File Contents
airavataAPI Definitions of services; top level file for other
Thrift files.
airavataErrors Definitions of exceptions thrown by the API
services.
airavataDataModel Top level container for data models.
workspaceModel Definitions of the structs (Project, User,
Gateway, Group); includes experimentModel
experimentModel Definitions of data structures that constitute the
experimentModel.
applicationCatalogAPI App catalog service definitions.
applicationCatalogDataModel App catalog data model definitions.