1.
Apache Systems Projects in the Real World<br />Srinath Perera Ph.D. <br />Senior Software Architect, WSO2 Inc.<br />Member, Apache Software Foundation<br />Visiting Faculty, University of Moratuwa<br />Research Scientist, Lanka Software Foundation<br />

2.
Goals of this Talk<br />Intro to Apache and Opensource<br />Describe a large Scale E-Science Project build on Apache Technology and some open problems. <br />Apache Airavata<br />Discuss “should your project move to Apache?”<br />photo by John Trainoron Flickr http://www.flickr.com/photos/trainor/2902023575/, Licensed under CC<br />

3.
Open Source<br />Basic definition is code accessible to everyone. <br />Yes, you can write something and make it opensource.<br />But Community is one of the key aspects. <br />Often build by volunteers (at least not payed by the project)<br />Does serious Crowdsourcing <br />Ideally, Code contributions , governance, and decision model all open and decentralized. <br />Not all opensource projects are equal (different license) <br />GPL License – Linux etc., you have to contribute back changes <br />Apache License – Commercial friendly <br />Copyright digitalART2 and licensed for reuse under CC License , http://www.flickr.com/photos/digitalart/2101765353/<br />

4.
How does a Opensource Work?<br />Open code repository (SVN or Git etc.)<br />Two parts of the community <br />Developer Community<br />User Community<br />Communication through Mailing lists / IRC Channel<br />Develop mailing list<br />User mailing list <br />Bug tracking database to track errors (Jira, Bugzilla)<br />People submit improvements as patches through Jira etc. <br />Committers have write access to repository<br />Committers review and apply patches, and when you submit lot of them, they will make you a committer. <br />

5.
Success Stories <br />Apache Web Server<br />Linux <br />MySQL <br />Apache Tomcat <br />Apache Axis2 <br />Apache Synapse/WSO2 ESB<br />Firefox<br />Eclipse <br />…<br />Victory<br />Gartner Predicted that by 2012 most systems will use open source components <br />Copyright kafka4prez and licensed for reuse under CC License , http://www.flickr.com/photos/kafka4prez/198465913 <br />

6.
Why People Contribute?<br />Because they Enjoy it <br />To work with smart people<br />Because they get payedto do it. <br />If you are a reputed opensourcedeveloper, bets are that you can get someone to pay you for contributing to opensource. <br />Visibility, to Make an impact <br />Recognition, prestige<br />To Improve your brand / profile <br />To get into Grad school<br />As a Business Strategy <br />Building or supporting an opensource project may be a long term strategic action. <br />Great investments need faith and patience<br />Copyright U. S. Fish and Wildlife Service and licensed for reuse under CC License , http://www.flickr.com/photos/usfwsnortheast/4754624921 and Copyright WxMom and licensed for reuse under CC License , http://www.flickr.com/photos/wxmom/1359996991. <br />

7.
Open Source Business Model<br />Opensourceprojects occupy a significant portion of the middleware space and many others. <br />Many commercial products are powered by Open source projects<br />Many large companies invest a significant amount of resources on opensourceprojects (sometime 1000s)<br />Often there are companies around opensource Projects <br />Business models <br />Build an improved pro versions and sell them<br />Sell production support <br />Provide Consultancy, learning etc. <br />Copyright Emdotand licensed for reuse under CC License, http://www.flickr.com/photos/emdot/2418695<br />

8.
Apache Software Foundation <br />Build on the Success of Apache Web Server <br />Home to many successful and highly influential Open Source Project like Apache Web Server<br />Governed by Apache License<br />Can edit and redistribute, and even sell <br />Not viral, you are free to make money on top it <br />Community is the Key <br />User Community <br />Developer Community <br />Open development model with Open decisions <br />Communication through mailing lists <br />Warm Springs Chiricahua Apache<br />Copyright Jeff Kubina and licensed for reuse under CC License , http://www.flickr.com/photos/95118988@N00/416015918<br />

9.
Apache System Projects<br />Web Service Support <br />Apache Axis2, Apache Rampart, Apache Sandesha, Apache CXF .. <br />Workflow Engine <br />Apache ODE<br />Enterprise Service Bus<br />Apache Synapse <br />Apache Camel <br />Messaging <br />Apache Qpid/ ActiveMQ<br />Data Storages <br />Apache Cassandra, CouchDB, Apache OODT <br />J2EE Container <br />Apace Geronimo<br />…<br />Copyright ind{yeah} and licensed for reuse under CC License , http://www.flickr.com/photos/flickcoolpix/3566848458/<br />

10.
A Large E-Science Project as a CaseStudy<br />

11.
E-Science <br /><ul><li>Continuation of High Performance Computing, Parallel Computing, and Grid.

12.
Underline theme is “Cyber-infrastructures to support Scientific Research”.

13.
Build around “Computation” as the third pillar of Science (along with Analysis and Experimentation).

14.
Characterized by wide range of computing (CPU minutes to CPU years) and Data (few KB to PBs of data) requirements.

15.
Based on Real life usecases. </li></li></ul><li>“Tis strange—but true; for truth is always strange,Stranger than fiction.” ---- Lord Byron, Don Juan (1818-24)<br /><ul><li>E-Science joins Theory with Real life data

16.
Real Life Applications often go beyond our experiences.

17.
Most Weather models are calculated much less than ideal resolutions, otherwise a 24 hour forecast takes more than 24 hours !!!

18.
Physics Usecases (e.g. Large Hadron Collider), Telescopes, Genome Analysis generate Tera bytes of data in days if not hours, and moving a 1TB takes hours even in a 10 GB networks of TeraGrid.

19.
Scale, geographical distribution of resources, Heterogeneity makes these usecases Complex. </li></ul>Surprise<br />Copyright Nrbelex and licensed for reuse under CC License , http://www.flickr.com/photos/nrbelex/529393643<br />

20.
Linked Environments for Atmospheric Discovery (LEAD) <br /><ul><li>U.S. NSF funded, 10+ Universities, 11M $, 5 Years.

21.
Used for U.S. National Weather forecasts by NOAA.

22.
Presented to U.S. Congress as an example to justify Scientific research spending by U.S. NSF.

23.
Have brought the state of the art forecasting capabilities to wider audience ranging from hardcore scientists to high schools students. </li></ul>Copyright f2n_downtown and licensed for reuse under CC License , http://www.flickr.com/photos/myneighborhood/4809104443<br />

24.
LEAD: Dynamic Weather Analysis in U.S. Wide Scale <br />

25.
Why is it Hard? <br /><ul><li>Geographically Distributed Sensors, Computing Power, Storage, and Expertise.

26.
Handling Failures and Recovery

27.
Long Running Jobs (> 1 Hour).

28.
Large Scale Jobs (10-1000+ processors).

29.
Large Sized Data (KBs to GB of data).

30.
Need to serve many parallel users.

31.
Usage spikes. </li></ul>Copyright Wonderlane and licensed for reuse under CC License , http://www.flickr.com/photos/wonderlane/3302165946<br />

32.
LEAD as an Example<br /><ul><li>Assume a Hurricane has developed, and 1000 scientists across U.S. come to the LEAD portal to run forecasts.

33.
Lets assume,

34.
Each user run 3 workflows.

35.
Each Workflow has 6 services, generates about 300 notifications, moves 50 100MB files, generates 50 100MB files, and runs for one hour.

36.
Each Service needs 5 CPUs Hours .</li></ul>Copyright gletham GIS, Social, Mobile Tech Images and licensed for reuse under CC License, http://www.flickr.com/photos/gisuser/54062274/<br />

37.
Which Means <br /><ul><li>3000 Parallel workflows

38.
Need 90,000 CPUs per Hour

39.
250 TPS for messaging System

40.
Move 8GB/Sec through the network

41.
Generate 15TB data per Hour </li></ul>Do the math<br />Not all of this can be handled now, but they give us an idea about the challenge. <br />Copyright matsuyuki and licensed for reuse under CC License, http://www.flickr.com/photos/matsuyuki/5461363022<br />

42.
SOA, E-Science and LEAD<br /><ul><li>E-Science infrastructures are distributed, complex, and heterogeneous.

43.
SOA is designed to handle just the like.

44.
LEAD is based on many SOA Specs

45.
WSDL, SOAP, WS-Addressing for Communication

46.
WS-BPEL for Workflows

47.
WS-Eventing for Messaging

48.
WSDM for service Management

49.
LEAD People have closely worked with and contributed to Web Services, pushing its limits to apply it to LEAD.</li></li></ul><li>LEAD Architecture <br />

50.
Workflow Subsystem<br />

51.
Workflow Subsystem Challenges <br />Maximizing Resource Utilization<br />Utilizing the Cloud <br />Cloud Bursting <br />Handling Priorities <br />Scaling up <br />Service and Workflow Governance <br />Execution Delegation <br />Copyright Doug Lee and licensed for reuse under CC License, http://www.geograph.org.uk/photo/1893583<br />

52.
Data Subsystem<br />

53.
Data Subsystem Challenges<br />Large Scale data Repositories <br />To detect, collect metadata, and store <br />To Search <br />Replica Management <br />Data Mining <br />CEP<br />Clustering algorithms etc. <br />Data Provenance <br />Data Quality <br />Copyright Anne Petty and licensed for reuse under CC License, http://www.geograph.org.uk/photo/101401<br />

54.
Messaging Subsystem<br />

55.
Messaging Subsystem Challenges<br />Underline model is Publish/ Subscribe pattern <br />Challenges are<br />How to scale up? Supporting large number of users and supporting large number of subscriptions<br />Avoid single Point of Failure<br />Ensure guaranteed delivery <br />Security within Publish/Subscribe pattern <br />Related Projects <br />WS-Messenger<br />Narada Broker<br />Apache Qpid<br />Copyright Dave Croker and licensed for reuse under CC License, http://www.geograph.org.uk/photo/689155<br />

56.
LEAD & Apache WS History<br /><ul><li>LEAD and Apache teams both has contributed to other (and there is overlap)

57.
LEAD is older than Axis2, and it forked off in Axis era, mainly because of Async messaging support.

58.
Five years ago LEAD implemented many tools (e.g. Registries, Async Messaging, Workflow Engine), that are hot topics now.

59.
Team receive Continuing funding to make it Open Source under OGCE

60.
LEAD code base now based on Axis2, ODE and others

61.
Moved into Apache as “Apache Airavata”</li></li></ul><li>LEAD with Apache Projects<br /><ul><li>LEAD Switched to Apache ODE for workflow execution more than 3 years ago.

62.
LEAD data subsystems switched to Axis2 about 3 years ago.

63.
Job Submission was switched to Axis2 about 2 years back.

64.
Service Factory is being converted to Axis2 about year back.

65.
Conversion of Messaging System about year back (Through a Indiana University and LSF collaboration). </li></li></ul><li>Apache Airavata<br />All partners agreed that best option for OGCE Project to continue through is Apache Project<br />Joined Apache Incubator about 2 months back <br />Includes following subprojects<br />Xbaya workflow composer <br />WS-Messenger as the Messaging system<br />Generic Service Toolkit <br />Service Registry <br />Copyright ZeePack and licensed for reuse under CC License, http://www.flickr.com/photos/zeepack/3681815248<br />

66.
Should You Try toMove your Project to Apache?<br />

67.
Apache as a Sustainability model for Research projects<br /><ul><li>Industry values “People”, we (opensource) value “Code”, and Academia values “Ideas”.

68.
Most NSF Grants, now, ask for a Sustainability Model as part of Proposals.

69.
One option is a commercial spin off.</li></ul>Diamonds are<br />Forever<br /><ul><li>Doing it in a opensource way, building a community and users around a project is also a potential Solution.

70.
Many Challenges: ownership, need to renounce control, active engagement of the community are the key.

71.
“Source Open” is not good enough!!

72.
“Dump and Run” does not work either. </li></ul>Copyright stephend9 and licensed for reuse under CC License, http://www.flickr.com/photos/stephend9/372996705<br />

73.
Pros & Cons<br />

74.
How does the Model Works?<br />Need a Champion <br />Have to submit a Proposal to Apache Incubator <br />If accepted, will be placed in the incubator <br />Team should work to build the community <br />Users <br />Developers <br />Diversity of the community <br />Graduation <br />More users usually means more contribution <br />Apache Board continues to monitor for compliance <br />

75.
Conclusion<br /><ul><li>Wanted to share a Real Life, Large-Scale SOA Usecase

76.
Wanted to show LEAD-Apache interactions as a real Life Case Study of interactions between Apache and an Academic Project.

77.
Wanted to Showcase Apache as a Sustainability Mechanism, if it is done right.

78.
Wanted to Give you a sense of Some open problems and kind of problems Distributed Systems and E-Science trying to solve. </li></li></ul><li>Questions?<br />Copyright by romainguy, and licensed for reuse under CC License http://www.flickr.com/photos/romainguy/249370084<br />