This is a talk that I gave at the ScienceCloud 2010 Workshop in Chicago on June 21, 2010.

1. Open Science Data Cloud Robert Grossman Open Cloud Consortium

2. Today is a good day to get involved with the Open Science Data Cloud.

3. Part 1: Basic Facts About the OSDC Astronomical data Biological data (Bionimbus) Networking data Image processing for disaster relief 3

4. Who are we?

5. 501(3)(c) Not-for-profit corporation Supports the development of standards, interoperability frameworks, and reference implementations. Manages testbeds: Open Cloud Testbed and IntercloudTestbed. Manages cloud computing infrastructure to support scientific research: Open Science Data Cloud. Develops benchmarks. 5 www.opencloudconsortium.org

6. OCC Members Companies: Aerospace, Booz Allen Hamilton, Cisco, InfoBlox, Open Data Group, Raytheon, Yahoo Universities: CalIT2, Johns Hopkins, MIT Lincoln Lab, Northwestern Univ., University of Illinois at Chicago, University of Chicago Government agencies: NASA Open Source Projects: Sector Project 6

8. Open Science Data Cloud

9. IntercloudTestbed

11. What Are the Projects?

12. Project 1: Bionimubs 10 www.cistrack.org

14. Sector LLPR varies between 0.61 and 0.98 Recent Sloan Digital Sky Survey (SDSS) data release is 14 TB in size.

15. Project 3: Image Processing in the Cloud Mapper Input Key: Bounding Box Mapper Output Key: Bounding Box Mapper Output Value: + Timestamp (minx = -135.0 miny = 45.0 maxx = -112.5 maxy = 67.5) Mapper Input Value: Mapper Output Key: Bounding Box Mapper Output Value: + Timestamp + Timestamp Mapper Output Key: Bounding Box Mapper Output Value: + Timestamp Mapper Output Key: Bounding Box Step 1: Input to Mapper Mapper Output Value: + Timestamp Mapper resizes and/or cuts up the original image into pieces to output Bounding Boxes Mapper Output Key: Bounding Box Mapper Output Value: + Timestamp Mapper Output Key: Bounding Box Mapper Output Value: + Timestamp Mapper Output Key: Bounding Box Mapper Output Value: + Timestamp Mapper Output Key: Bounding Box Mapper Output Value: + Timestamp Step 3: Mapper Output Step 2: Processing in Mapper

16. Project 4: Anomalies in Network Data 13

17. What is the OSDC?

18. Hosted, managed, distributed facility to: Manage & archive your medium and large datasets Provide computational resources to analyze it Provide networking to share it with your colleagues and the public.

19. Long Time Goal Build a (small) data center for science.

20. And preserve your data the same way that libraries preserve books & museums preserve art.

21. Why do it?

22. Work on something that matters to you more than money [and, presumably, papers]. Create more value than you capture. Take the long view. Work on Stuff That MattersTim O’Reilly, Jan 11, 2009

23. What is similar?

24. Internet Archive

25. Wayback Machine

26. Part 2:Why Another Cloud Project?

27. Variety of analysis Scientist with laptop Wide Open Science Data Cloud Med High energy physics, astronomy Low Data Size Medium to Large Small Very Large Dedicated infrastructure No infrastructure General infrastructure

28. Persistent data Large data clouds Med databases HPC Small Cycles Large & spec. clusters Small to medium clusters Single workstations

29. Who do you most trust to manage your data for 100 years? Companies may not be here tomorrow. Government agencies have a role, but not always easy to use. Think of a not for profit with that mission.

30. Part 3:Technical Approach

31. Condominium Clouds In a condominium cloud, you buy your own rack or bunch of racks. The racks are managed and operated by the condominium association, in this case the OCC. If your rack is 120 TB, you get the rights to c. 40 TB of storage in the cloud. The rest is a shared resource. The Open Cloud Testbed is a condo cloud managed by the OCC. 28

32. Condo Clouds Open source software stack: Hadoop, Sector, Eucalyptus, Nova, NoSQLDBs, Raywulf rack

33. Data Migration Challenge: data migration. Solution: use Hadoop style replication.

34. Operating Model Operating model requires constant cap ex investment each year, for example 10 racks or $1M. (Cap in PB)

37. But No Vendor Neutral VN Standard That That scales to 100,000+ VMs Supported by multiple vendors Spans multiple physical switches Supports VN Mobility Provides strong isolation of VN Is easy for VMs to join and leave VNs Includes management interfaces …. OCC has a working group working on VN standards

39. Cloud Compute Services

41. Virtual Data Centers (VDC)

42. Virtual Networks (VN)

43. Virtual Machines (VM)

44. Physical ResourcesOpen Cloud Computing Interface (OCCI) Open Virtualization Format (OVF)

45. One Day We Hope to Peer Open Science Data Cloud

46. More Challenges: Finding a Business Model That Works Long Term Challenge: raising constant amount of funding each year. To date: talking to foundations.

47. Thank You For more information: www.opencloudconsortium.org rgrossman.com (for research papers, etc.)