Presentation at CodeStock 2010 on the lessons learned while building an application that handles post-processing of scientific data in the Cloud.

1. Rob Gillen Azure: Lessons From The Field

2. CodeStock is proudly partnered with: RecruitWise and Staff with Excellence - www.recruitwise.jobs Send instant feedback on this session via Twitter: Send a direct message with the room number to @CodeStock d codestock401 This speaker is great! For more information on sending feedback using Twitter while at CodeStock, please see the “CodeStock README” in your CodeStock guide.

3. what we do Consulting | Debugging | Training who we areFounded by top technical and business experts, we are a fast-growinggroup of outstanding consulting and training professionals who pull out all the stops to solve their clients’ problems. how we do it

4. Lessons from the Field: Azure for Science Rob Gillen gillenre@ornl.gov rob.gillenfamily.net @argodev

8. Composite applications

10. Nation’s largest concentrationof open source materials research

11. $1.6B budget

12. 4,350 employees

13. 3,900 researchguests annually

14. $350 million investedin modernization

15. Nation’s most diverse energy portfolio

16. Operating the world’s most intense pulsed neutron source

19. World’s most powerful open scientific computing facility

20. Peak speed of 2.33 petaflops (> two thousand trillion calculations/sec)

21. 18,688 nodes, 224,526 compute cores, 299 TB RAM, 10,000 TB Disk

22. 4,352 ft2 floor space

23. Exascale system by the end of the next decade

24. Focus on computationally intensive projects of large scale and high scientific impact

25. Addressing key science and technology issues

26. Climate

27. Fusion

28. Materials

29. Bioenergy

30. 1st and 4th fastest super computers in the world.The world’s most powerful system for open science

32. Data-Parallelized

35. Programming Struggles

38. Private (On-Premise) Types of Clouds Infrastructure (as a Service) Platform (as a Service)

39. Application Services “Dublin” “Velocity” Frameworks “Geneva” Security Access Control Project “Sydney” Connectivity Service Bus SQL Azure Data Sync Data Compute Windows Azure Platform Table Storage Blob Storage Queue Drive Content Delivery Network Storage

40. Windows Azure Compute Development, service hosting, & management environment .NET, Java PHP, Python, Ruby, native code (C/C++, Win32, etc.) ASP.NET providers, FastCGI, memcached, MySQL, Tomcat Full-trust – supports standard languages and APIs Secure certificate store Management API’s, and logging and diagnostics systems Multiple roles – Web, Worker, Virtual Machine (VHD) Multiple VM sizes 1.6 GHz CPU x64, 1.75GB RAM, 100Mbps network, 250GB volatile storage Small (1X), Medium (2X), Large (4X), X-Large (8X) In-place rolling upgrades, organized by upgrade domains Walk each upgrade domain one at a time Compute

41. Windows Azure Diagnostics Configurable trace, performance counter, Windows event log, IIS log & file buffering Local data buffering quota management Query & modify from the cloud and from the desktop per role instance Transfer to storage scheduled & on-demand Filter by data type, verbosity & time range Compute

42. Windows Azure Storage Rich data abstractions – tables, blobs, queues, drives, CDN Capacity (100TB), throughput (100MB/sec), transactions (1K req/sec) High accessibility Supports geo-location Language & platform agnostic REST APIs URL: http://<account>.<store>.core.windows.net Client libraries for .NET, Java, PHP, etc. High durability – data is replicated 3 times within a cluster, and (Feb 2010) across datacenters High scalability – data is automatically partitioned and load balanced across servers Storage Storage

43. Windows Azure Table Storage Designed for structured data, not relational data Data definition is part of the application A Table is a set of Entities (records) An Entity is a set of Properties (fields) No fixed schema Each property is stored as a <name, typed value> pair Two entities within the same table can have different properties No schema is enforced Table Storage

44. Windows Azure Blob Storage Storage for large, named files plus their metadata Block Blob Targeted at streaming workloads Each blob consists of a sequence of blocks Each block is identified by a Block ID Size limit 200GB per blob Page Blob Targeted at random read/write workloads Each blob consists of an array of pages Each page is identified by its offset from the start of the blob Size limit 1TB per blob Blob Storage

45. Windows Azure Queue Performance efficient, highly available and provide reliable message delivery Asynchronous work dispatch Inter-role communication Polling based model; best-effort FIFO data structure Queue operations Create Queue Delete Queue List Queues Get/Set Queue Metadata Message operations Add Message Get Message(s) Peek Message(s) Delete Message Queue

46. Windows Azure Drive Provides a durable NTFS volume for Windows Azure applications to use Use existing NTFS APIs to access a durable drive Durability and survival of data on application failover Enables migrating existing NTFS applications to the cloud Drives can be up to 1TB; a VM can dynamically mount up to 8 drives A Windows Azure Drive is a Page Blob Example, mount Page Blob as X:br />http://<account>.blob.core.windows.net/<container>/<blob> All writes to drive are made durable to the Page Blob Drive made durable through standard Page Blob replication Drive

47. Windows Azure Content Delivery Network Provides high-bandwidth global blob content delivery 18 locations globally (US, Europe, Asia, Australia and South America), and growing Blob service URL vs. CDN URL Blob URL: http://<account>.blob.core.windows.net/ CDN URL: http://<guid>.vo.msecnd.net/ Support for custom domain names Access details Blobs are cached in CDN until the TTL passes Use per-blob HTTP Cache-Control policy for TTL (new) CDN provides only anonymous HTTP access Content Delivery Network

49. Service-oriented composition

50. Eventual consistency

52. Federated identity

54. Aware of application lifecycles

56. Logical partitioning

57. Distributed in-memory cache

59. Parallel & distributed processing

60. Asynchronous distributed communication

61. Idempotent (handle duplicity)

64. Transform data to be consumable by general processes

67. Generate base “heat map”

68. Combine heat map and base map

70. Query-able

72. If you wanted to look at all 35 TB in the form of these lat/lon plots and if…

73. Every 10 seconds you displayed another map

74. You worked 24 hours/day, 365 days/year

76. 1,825 CSV files generated.

77. 815.84 MB total

78. Average file size is around 457.76 KB

79. Each CSV represented 12,690 data points (lat/lon/temp)

80. 3,650 images generated

81. 145.03 MB total

82. Heat Maps avg. 31.25 KB

83. Combined images avg. 49 KB

86. Partition keys are not queryable… store them

88. Composite applications

89. Automated agents / jobs

93. Flatten: CSV Upload Rate

94. Flatten: Queue Insert Duration

95. Flatten: Single Table Entity Insert

96. ImageGen: CSV File Download Duration

97. ImageGen: CSV File Download Rate

98. Finished Product

99. Parallelized Uploads for Faster Transfer

101. ODATA Service

102. ODATA Browser

103. Resources of Interest Blog: http://rob.gillenfamily.net (source for tools, extensions, etc.) Azure Scope: http://azurescope.cloudapp.net/ (perf tests, metrics, source, etc.) Excel-Driven Monte Carlo Simulation: http://code.msdn.microsoft.com/fullmonte ODATA Feed/Browser: http://data.sciencecloud.us

104. Thank you gillenre@ornl.gov rob.gillenfamily.net

105. The Microsoft Cloud Data Center Infrastructure

106. The Microsoft Cloud Data Center Infrastructure

107. The Microsoft Cloud ~100 Globally Distributed Data Centers Quincy, WA Chicago, IL San Antonio, TX Dublin, Ireland Generation 4 DCs