2. Presentation Outline (hidden slide): Technical Level: 300 Intended Audience: Developers Objectives (what do you want the audience to take away): 1. To be able to build and deploy a Windows Azure Service 2. To be able to use blobs, tables and queues effectively 3. Understand how to use the API for logging Presentation Outline (including demos): Compute Demo: Building and Deploying a Service Demo: Logging and Configuration Demo: Worker Roles Storage Demo: Using Blobs Demo: Using Queues Demo: Using Tables Advanced Table Concepts
4. Windows Azure Windows Azure is the foundation of Microsoft’s Cloud Platform It is an “Operating System in the Cloud” and provides Essential Services for the cloud Virtualized Computation Scalable Storage Automatic Management Developer SDK
5. Role Programming Model Inherits RoleEntryPoint OnStart() Method Called by Fabric on startup, allows you to perform initialization tasks. Reports Busy status to load balancer until you return true. OnStop() Method Called when role is to be shutdown, graceful exit. Run() Method Main logic is here – can do anything, typically loop and never exit.
6. Web Role ASP.NET 3.5 SP1 – 64bit IIS7 Hostable Web Core Hosts Webformsor MVC FastCGI applications (e.g. PHP) Http(s)/TCP WebRole New web.roleConfig to support FastCGI Support same Role model semantics as Worker Role (e.g. OnStart, Run, etc.)
9. Configuration Service Configuration Serviceconfiguration.csdef– Service Model ServiceConfiguration.cscfg – instance data RoleEnvironment.GetConfigurationSettingValue() Don’t use web.config for values you wish to change at runtime Web.config change requires re-deploy
10. Monitoring No Debugging in Cloud Instrument your application using Trace, Debug DiagnosticMonitorTraceListener Use Diagnostics API to Configure and Collect Event Logs Performance Counters Trace/Debug information (logging) IIS Logs, Failed Request Logs Crash Dumps or Arbitrary files Request data on demand or scheduled Transferred into your table and/or blob storage Everything is remotely configurable
12. Using Queues to Decouple Roles Use Queue to Decouple Design for Idempotency multiple applications of the operation does not change the result Web Role Worker Role Worker Role Web Role Worker Role Worker Role Storage Queue LB LB
14. Upgrading Your Application Two Models: VIP Swap and In-Place Upgrade VIP Swap: Uses Staging and Production environments. Allows to quickly swap environments. Production: v1 Staging: v2, after swap then Production: v2 Staging: v1. In-Place Upgrade Performs a rolling upgrade on live service. Entire service or a single role Manual or Automatic across update domains
15. Storage Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – A durable NTFS volume backed by blob storage Tables – Provide structured storage; A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application
16. Storage in the Dev Fabric Provides a local “Mock” storage Emulates storage in cloud Allows offline development Requires SQL Express 2005/2008
17. Storage in the cloud Create a Storage Account Endpoints Access Keys CDN
18. Windows Azure Storage Account User creates a globally unique storage account name Can choose geo-location to host storage account “US Anywhere”, “US North Central”, “US South Central”, Can co-locate storage account with compute account Receive a 256 bit secret key when creating account Storage Account Capacity at Commercial Availability Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription
19. Blob Containers Number of Blob Containers Can have has many Blob Containers that will fit within the storage account limit Blob Container A container holds a set of blobs Set access policies at the container level Private or Public accessible Associate Metadata with Container Metadata are <name, value> pairs Up to 8KB per container List the blobs in a container Can now include Blob Metadata and MD5 (new)
20. Blob Features and Functions Store Large Objects (100s of GBs in size) Associate Metadata with Blob Metadata is <name, value> pairs, Up to 8KB per blob Set/Get with or separate from blob data bits Standard REST Interface PutBlob Inserts a new blob, overwrites the existing blob GetBlob Get whole blob or a specific range DeleteBlob CopyBlob(new) SnapshotBlob(new) LeaseBlob(new)
21. Blocks or Pages Blob Container Account Blob Storage ConceptsKey concepts account, container, blob and blocks/pages IMG001.JPG Pictures IMG002.JPG Account Block/Page 1 Movies MOV1.AVI Block/Page 2 Block /Page 3
22. Two Types of Blobs Under the Hood Block Blob Targeted at streaming workloads Each blob consists of a sequence of blocks Each block is identified by a Block ID Size limit 200GB per blob Page Blob (new) Targeted at random read/write workloads Each blob consists of an array of pages Each page is identified by its offset from the start of the blob Size limit 1TB per blob
23. Block Blob Details Block can be up to 4MB each Each block can be variable size Each block has a 64 byte ID Scoped by blob name and stored with the blob Block operation PutBlock Puts an uncommitted block defined by the block ID for the blob Block List Operations PutBlockList Provide the list of blocks to comprise the readable version of the blob Can use blocks from uncommitted or committed list to update blob GetBlockList Returns the list of blocks, committed or uncommitted for a blob Block ID and Size of Block is returned for each block
24. Page Blob Details Page Blob is created with a Max Blob Size Can change the max size of the blob at anytime. Address space is broken up into fixed sized 512 byte pages for updates Page update operations – must be page aligned PutPage- limited to 4MB Overwrite range of pages starting at the specified offset ClearPage- can specify up to max size of blob Clear range of pages at the offset Reading a Page Blob GetBlob – can read from any byte offset for any valid range What parts of the Page Blob have stored pages in them GetPageRange Get valid page ranges in the blob Only charged for pages with data stored in them
25. Choosing Between Block and Page Blob Block Blob Targeted at streaming workloads Update semantics Upload a bunch of blocks. Then commit change. Concurrency: ETag Checks Page Blob Targeted at random read/write workloads Update Semantics Immediate update Concurrency: Leases
27. Summary Of Windows Azure Blobs Blob Access Patterns Block Blobs – streaming Page Blobs – random read/write (new) New Blob Operations Copy, Snapshot, and Lease work for both types New Ways of Accessing and Serving Blob Content (see MSDN) Content Delivery Network access Custom Domain Names Root Blob Container Shared Access Signatures Additional new features ListBlob, GetBlob, Blob Properties (see MSDN docs)
28. Queues Simple asynchronous dispatch queue Create and delete queues Message: Retrieved at least once Max size 8kb Operations: put get delete
29. Queue Storage ConceptsAccount, queue and message Message Queue Account 128x128, http://… Thumbnail Jobs 256x256, http://… Account http://… Indexing Jobs http://…
31. Summary Of Queues Provide reliable message delivery Allows Messages to be retrieved and processed at least once No limit on number of messages stored in a Queue Message size is <=8KB
32. Tables Entities and properties (rows & columns) Tables scoped by account Designed for billions+ Scale-out using partitions Partition key & row key Operations performed on partitions Efficient queries No limit on number of partitions Use ADO.NET Data Services
34. Entities and Properties Each Entity can have up to 255 properties Every Entity has fixed key properties Partition key Row key Timestamp No fixed schema for rest of properties 2 entities in the same table can have different properties Properties stored as <Name, TypedValue> pairs Each entity has a system maintained version
35. Property Types Partition key and Row key String (up to 1KB) Other properties String (up to 64KB) Binary (up to 64KB) Bool DateTime GUID Int Int64 Double
36. Partition Key And Partition Every Entity has a partition key All entities in a table with the same partition key value live in the same partition Need to choose partitioning scheme to make data access scalable
37. Partitioning Guidelines Performance Use a PartitionKey that is common in your queries Always try to specify the partition key in the query Entities with same partition key value are clustered Batch capabilities and transaction are supported within a partition only. Scalability We monitor partition traffic Automatically load balance partitions Each partition can potentially be served by a different storage node Scale to meet the traffic needs of your application More partitions – makes it easier to balance load
39. ClientB Client A Concurrent Updates Version Rating 1: Ch9, Jan-2, 4 1: Ch9, Jan-2, 5 2: Ch9, Jan-2, 5 Error: 412 5 : Ch9, Jan-1, 3 If-Match: 1 Ch9, Jan-2, 4 1 : Ch9, Jan-2, 2 1 : Ch9, Jan-2, 2 1 : Ch9, Jan-2, 2 2: Ch9, Jan-2, 5 If-Match: 1 Ch9, Jan-2, 5 9 : Ch9, Jan-3, 6 Use standard HTTP mechanisms – Etag and If-Match Get entity – get system maintained version as ETag Update Entities Locally – change rating Send Update with version check - IF-Match with Etag Success if version matches, and update version on Client-A Precondition failed (412) if version does not match
40. .NET: LINQ Take(N) function Getting the Top N entities serviceUri = new Uri("http://<account>.table.core.windows.net"); DataServiceContextcontext = new DataServiceContext(serviceUri); varallMessages = context.CreateQuery<Message>("Messages"); foreach (Message messagein allMessages.Take(100)) { Console.WriteLine(message.Name); } REST: $top=N query string option GET http://<serviceUri>/Messages?$top=100
41. Pagination – Continuation Tokens Send a request GET http://<serviceUri>/Messages?$filter=...&$top=100 Get continuation token in response header Messages x-ms-continuation-NextPartitionKey: xxxxxxx x-ms-continuation-NextRowKey: yyyyyy Ch9, Date1, 100 Ch9, Date2, Ch9, … Set HTTP query parameters Ch9,Date100, GET http://<Uri>/Messages?$filter=...&$top=100 Ch9,Date101, &NextPartitionKey=xxxxxxx &NextRowKey=yyyyyy Ch9, …
42. Single Table Consistency ACID transactions for single entity CUD Insert/update/delete Snapshot isolation for query within a single partition Consistent view from start time of the query No dirty (uncommitted) reads Does not block concurrent updates No snapshot isolation across partitions No snapshot isolation across different continuations of a query Batch transactions within partition only
43. Cross Table Consistency Application is responsible for maintaining consistency Example When a channel is deleted, delete all the messages for that channel Failures can occur in the middle Example - Application fails after deleting some messages Use Windows Azure Queues to help ensure completion of operation
44. Worker Queue Messages Cross Table Consistency Delete channel Delete messages worker Front end Ch1, Msg1 Del Ch1 Del Ch5 Del Ch11 Del Ch1 2 Ch1, Msg2 Ch1, Msg3 Front End Ch2, Msg1 Ch2, Msg2 Channels Ch3, Msg1 Ch1,… Ch2,… 1. Dequeue DelCh1 2. Delete Ch1 from Channels 3. Delete from Messages 4. Delete queue entry
45. Delete channel Delete messages worker Front end Worker 1 Del Ch1h Queue Messages Resuming After Failure Ch1, Msg1 Del Ch1 Del Ch5 Del Ch11 Del Ch1 2 Ch1, Msg2 Ch1, Msg3 Front End Ch2, Msg1 Worker2 Ch2, Msg2 Channels Ch3, Msg1 Ch1,… Ch2,… 1. Dequeue DelCh1 and start delete 2. Fails after deleting Ch1 and Msg1 3. DelCh1 is visible again 4. Dequeue DelCh1 again 5. Repeat delete operations
46. Table Summary Windows Azure tables are Massively Scalable Highly Available Simple familiar API Use .NET –- ADO.NET Data Services and LINQ Or use REST Leverage your .NET expertise
47. Learning Windows Azure www.windowsazure.com http://channel9.msdn.com/learn Download the SDK You don’t need cloud access to develop! Look at the samples in the SDK Windows Azure Platform Training Kit 3 Windows Azure labs Follow the team bloggers
