Presentation on Amazon Web Services which I presented at DDD North on Saturday 8th October 2011

1. Taking advantage of the Amazon Web Services Family @Ben_Hall Ben@BenHall.me.uk Blog.BenHall.me.uk MaydayHQ.com

2. Agenda S3 EC2 Structuring applications Others

3. What is AWS? The way most startups survive Collection of services each solving one small part of the overall problem Platform / Foundation

5. http://www.cargolaw.com/images/disaster2007.Ital.Florida7.GIF Amazon S3Amazon Simple Storage Service

6. Amazon Product ImagesTwitter Profile ImagesSmugMug – 10tb each monthDropboxCurrently has 449 billion files

7. AppHarbour’s (or any PaaS) storage recommendation

10. Designed to provide 99.99% durability and 99.99% availability Designed to provide 99.999999999% durability and 99.99% availability Also versioning allows you to preserve, retrieve, and restore every version of every object stored in your bucket

12. How to access files?

13. Bucket Name https://s3.amazonaws.com/DDDNorth_Demo_1/ ddd_north_s3.html File Name

16. REST API GET DELETE PUT http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html

17. GET / HTTP/1.1 Host: BucketName.s3.amazonaws.com Authorization: AWS 15B4D3461F177624206A:xQE0diMbLRepdf3YB+FIEXAMPLE= <?xml version="1.0" encoding="UTF-8"?> <ListBucketResultxmlns="http://s3.amazonaws.com/doc/2006-03-01"> <Name>bucket</Name> <Prefix/> <Marker/> <MaxKeys>1000</MaxKeys> <IsTruncated>false</IsTruncated> <Contents> <Key>my-image.jpg</Key> <LastModified>2009-10-12T17:50:30.000Z</LastModified> <ETag>&quot;fba9dede5f27731c9771645a39863328&quot;</ETag> <Size>434234</Size> <StorageClass>STANDARD</StorageClass> <Owner> <ID>8a6925ce4a7f21c32aa379004fef</ID> <DisplayName>mtd@amazon.com</DisplayName> </Owner> </Contents> </ListBucketResult>

18. PUT /my-image.jpg HTTP/1.1 Host: myBucket.s3.amazonaws.com Authorization: AWS 15B4D3461F177624206A:xQE0diMbLRepdf3YB+FIEXAMPLE= Content-Type: text/plain Content-Length: 11434 Expect: 100-continue HTTP/1.1 100 Continue HTTP/1.1 200 OK x-amz-id-2: LriYPLdmOdAiIfgSm/F1YsViT1LW94/xUQxMsF7xiEb1a0wiIOIxl+zbwZ163pt7 x-amz-request-id: 0A49CE4060975EAC x-amz-version-id: 43jfkodU8493jnFJD9fjj3HHNVfdsQUIFDNsidf038jfdsjGFDSIRp Date: Wed, 12 Oct 2009 17:50:00 GMT ETag: "fbacf535f27731c9771645a39863328" Content-Length: 0 Connection: close Server: AmazonS3

19. .NET SDK using (AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretKey)) { MemoryStreamms = new MemoryStream(); PutObjectRequest request = new PutObjectRequest(); request.WithBucketName("Demo_1") .WithCannedACL(S3CannedACL.PublicRead) .WithKey("image.jpg").InputStream = file.InputStream; S3Response response = client.PutObject(request); } http://aws.amazon.com/sdkfornet/

20. .NET SDK ListObjectsRequest request = new ListObjectsRequest(); request.BucketName = bucketName; using (ListObjectsResponse response = client.ListObjects(request)) { foreach (S3Object entry in response.S3Objects) { Console.WriteLine("key = {0} size = {1}", entry.Key, entry.Size); } }

21. https://s3.amazonaws.com/DDDNorth_Demo_1/ ddd_north_s3.html?torrent S3 Torrent Support

24. CloudFront Video Streaming

25. Index Page

26. Bucket Name

28. date = Time.now.strftime("%a, %d %b %Y %H:%M:%S %Z") digest = Base64.encode64(OpenSSL::HMAC.digest(OpenSSL::Digest::Digest.new('sha1'), aws_secret, date)).strip uri = URI.parse("https://cloudfront.amazonaws.com/2010-08-01/distribution/#{distribution}/invalidation") req = Net::HTTP::Post.new(uri.path) req.initialize_http_header({ 'x-amz-date' => date, 'Content-Type' => 'text/xml', 'Authorization' => "AWS %s:%s" % [aws_account, digest] }) req.body = "<InvalidationBatch><Path>" + path + "</Path><CallerReference>ref_#{Time.now.utc.to_i}</CallerReference></InvalidationBatch>" http = Net::HTTP.new(uri.host, uri.port) http.use_ssl = true http.verify_mode = OpenSSL::SSL::VERIFY_NONE res = http.request(req)

29. WARNING HTTPS and CNAMEs are not supported S3 bucket name must only contain lower case alphanumeric characters, periods, or hyphens Bucket names must be unique across entire AWS You can’t rename – Delete + Recreate (which costs)

30. Cost Data transfer from EC2 machines in same region are free AppHarbour runs out of US East Region, meaning transfers to US Standard are free.

31. 1gb stored, 10gb downloaded via 1,000,000 requests == $2.22 per month 1gb stored, 1tb downloaded via 10,000,000 requests == $132.90 per month CloudFront has additional pricing. 10gb = $2 http://calculator.s3.amazonaws.com/calc5.html

33. Amazon EC2Amazon Elastic compute cloud http://www.flickr.com/photos/clarkk/38817024/

34. Heroku, AppHarbouretcZynga > 12,000 nodesAmazon SilkAmazon

35. Built on top of XEN virtualisationYes, you could run a private EC2

41. Be safe

49. Instance Types On demand Reserved Spot

50. Spot Instance Bidding Think Differently. Design a system capable of supporting it Save!

53. Instance Sizes 32bit – micro, small, medium 64bit – Extra-large > Super computer – GPU 1 ECU (biggest == 20) == 2007 Xeon processor / 1.0-1.2 GHz 2007 Opteron

54. Availability Zones Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zone 2-3 per region.

55. Regions US East (Northern Virginia) US West (Northern California) EU (Ireland) Asia Pacific (Singapore) Asia Pacific (Tokyo)

56. Business / Technical Decisions Geo-location It’s a data centre, it can die. Scale across multiple regions for maximum up-time.

57. EBS & Snapshots EBS == Hard drive. Can increase volume size, but means taking machine down. Snapshots == Backup. Can attach to new / different AMI

60. Keypairs BE CAREFUL – DON’T LOSE THE FILE Used to generate password / SSH auth What do you do if you lose the key pair? Storing in AMI? Encrypt with key passed in via user-data

61. User Data Install-lamp #!/bin/bash set -e -x export DEBIAN_FRONTEND=noninteractive apt-get update && apt-get upgrade -y tasksel install lamp-server echo "Please remember to set the MySQL root password!” ec2-run-instances --key KEYPAIR --user-data-file install-lamp ami-bf5eb9d6

62. AMI Amazon Machine Image Official base snapshots or private AMI images allow you to spin up multiple machines. Great when combined with Spot Pricing. Before creating AMI for Windows: Disable SysPrep (BundleConfig.xml) Disable machine renaming (Config.xml)

67. CloudWatch Alerts

68. Auto-scaling Spin up / down nodes

70. Starting machines via cmd

71. ec2-describe-images -o amazon IMAGE ami-23b6534a ec2-public-images/fedora-core4-apache.manifest.xml IMAGE ami-25b6534c ec2-public-images/fedora-core4-apache-mysql.manifest.xml IMAGE ami-26b6534f ec2-public-images/developer-image.manifest.xml IMAGE ami-2bb65342 ec2-public-images/getting-started.manifest.xml ec2-run-instances ami-23b6534a -k windows RESERVATION r-xxxxxxxxxxxxxxxxxxxx default INSTANCE i-xxxxxxxx ami-23b6534a pending windows ec2-describe-instances RESERVATION r-xxxxxxxxxxxxxxxxxxxx default INSTANCE i-xxxxxxxx ami-23b6534a ec2-xx-xxx-xx-xx.compute-1.amazonaws.com ec2-terminate-instances i-xxxxxxxx

72. Via .NET SDK

73. public void LaunchEc2ImageWithSqlServerInstalled() { RunningInstancetargetMachineToStart = null; string ami = "ami-e0916389"; varrunInstancesRequest = new RunInstancesRequest(); runInstancesRequest.ImageId = ami; runInstancesRequest.MinCount = 1; runInstancesRequest.MaxCount = 1; runInstancesRequest.KeyName = "aws"; runInstancesRequest.InstanceType = "t1.micro"; runInstancesRequest.Placement = new Placement { AvailabilityZone = "us-east-1b" }; var response = _ec2.RunInstances(runInstancesRequest); foreach (var instance in response.RunInstancesResult.Reservation.RunningInstance) { if(instance.InstanceState.Name == "pending") //NEED TO WAIT FOR "Running” targetMachineToStart = WaitUntilInstanceStateIs(instance.InstanceId, "running"); } }

74. DescribeInstancesRequest request = new DescribeInstancesRequest(); varresponse = _ec2.DescribeInstances(request); varcount = response.DescribeInstancesResult.Reservation.Count; foreach(var reservation in response.DescribeInstancesResult.Reservation) { var instance = reservation.RunningInstance[0]; }

75. public string GetPassword(string instanceId) { string rsaPrivateKey; using (StreamReader reader = new StreamReader(pathToPem)) { rsaPrivateKey = reader.ReadToEnd(); } GetPasswordDataResult result = _ec2.GetPasswordData(new GetPasswordDataRequest() .WithInstanceId(instanceId)).GetPasswordDataResult; var pass = result.GetDecryptedPassword(rsaPrivateKey); Console.WriteLine(pass); return pass; }

76. WinRM string path = “C:indowsystem32inrs.exe” string args = string.Format("-r:{0} -u:Administrator -p:{1} quot;{2}quot;", machine.Url, GetPassword(machine.InstanceId), cmd); Process.start(path, args); Remotely connects to EC2 machine, executes command.

77. List<String> commands = new List<string>(); commands.Add(@"New-Item c:mp -type directory -ErrorActionSilentlyContinue"); commands.Add(@"$memcached_url = ""http://www.meerkatalyst.com/beta/agent.zip"""); commands.Add(@"$memcached_path = ""c:mpeerkatalyst_agent.zip"""); commands.Add(@"$zip_url = ""https://s3.amazonaws.com/meerkatalyst-us/utilities/unzip.exe"""); commands.Add(@"$zip_path = ""c:mpnzip.exe"""); commands.Add("$client = new-object System.Net.WebClient"); commands.Add("$client.DownloadFile( $memcached_url, $memcached_path )"); commands.Add("$client.DownloadFile( $zip_url, $zip_path )"); commands.Add(@"invoke-expression ""C:mpnzip.exe -o C:mpeerkatalyst_agent.zip -d C:mpgent"""); StringBuilder builder = new StringBuilder(); foreach (var command in commands) builder.AppendFormat("echo {0} >> script.ps1 && ", command); cmd = “string.Format("{0} powershell -File script.ps1 && del script.ps1 && {1} &", builder, pathToAgent)” var output = ExecuteRemoteCommand(cmd, machine); //ask me for the code

78. WARNINGS http://www.flickr.com/photos/gagilas/2659695352/

79. Temp Data It will get deleted on reboots. RabbitMQ + Hbase both used temp locations as default storage. Reboot == lost work. Takes a long time to debug.

80. Sharepoint / SQL Server + Rename Microsoft products generally don’t like a machine to be renamed. Reboots == rename as it’s based on IP. Return it off as described earlier.

81. Regions, Zones. They can go down.

83. Unofficial AMIs They are configured by someone else… Who knows what they have done.

87. Double prices if you want SQL Server More of less anyway. Large $0.48 per hour => $1.08 per hour Extra Large $0.96 per hour => $1.56 per hour

89. Architecture & AWSApplication considerations for AWS http://www.flickr.com/photos/wouterpostma/695850212/

90. Configuration Self discovery Use DNS + Load Balancers to reduce config changes Internal DNS on EC2? HAProxy Store in external system which can update / replicate to other nodes.

91. Hybrid Cloud Providers One won’t fit all Optimize for use-case Best of breed CAN BE REALLY EXPENSIVE + COMPLEX!

92. Think! At some point and scale, the cloud doesn’t make sense. Zynga has around 12,000 EC2 nodes Once game is proven, moved to private cloud. EC2 for experiments and additional load.

93. Small, isolated components Design system for scale Design system for failure Message Queues are great! Use them. Keep everything async

94. User’s Browser Javascript File via Lighttpd ELB Single EC2 Machine Easily add additional nodes to ELB User’s Browser Javascript File via S3 CloudFront

95. Heroku + NodeJS Data Collection Ruby Data Processor RabbitMQ Queue HBase Data Storage Heroku + Rails UI Single EC2 Machine Three separate components, designed to scale each individually when required. Deployment, Scale, Cost

96. Chaos Monkey Kill random machines. See what breaks, fix it. Amazon will do it without telling you, be prepared and plan ahead! http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html

97. Other AWS ServicesElasticCache, MapReduce, SQS, SNS, etc http://www.flickr.com/photos/kky/704056791/

100. Elastic Beanstalk – Easily deploy Java based applications + Stack VPC – virtual network topology Elastic MapReduce – Hosted Hadoop CloudFormation – Similar to Chef / Puppet. Start group of machines RDS – Hosted MySQL / Oracle ElastiCache – Hosted Memcached SNS - Simple Notification Service. Similar to a queue IAM – Sharing access to AWS with a team

101. Amazon Mechanical Turk EC2 for Humans! Crowdsourcing work Grockit used it to verify content - http://mechanicalturk.typepad.com/blog/2011/10/editors-note-ari-bader-natal-is-the-chief-learning-architect-at-grockit-the-fast-growing-online-social-learning-startup-bac.html “Usability Test on www.somewebsite.com #1148” - $10 “Click on google +1 for my website - $0.01”

102. http://awsdocs.s3.amazonaws.com/toolkit-visualstudio/latest/aws-tkv-ug.pdf AWS Toolkit for Visual Studio http://aws.amazon.com/visualstudio/

110. Summary http://www.flickr.com/photos/leon_homan/2856628778/

111. Amazon Web Services Fast Cheap Flexible http://highscalability.com/

112. THANK YOU! @Ben_Hall Ben@BenHall.me.uk Blog.BenHall.me.uk