Big Data Analytics with Amazon Web Services
•
24 gostaram•7,263 visualizações
Slides from the online seminar, August 1st 2012.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais de Amazon Web Services
Mais de Amazon Web Services (20)
Último
Último (20)
Big Data Analytics with Amazon Web Services
- 1. Big Data Analytics with Amazon Web Services Dr. Matt Wood An Online Seminar for Partners. Wednesday 1st August.
- 2. Hello, and thank you.
- 3. Big Data Analytics An introduction
- 4. Big Data Analytics An introduction The story of analytics on AWS
- 5. Big Data Analytics An introduction The story of analytics on AWS Integrating partners
- 6. Big Data Analytics An introduction The story of analytics on AWS Integrating partners Partner success stories
- 8. Data for competitive advantage.
- 9. Using data Customer segmentation, financial modeling, system analysis, line-of-sight, business intelligence.
- 10. Generation Collection & storage Analytics & computation Collaboration & sharing
- 11. Cost of data generation is falling.
- 12. lower cost, increased throughput Generation Collection & storage Analytics & computation Collaboration & sharing
- 13. Generation HIGHLY CONSTRAINED Collection & storage Analytics & computation Collaboration & sharing
- 14. Very high barrier to turning data into information.
- 15. Move from a data generation challenge to analytics challenge.
- 16. Enter the Cloud.
- 19. Move to a distributed data approach.
- 20. Maturation of two things.
- 21. Software for distributed storage and analysis Maturation of two things.
- 22. Software for distributed storage and analysis Maturation of two things. Infrastructure for distributed storage and analysis
- 23. Software Frameworks for data-intensive workloads. Distributed by design.
- 24. Infrastructure Platform for data-intensive workloads. Distributed by design.
- 26. Generation HIGHLY CONSTRAINED Collection & storage Analytics & computation Collaboration & sharing
- 27. Generation Collection & storage Analytics & computation Collaboration & sharing
- 28. Lower the barrier to entry.
- 29. Accelerate time to market and increase agility.
- 30. Enable new business opportunities.
- 31. Washington Post Pinterest NASA
- 32. “AWS enables Pfizer to explore difficult or deep scientific questions in a timely, scalable manner and helps us make better decisions more quickly” Michael Miller, Pfizer
- 33. 2 THE STORY OF ANALYTICS
- 34. EC2 Utility computing. 6 years young.
- 35. Scale out systems Embarrassingly parallel problems. Queue based distribution. Small, medium and high scale.
- 39. Cost optimization. EC2 Utility computing. 6 years young.
- 40. Achieving economies of scale 100% Time
- 41. Achieving economies of scale 100% Reserved capacity Time
- 42. Achieving economies of scale 100% On-demand Reserved capacity Time
- 43. Achieving economies of scale UNUSED CAPACITY 100% On-demand Reserved capacity Time
- 44. Spot Instances Bid on unused EC2 capacity. Very large discount. Perfect for batch runs. Balance cost and scale.
- 45. $650 per hour
- 46. Map/reduce Pattern for distributed computing. Software frameworks such as Hadoop. Write two functions. Scale up.
- 47. Map/reduce Pattern for distributed computing. Software frameworks such as Hadoop. Write two functions. Scale up. Complex cluster configuration and management.
- 48. Amazon Elastic MapReduce Managed Hadoop clusters. Easy to provision and monitor. Write two functions. Scale up. Optimized for S3 access.
- 49. S3 Input data UNDER THE i i HOOD
- 50. S3 Input data Code Elastic MapReduce UNDER THE i i HOOD
- 51. S3 Input data Code Elastic Name MapReduce node UNDER THE i i HOOD
- 52. S3 Input data Code Elastic Name MapReduce node Elastic cluster UNDER THE i i HOOD
- 53. S3 Input data Code Elastic Name MapReduce node HDFS Elastic cluster UNDER THE i i HOOD
- 54. S3 Input data Code Elastic Name MapReduce node Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster UNDER THE i i HOOD
- 55. S3 Input data Code Elastic Name Output MapReduce node S3 + SimpleDB Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster UNDER THE i i HOOD
- 56. S3 Input data Output S3 + SimpleDB UNDER THE i i HOOD
- 71. Performance
- 73. UNDER THE i i Cluster Compute HOOD Intel Xeon E5-2670 10 gig E non-blocking network 60.5 Gb Placement groupings
- 74. UNDER THE i i Cluster Compute HOOD Intel Xeon E5-2670 10 gig E non-blocking network 60.5 Gb Placement groupings + GPU enabled instances
- 76. IO performance Performance Compute performance
- 78. DynamoDB Predictable, consistent performance Unlimited storage Single digit millisecond latencies No schema for unstructured data Backed on solid state drives
- 79. ...and SSDs for all. New Hi1 storage instances.
- 80. UNDER THE i i hi1.4xlarge HOOD 2 x 1Tb SSDs 10 GigE network HVM: 90k IOPS read, 9k to 75k write PV: 120k IOPS read, 10k to 85k write
- 81. “The hi1.4xlarge configuration is about half the system cost for the same throughput.” Netflix http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html
- 83. Provisioned IOPS Provision required IO performance
- 84. Provisioned IOPS Provision required IO performance + EBS-optimized instances with dedicated throughput
- 85. Generation Collection & storage Analytics & computation Collaboration & sharing
- 86. Performance + ease of use
- 88. Extend platform with partners
- 89. Innovate on behalf of customers
- 90. Remove undifferentiated heavy lifting
- 91. MapR distribution for EMR Rolled the Amazon Hadoop optimizations into MapR Choice for EMR customers Easy deployment for MapR customers
- 92. MapR distribution for EMR Hadoop distribution Integrated into EMR NFS and ODBC drivers High availability and cluster mirroring
- 93. Informatica on EMR Enterprise data toolchain “Swiss army knife” for data formats Data integration Available to all on EMR
- 94. AWS Marketplace Karmasphere, Marketshare, Acunu Cassandra, Metamarkets, Aspera and more. aws.amazon.com/marketplace
- 96. Razorfish
- 97. 3.5 billion records 71MM unique cookies 1.7MM targeted ads per day
- 98. 3.5 billion records 71MM unique cookies 1.7MM targeted ads per day 500% improvement in return on ad spend.
- 99. Cycle Computing + Schrodinger
- 100. 30k cores, $4200 an hour (compared to $10+ million)
- 101. Marketshare + Ticketmaster Optimize live event pricing
- 102. Reduced developer infrastructure management time by 3 hours a day
- 103. Thank you!
- 104. Q&A matthew@amazon.com @mza on Twitter