TECHTalks - Philadelphia PA - Brien Blandford
•Download as PPTX, PDF•
0 likes•80 views
Data lakes and serverless computing - Brien Blandford - AWS Partner Solution Architect - October 2nd 2019
Recommended
More Related Content
What's hot
What's hot (20)
Similar to TECHTalks - Philadelphia PA - Brien Blandford
Similar to TECHTalks - Philadelphia PA - Brien Blandford (20)
More from EagleDream Technologies
More from EagleDream Technologies (12)
Recently uploaded
Recently uploaded (20)
TECHTalks - Philadelphia PA - Brien Blandford
- 1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 2 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Brien Blandford Partner Solutions Architect Amazon Web Services brieb@amazon.com Data Lakes and Serverless Computing
- 2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 3 The world’s most valuable resource is *Copyright: The Economist, 2017, David Parkins Data is an Asset
- 3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 4 * IDC, Data Age 20215: The Evolution of Data to Life-Critical Don’t Focus on Big Data, Focus on the Data That’s Big, April 2017. Data every 5 years Thereismore data than people think 01100111 01100101 01110100 00100000 01100001 00100000 01101000 01101111 01100010 01100010 01111001 00001101 00001010 01100111 01100101 01110100 00100000 01100001 00100000 01101000 01101111 01100010 01100010 01111001 00001101 00001010 01100111 01100101 01110100 00100000 01100001 00100000 01101000 01101111 01100010 01100010 01111001 00001101 00001010 01100111 01100101 01110100 00100000 01100001 00100000 01101000 01101111 01100010 01100010 01111001 00001101 00001010 01100111 01100101 01110100 00100000 01100001 00100000 01101000 01101111 01100010 01100010 01111001 00001101 0000101001100111 01100101 01110100 00100000 01100001 00100000 01101000 01101111 01100010 01100010 01111001 00001101 00001010 01100111 01100101 01110100 00100000 01100001 00100000 01101000 01101111 01100010 01100010 01111001 00001101 00001010 01100111 01100101 01110100 00100000 01100001 00100000 01101000 01101111 01100010 01100010 01111001 00001101 00001010 0110011 01100010 01100010 01111001 00001101 0000101001100111 01100101 01110100 00100000 15 years live for Data platforms need to 1,000x scale >10x grows
- 4. CHALLENGE Need to create constant feedback loop for designers Gain up-to-the-minute understanding of gamer satisfaction to guarantee gamers are engaged, thus resulting in the most popular game played in the world Fortnite | 125+ million players
- 5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 6 Equinox Fitness Cubs is a company with integrated luxury and lifestyle offerings centered on movement, nutrition and regeneration. Equinox built connected experiences using applications that connect to Apple Health and built data collection in their exercise equipment. More than 200 locations within every major city across the U.S., London, and Canada
- 6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 7 © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How do we build new types of applications that can leverage this data?
- 7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 8 Two Innovations in Tech Serverless Data Lakes
- 8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 9 What is Serverless?
- 9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 10 Amazon.com traffic November 76% 24%
- 10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 11 What changes have to be made in this new world? Architectural patterns Operational model Software delivery
- 11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 12 monolithic application + teams 2001 Lesson learned: decompose for agility 2002 microservices + 2 pizza teams
- 12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 13 On-premise server provisioning X max demand 110% X cost 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 230 1 2 3 4 5 X
- 13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 14 Auto-scaling 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 230 1 2 3 4 5 X max demand 61% X cost
- 14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 15 Auto-scaling more granularly 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 230 1 2 3 4 5 X max demand 51% X cost
- 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 16 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 230 1 2 3 4 5 Auto-scaling with reserved instances X max demand 47% X cost
- 16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 17 What is Serverless? No infrastructure provisioning, no management Automatic scaling Pay for value Highly available and secure
- 17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 18 Comparison ofOperational Responsibility AWS Lambda Serverless Functions AWS Fargate Serverless Containers Amazon ECS/EKS Container Management Service Amazon EC2 Compute as a Service More opinionated Less opinionated AWS manages Customer manages • Data source integrations • Physical hardware, software, networking, and facilities • Provisioning • Application code • Container orchestration, provisioning • Cluster scaling • Physical hardware, host OS/kernel, networking, and facilities • Container orchestration control plane • Physical hardware software, networking, and facilities • Application code • Data source integrations • Work clusters • Security config and updates, network config, firewall, management tasks • Physical hardware software, networking, and facilities • Application code • Data source integrations • Scaling • Security config and updates, network config, management tasks • Provisioning, managing scaling and patching of servers • Application code
- 18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 19 On-Premises Cloud Less Compute Virtual Machine EC2 Elastic Beanstalk AWS LambdaFargate Databases MySQL MySQL on EC2 RDS MySQL RDS Aurora Aurora Serverless DynamoDB Storage Storage S3 Messaging ESBs Amazon MQ Kinesis SQS / SNS Analytics Hadoop Hadoop on EC2 EMR Elasticsearch Service Athena FromOn-Prem to Serverless AWS Storage GW Amazon EBS More Serverless
- 19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 20 Cloud-native architectures are small pieces, loosely joined
- 20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 21 SoWhat are Data Lakes?
- 21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 22 Customers want more value from their data Growing exponentially From new sources Increasingly diverse Used by many people Analyzed by many applications
- 22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 23 Companies want more value from their data Complications Siloed approaches don’t work anymore It’s too expensive and limiting to store data on-premises Implication A new approach is needed to extract insights and value
- 23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 24 Cloud data lakes are the future Data lake Customers want: To move to a single store, i.e., a data lake in the cloud To store data securely in standard formats To grow to any scale with low costs To analyze their data in a variety of ways To easily access and analyze data
- 24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 25 Store exabytes of data Stage from landing dock to transformed to curated – Make available in each stage Load, transform, and catalog once Make data available to many tools Open formats and interfaces support innovationBig Data Huge Data Bulk Ingestion RealTime Streams Data Lake Data Warehouse Analytics Adhoc Query Realtime Data Elasticsearch Data lakes help you cost-effectively scale Media Ingestion AI/ML Services Visualization What Makes a Data Lake
- 25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 27 How it works – Serverless Data Lakes and analytics IAM Encryption OLTP ERP CRM LOB Devices Web Sensors Social Streaming Build Data Lakes quickly • Identify, crawl, and catalog sources • Ingest and clean data • Transform into optimal formats Simplify security management • Enforce encryption • Define access policies • Implement audit login Enable self-service and combined analytics • Analysts discover all data available for analysis from a single data catalog • Use multiple analytics tools over the same data Adhoc Query Data Warehouse AI/ML Services Analytics Visualization Data Catalog Data Lake
- 26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 29 Amazon S3 Data lake AWS Glue (ETL & Data Catalog) Athena Amazon QuickSight Serverless. Zero infrastructure. Zero administration Never pay for idle resources $ Availability and fault tolerance built in Automatically scales resources with usage AWS IoT Core AI/ML Devices Web Sensors Social Serverless Analytics – Deliver On-Demand Kinesis Firehose Consumer
- 27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark 30 What do you want to Build today?
Editor's Notes
- Let’s start with microservices, and how Amazon.com has experience transforming our organization to use microservices. Moving from monoliths to microservices is an organizational change, not just a development and software change.
- Brien Blandford is a Partner Solutions Architect for AWS. His role is to provide our partners with the necessary resources to deliver results to our mutual customers. Brien started with AWS in 2016 with the infrastructure team (yes, he has been in many AWS data centers!) in Seattle, but returned to the New York City region in 2018.
- https://aws.amazon.com/blogs/gametech/epic-fortnite-all-in-on-aws-cloud/
- https://aws.amazon.com/blogs/big-data/closing-the-customer-journey-loop-with-amazon-redshift-at-equinox-fitness-clubs/
- *and do it cost-effectively?
- We are going to cover the basics of serverless compute, how it changes your paradigm for software delivery, maintenance, and use Then we will discuss Data Lakes and the natural affinity for serverless architectures
- Let’s start with microservices, and how Amazon.com has experience transforming our organization to use microservices. Moving from monoliths to microservices is an organizational change, not just a development and software change.
- This kind of approach is a natural fit for building modern applications, which are serverless microservices, delivered with automated release pipelines. So lets look at what changes need to be made and how AWS has helped many of these customers with those changes This is just Amazon’s story but hundreds of thousands of customers have embarked on this journey as well
- We’ll start by using Amazon as an example of building modern applications. At Amazon the way we build software is really driven by maximizing innovation. And specifically, the kind of innovation that lets you do lots and lots of experiments. And while that’s always been part of our DNA, back in 2001 our application itself looked a whole lot different. At that time, Amazon had one giant monolith and one giant database. It wasn’t very agile. IF you wanted to add a new product, you had to do that by editing stuff that was designed for our first product – the book store. This wasn’t conducive to innovation at all. So in 2001 we changed the way we built applications and decomposed to microservices and 2 pizza teams. And yes, the move to microservices was a technical decision but it was also supported by organizational design patterns and a culture that granted teams more ownership over a smaller piece of code - a microservice.
- Assuming one provisions for a little more (10%) of peak anticipation
- 72 possible squares 44 Used
- 120 possible squares 61 Used
- 120 possible squares 24 Reserved 30 on demand
- So what is serverless? When we say serverless, we mean it’s the removal of the undifferentiated heavy lifting that is server operations. This is an important distinction for customers because it allows customers to focus on the building of the application rather than the management and scaling of the infrastructure to support the application. This means not thinking about infrastructure or scaling. It means you only pay for what you value, and it means availability and security are built in. --- These are the tenets that define serverless as an operational model (unless you know better ones): No infrastructure to provision or manage (no servers to provision, operate, patch, etc.) Automatically scales by unit of consumption (scales by unit of work/consumption rather than by server unit) Pay for value billing model (if you value consistent throughput or execution duration you only pay for that unit rather than by server unit) Built-in availability and fault tolerance (no need to architect for availability because it is built into the service)
- You have heard us talk about shared responsibility in the past. Simply stated, a shared responsibility model implies there are parts of the system that AWS is responsible for and there are parts of the systems that you as a customer must take responsibility for. In many cases we provide tools and there is a rich ecosystem of open source and commercial products that makes it easier for all of you to own your side of the responsibility box. There is no one hard line on where this line is drawn between the two parts of shared responisbility. One one side of the spectrum you can leverage the power and flexibility of EC2 to run your own database. You can use something like RDS to simplify the management of the database or use RDS Aurora to completely offload the database storage infrastructure to an AWS managed backend. Alternatively you can move to a fully managed database like DynamoDB where you create tables, put data in those tables, and query the data. There is no infrastructure to manage. QoS is part of the the database service with sufficient knobs and dials to give you the right level of control. The question we keep asking ourselves is: how can we draw this line in a way that allows our customers to innovate on business problems, but makes the underlying infrastructure less visible. Any time you spend coraling infrastructure is undifferentiated heavy lifting. Over the last 12 years as more and more people start in the cloud or move major applications to the cloud, this definition of "undifferentiated havy lifting has evolved"
- You have heard us talk about shared responsibility in the past. Simply stated, a shared responsibility model implies there are parts of the system that AWS is responsible for and there are parts of the systems that you as a customer must take responsibility for. In many cases we provide tools and there is a rich ecosystem of open source and commercial products that makes it easier for all of you to own your side of the responsibility box. There is no one hard line on where this line is drawn between the two parts of shared responsibility. One one side of the spectrum you can leverage the power and flexibility of EC2 to run your own database. You can use something like RDS to simplify the management of the database or use RDS Aurora to completely offload the database storage infrastructure to an AWS managed backend. Alternatively you can move to a fully managed database like DynamoDB where you create tables, put data in those tables, and query the data. There is no infrastructure to manage. QoS is part of the the database service with sufficient knobs and dials to give you the right level of control. The question we keep asking ourselves is: how can we draw this line in a way that allows our customers to innovate on business problems, but makes the underlying infrastructure less visible. Any time you spend corralling infrastructure is undifferentiated heavy lifting. Over the last 12 years as more and more people start in the cloud or move major applications to the cloud, this definition of "undifferentiated heavy lifting has evolved" This is essentially a recap of what we discussed earlier with databases. There is a spectrum of shared responsibility you have over your options for compute. With EC2, you can build and run things, but you manage integrations, scaling, security config, provisioning, patching etc. in addition to your code. Compare that to Lambda, where all you mange is your application code.
- So changes to the architectural pattern are really defined by this quote. If you can loosely couple things, and do lots and lots of experienemts, you’re going to be able to learn faster, and produce more innovation.