Enviar pesquisa
Carregar
Serverless Architectural Patterns
•
3 gostaram
•
621 visualizações
Amazon Web Services
Seguir
AWS Builders' Days
Leia menos
Leia mais
Denunciar
Compartilhar
Denunciar
Compartilhar
1 de 49
Recomendados
Building Global Serverless Backends
Building Global Serverless Backends
Amazon Web Services
Containers State of the Union
Containers State of the Union
Amazon Web Services
Advanced Container Management and Scheduling
Advanced Container Management and Scheduling
Amazon Web Services
State of the Union: Containers on AWS
State of the Union: Containers on AWS
Amazon Web Services
Deep Dive into AWS Fargate
Deep Dive into AWS Fargate
Amazon Web Services
Amazon ECS Deep Dive
Amazon ECS Deep Dive
Amazon Web Services
Serverless Developer Experience
Serverless Developer Experience
Amazon Web Services
Intro to Amazon ECS
Intro to Amazon ECS
Amazon Web Services
Recomendados
Building Global Serverless Backends
Building Global Serverless Backends
Amazon Web Services
Containers State of the Union
Containers State of the Union
Amazon Web Services
Advanced Container Management and Scheduling
Advanced Container Management and Scheduling
Amazon Web Services
State of the Union: Containers on AWS
State of the Union: Containers on AWS
Amazon Web Services
Deep Dive into AWS Fargate
Deep Dive into AWS Fargate
Amazon Web Services
Amazon ECS Deep Dive
Amazon ECS Deep Dive
Amazon Web Services
Serverless Developer Experience
Serverless Developer Experience
Amazon Web Services
Intro to Amazon ECS
Intro to Amazon ECS
Amazon Web Services
How To Run Your Containers on AWS with ECS & Fargate: Collision 2018
How To Run Your Containers on AWS with ECS & Fargate: Collision 2018
Amazon Web Services
Getting Started with Kubernetes on AWS
Getting Started with Kubernetes on AWS
Amazon Web Services
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Amazon Web Services
Serverless Architectural Patterns and Best Practices
Serverless Architectural Patterns and Best Practices
Amazon Web Services
CON209_Interstella 8888 Learn How to Use Docker on AWS
CON209_Interstella 8888 Learn How to Use Docker on AWS
Amazon Web Services
Serverless - State Of the Union
Serverless - State Of the Union
Amazon Web Services
Kubernetes on AWS
Kubernetes on AWS
Amazon Web Services
Save 90% on Your Containerized Workloads - August 2017 AWS Online Tech Talks
Save 90% on Your Containerized Workloads - August 2017 AWS Online Tech Talks
Amazon Web Services
DEV317_Deep Dive on AWS CloudFormation
DEV317_Deep Dive on AWS CloudFormation
Amazon Web Services
SMC302 Building Serverless Web Applications
SMC302 Building Serverless Web Applications
Amazon Web Services
CON307_Building Effective Container Images
CON307_Building Effective Container Images
Amazon Web Services
DEV315_Automating Lambda Deployments with GitHub, Jenkins, AWS CodePipeline a...
DEV315_Automating Lambda Deployments with GitHub, Jenkins, AWS CodePipeline a...
Amazon Web Services
NEW LAUNCH! Introducing AWS Fargate - CON214 - re:Invent 2017
NEW LAUNCH! Introducing AWS Fargate - CON214 - re:Invent 2017
Amazon Web Services
(ARC301) Scaling Up to Your First 10 Million Users
(ARC301) Scaling Up to Your First 10 Million Users
Amazon Web Services
Deep Learning for Developers (Advanced Workshop)
Deep Learning for Developers (Advanced Workshop)
Amazon Web Services
Learn how to build serverless applications using the AWS Serverless Platform-...
Learn how to build serverless applications using the AWS Serverless Platform-...
Amazon Web Services
Build a Java Spring Application on Amazon ECS - CON332 - re:Invent 2017
Build a Java Spring Application on Amazon ECS - CON332 - re:Invent 2017
Amazon Web Services
Introduction to AWS Batch
Introduction to AWS Batch
Amazon Web Services
Serverless Architectural Patterns and Best Practices - Madhu Shekar - AWS
Serverless Architectural Patterns and Best Practices - Madhu Shekar - AWS
CodeOps Technologies LLP
Serverless Architecture and Best Practices
Serverless Architecture and Best Practices
Amazon Web Services
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
Amazon Web Services
Serverless Architectures.pdf
Serverless Architectures.pdf
Amazon Web Services
Mais conteúdo relacionado
Mais procurados
How To Run Your Containers on AWS with ECS & Fargate: Collision 2018
How To Run Your Containers on AWS with ECS & Fargate: Collision 2018
Amazon Web Services
Getting Started with Kubernetes on AWS
Getting Started with Kubernetes on AWS
Amazon Web Services
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Amazon Web Services
Serverless Architectural Patterns and Best Practices
Serverless Architectural Patterns and Best Practices
Amazon Web Services
CON209_Interstella 8888 Learn How to Use Docker on AWS
CON209_Interstella 8888 Learn How to Use Docker on AWS
Amazon Web Services
Serverless - State Of the Union
Serverless - State Of the Union
Amazon Web Services
Kubernetes on AWS
Kubernetes on AWS
Amazon Web Services
Save 90% on Your Containerized Workloads - August 2017 AWS Online Tech Talks
Save 90% on Your Containerized Workloads - August 2017 AWS Online Tech Talks
Amazon Web Services
DEV317_Deep Dive on AWS CloudFormation
DEV317_Deep Dive on AWS CloudFormation
Amazon Web Services
SMC302 Building Serverless Web Applications
SMC302 Building Serverless Web Applications
Amazon Web Services
CON307_Building Effective Container Images
CON307_Building Effective Container Images
Amazon Web Services
DEV315_Automating Lambda Deployments with GitHub, Jenkins, AWS CodePipeline a...
DEV315_Automating Lambda Deployments with GitHub, Jenkins, AWS CodePipeline a...
Amazon Web Services
NEW LAUNCH! Introducing AWS Fargate - CON214 - re:Invent 2017
NEW LAUNCH! Introducing AWS Fargate - CON214 - re:Invent 2017
Amazon Web Services
(ARC301) Scaling Up to Your First 10 Million Users
(ARC301) Scaling Up to Your First 10 Million Users
Amazon Web Services
Deep Learning for Developers (Advanced Workshop)
Deep Learning for Developers (Advanced Workshop)
Amazon Web Services
Learn how to build serverless applications using the AWS Serverless Platform-...
Learn how to build serverless applications using the AWS Serverless Platform-...
Amazon Web Services
Build a Java Spring Application on Amazon ECS - CON332 - re:Invent 2017
Build a Java Spring Application on Amazon ECS - CON332 - re:Invent 2017
Amazon Web Services
Introduction to AWS Batch
Introduction to AWS Batch
Amazon Web Services
Mais procurados
(18)
How To Run Your Containers on AWS with ECS & Fargate: Collision 2018
How To Run Your Containers on AWS with ECS & Fargate: Collision 2018
Getting Started with Kubernetes on AWS
Getting Started with Kubernetes on AWS
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Serverless Architectural Patterns and Best Practices
Serverless Architectural Patterns and Best Practices
CON209_Interstella 8888 Learn How to Use Docker on AWS
CON209_Interstella 8888 Learn How to Use Docker on AWS
Serverless - State Of the Union
Serverless - State Of the Union
Kubernetes on AWS
Kubernetes on AWS
Save 90% on Your Containerized Workloads - August 2017 AWS Online Tech Talks
Save 90% on Your Containerized Workloads - August 2017 AWS Online Tech Talks
DEV317_Deep Dive on AWS CloudFormation
DEV317_Deep Dive on AWS CloudFormation
SMC302 Building Serverless Web Applications
SMC302 Building Serverless Web Applications
CON307_Building Effective Container Images
CON307_Building Effective Container Images
DEV315_Automating Lambda Deployments with GitHub, Jenkins, AWS CodePipeline a...
DEV315_Automating Lambda Deployments with GitHub, Jenkins, AWS CodePipeline a...
NEW LAUNCH! Introducing AWS Fargate - CON214 - re:Invent 2017
NEW LAUNCH! Introducing AWS Fargate - CON214 - re:Invent 2017
(ARC301) Scaling Up to Your First 10 Million Users
(ARC301) Scaling Up to Your First 10 Million Users
Deep Learning for Developers (Advanced Workshop)
Deep Learning for Developers (Advanced Workshop)
Learn how to build serverless applications using the AWS Serverless Platform-...
Learn how to build serverless applications using the AWS Serverless Platform-...
Build a Java Spring Application on Amazon ECS - CON332 - re:Invent 2017
Build a Java Spring Application on Amazon ECS - CON332 - re:Invent 2017
Introduction to AWS Batch
Introduction to AWS Batch
Semelhante a Serverless Architectural Patterns
Serverless Architectural Patterns and Best Practices - Madhu Shekar - AWS
Serverless Architectural Patterns and Best Practices - Madhu Shekar - AWS
CodeOps Technologies LLP
Serverless Architecture and Best Practices
Serverless Architecture and Best Practices
Amazon Web Services
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
Amazon Web Services
Serverless Architectures.pdf
Serverless Architectures.pdf
Amazon Web Services
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
Amazon Web Services
Getting started with Serverless on AWS
Getting started with Serverless on AWS
Adrian Hornsby
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
Amazon Web Services
ABD201-Big Data Architectural Patterns and Best Practices on AWS
ABD201-Big Data Architectural Patterns and Best Practices on AWS
Amazon Web Services
Serverless Architectural Patterns: Collision 2018
Serverless Architectural Patterns: Collision 2018
Amazon Web Services
Serverless use cases with AWS Lambda
Serverless use cases with AWS Lambda
Boaz Ziniman
Deep Dive on New Features in Amazon S3 & Glacier - AWS Online Tech Talks
Deep Dive on New Features in Amazon S3 & Glacier - AWS Online Tech Talks
Amazon Web Services
Genomics on aws-webinar-april2018
Genomics on aws-webinar-april2018
Brendan Bouffler
21st Century Analytics with Zopa
21st Century Analytics with Zopa
Amazon Web Services
Scaling Up to Your First 10 Million Users
Scaling Up to Your First 10 Million Users
Amazon Web Services
High-Throughput Genomics on AWS - LFS309 - re:Invent 2017
High-Throughput Genomics on AWS - LFS309 - re:Invent 2017
Amazon Web Services
LFS309-High-Throughput Genomics on AWS.pdf
LFS309-High-Throughput Genomics on AWS.pdf
Amazon Web Services
Building Serverless Applications with Amazon DynamoDB & AWS Lambda - Workshop...
Building Serverless Applications with Amazon DynamoDB & AWS Lambda - Workshop...
Amazon Web Services
Implementazione di una soluzione Data Lake.pdf
Implementazione di una soluzione Data Lake.pdf
Amazon Web Services
Serverless architecture-patterns-and-best-practices
Serverless architecture-patterns-and-best-practices
saifam
Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018
Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018
Amazon Web Services
Semelhante a Serverless Architectural Patterns
(20)
Serverless Architectural Patterns and Best Practices - Madhu Shekar - AWS
Serverless Architectural Patterns and Best Practices - Madhu Shekar - AWS
Serverless Architecture and Best Practices
Serverless Architecture and Best Practices
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
Serverless Architectures.pdf
Serverless Architectures.pdf
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
Getting started with Serverless on AWS
Getting started with Serverless on AWS
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
ABD201-Big Data Architectural Patterns and Best Practices on AWS
ABD201-Big Data Architectural Patterns and Best Practices on AWS
Serverless Architectural Patterns: Collision 2018
Serverless Architectural Patterns: Collision 2018
Serverless use cases with AWS Lambda
Serverless use cases with AWS Lambda
Deep Dive on New Features in Amazon S3 & Glacier - AWS Online Tech Talks
Deep Dive on New Features in Amazon S3 & Glacier - AWS Online Tech Talks
Genomics on aws-webinar-april2018
Genomics on aws-webinar-april2018
21st Century Analytics with Zopa
21st Century Analytics with Zopa
Scaling Up to Your First 10 Million Users
Scaling Up to Your First 10 Million Users
High-Throughput Genomics on AWS - LFS309 - re:Invent 2017
High-Throughput Genomics on AWS - LFS309 - re:Invent 2017
LFS309-High-Throughput Genomics on AWS.pdf
LFS309-High-Throughput Genomics on AWS.pdf
Building Serverless Applications with Amazon DynamoDB & AWS Lambda - Workshop...
Building Serverless Applications with Amazon DynamoDB & AWS Lambda - Workshop...
Implementazione di una soluzione Data Lake.pdf
Implementazione di una soluzione Data Lake.pdf
Serverless architecture-patterns-and-best-practices
Serverless architecture-patterns-and-best-practices
Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018
Serverless Stream Processing Tips & Tricks (ANT358) - AWS re:Invent 2018
Mais de Amazon Web Services
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Amazon Web Services
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Amazon Web Services
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
Amazon Web Services
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
Amazon Web Services
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
Amazon Web Services
Open banking as a service
Open banking as a service
Amazon Web Services
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Amazon Web Services
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
Amazon Web Services
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Amazon Web Services
Computer Vision con AWS
Computer Vision con AWS
Amazon Web Services
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
Amazon Web Services
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Amazon Web Services
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
Amazon Web Services
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Amazon Web Services
Tools for building your MVP on AWS
Tools for building your MVP on AWS
Amazon Web Services
How to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
Amazon Web Services
Building a web application without servers
Building a web application without servers
Amazon Web Services
Fundraising Essentials
Fundraising Essentials
Amazon Web Services
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
Amazon Web Services
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
Amazon Web Services
Mais de Amazon Web Services
(20)
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
Open banking as a service
Open banking as a service
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Computer Vision con AWS
Computer Vision con AWS
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Tools for building your MVP on AWS
Tools for building your MVP on AWS
How to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
Building a web application without servers
Building a web application without servers
Fundraising Essentials
Fundraising Essentials
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
Serverless Architectural Patterns
1.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Serverless Architectural Patterns Danilo Poccia Technical Evangelist danilop@amazon.com @danilop danilop
2.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Agenda • Web application • Data Lake • Stream processing • Operations automation • Real-Time Back-End (with GraphQL)
3.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Lambda Best Practices • Minimize package size to necessities • Separate the Lambda handler from core logic • Use Environment Variables to modify operational behavior • Self-contain dependencies in your function package • Leverage “Max Memory Used” to right-size your functions • Delete large unused functions (75GB limit)
4.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Pattern 1: Web App/Microservice/API
5.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Web application Data stored in Amazon DynamoDB Dynamic content in AWS Lambda Amazon API Gateway Browser Amazon CloudFront Amazon S3 Amazon Cognito
6.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Bustle Achieves 84% Cost Savings with AWS Lambda Bustle is a news, entertainment, lifestyle, and fashion website targeted towards women. With AWS Lambda, we eliminate the need to worry about operations Tyler Love CTO, Bustle ” “ • Bustle had trouble scaling and maintaining high availability for its website without heavy management • Moved to serverless architecture using AWS Lambda and Amazon API Gateway • Experienced approximately 84% in cost savings • Engineers are now focused on innovation
7.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Amazon API Gateway AWS Lambda Amazon DynamoDB Amazon S3 Amazon CloudFront • Bucket Policies • ACLs • Origin Access Identity (OAI) • Geo-Restriction • Signed Cookies • Signed URLs • DDOS Protection IAM AuthZ IAM Serverless web app security • Throttling • Caching • Usage Plans • ACM Browser Amazon Cognito
8.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Custom Authorizer Lambda functionClient Lambda function AmazonAPI Gateway Amazon DynamoDB AWS Identity & Access Management Custom Authorizers SAML Two types: • TOKEN - authorization token passed in a header • REQUEST – all headers, query strings, paths, stage variables or context variables.Custom Authorizer Cognito User Pool
9.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Authorization with Amazon Cognito Cognito User Pool (CUP) Amazon API Gateway Web Identity Provider User A User B User C Cognito Identity Pool (CIP) /web /cip /cup AWS Lambda Amazon DynamoDB Token AWS Credentials User B Data IAM Authorization API Resources C C A A A B BB
10.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Multi-Region with API Gateway us-west-2 us-east-1 Client Amazon Route 53 Regional API Endpoint Regional API Endpoint Custom Domain Name Custom Domain Name API Gateway API Gateway Lambda Lambda
11.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Useful Frameworks for Serverless Web Apps• AWS Chalice Python Serverless Framework https://github.com/aws/chalice Familiar decorator-based API similar to Flask/Bottle Similar to third-party frameworks, Zappa or Claudia.js • AWS Serverless Express Run Node.js Express apps https://github.com/awslabs/aws-serverless-express • Java - HttpServlet, Spring, Spark and Jersey https://github.com/awslabs/aws-serverless-java-container
12.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Pattern 2: Data Lake
13.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Serverless Data Lake Characteristics • Collect/Store/Process/Consume and Analyze all organizational data • Structured/Semi-Structured/Unstructured data • AI/ML and BI/Analytical use cases • Fast automated ingestion • Schema on Read • Complementary to EDW • Decoupled Compute and Storage
14.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. AWS Serverless Data Lake S3 Bucket(s) Key Management Service Amazon Athena AWS CloudTrail Amazon Cognito AWS IAM Amazon Kinesis Streams Amazon Kinesis Firehose Amazon ES Amazon QuickSight AWS Glue Amazon DynamoDB Amazon Macie Amazon API Gateway AWS IAM Amazon Redshift Spectrum AWS Direct Connect Ingest Catalog & Search Security & Auditing API/UI Analytics & Processing AWS Glue AWS Lambda
15.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. The Foundation…Amazon S3 • No need to run compute clusters for storage • Virtually unlimited number of objects and volume • Very high bandwidth – no aggregate throughput limit • Multiple storage classes • Versioning • Encryption • AWS CloudTrail Data Events • S3 Analytics and Inventory • AWS Config automated checks • S3 Object Tagging • S3 Select (NEW!) • Glacier Select (NEW!)
16.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Search and Data Catalog • DynamoDB as Metadata repository • Amazon Elasticsearch Catalog & Search AWS Lambda AWS Lambda Metadata Index (DynamoDB) Search Index (Amazon ES) ObjectCreated ObjectDeleted PutItem Update Stream Update Index Extract Search Fields S3 Bucket https://aws.amazon.com/answers/big-data/data-lake-solution/
17.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Instantly query your data lake on Amazon S3 AWS Glue Crawlers AWS Glue Data Catalog Amazon QuickSight Amazon Redshift Spectrum Amazon Athena S3 Bucket(s) Catalog & Search
18.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Analytics and Processing • Amazon QuickSight • Amazon Athena • AWS Lambda • Predictive Analytics • Amazon EMR • AWS Glue (ETL) Analytics & Processing
19.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Athena – Serverless Interactive Query Service 44.66 seconds...Data scanned: 169.53GB Cost: $5/TB or $0.005/GB = $0.85 SELECT gram, year, sum(count) FROM ngram WHERE gram = 'just say no' GROUP BY gram,year ORDER BY year ASC; Analytics & Processing
20.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Athena – Best Practices • Partition data s3://bucket/flight/parquet/year=1991/month=1/day=2/ • Columnar formats – Apache Parquet, AVRO, ORC • Compress files with splittable compression (bzip2) • Optimize file sizes https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-tips-for- amazon-athena/ Analytics & Processing
21.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Serverless batch processing AWS Lambda: Splitter Amazon S3 Object Amazon DynamoDB: Mapper Results AWS Lambda: Mappers …. …. AWS Lambda: Reducer Amazon S3 Results Analytics & Processing
22.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Fannie Mae Serverless Financial Modeling Financial Modeling is a Monte-Carlo simulation process to project future cash flows , which is used for managing the mortgage risk on daily basis: • Underwriting and valuation • Risk management • Financial reporting • Loss mitigation and loan removal • ~10 Quadrillion (10𝑥1015 ) of cash flow projections each month in hundreds of economic scenarios. • One simulation run of ~ 20 million mortgages takes 1.4 hours, >4 times faster than the existing process. Federal National Mortgage Association
23.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Pywren • Pywren Python library provides 10 TFLOPS of peak compute power with new default – 1,000 concurrent functions • Achieve over 60 GB/sec of read and 50 GB/sec of write performance using Amazon S3 https://pywren.io Analytics & Processing
24.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Pattern 3: Stream Processing
25.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Stream processing characteristics • High ingest rate • Near real-time processing (low latency from ingest to process) • Spiky traffic (lots of devices with intermittent network connections) • Message durability • Message ordering
26.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Streaming data ingestion Amazon CloudWatch: Delivery metrics Amazon S3: Buffered files Kinesis Agent Record Producers Amazon Redshift: Table loads Amazon Elasticsearch Service: Domain loads Amazon S3: Source record backup AWS Lambda: Transformations & enrichment Amazon DynamoDB: Lookup tables Raw records Lookup Transformed records Transformed recordsRaw records Amazon Kinesis Firehose: Delivery stream
27.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Best practices • Tune Firehose buffer size and buffer interval • Larger objects = fewer Lambda invocations, fewer S3 PUTs • Enable compression to reduce storage costs • Enable Source Record Backup for transformations • Recover from transformation errors • Follow Amazon Redshift Best Practices for Loading Data • How to handle time series, sorted data
28.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Sensor data collection IoT rules IoT actions MQTT Amazon S3: Raw records Amazon Kinesis Firehose: Delivery stream Amazon S3: Batched records Amazon Kinesis Streams: Real-time stream AWS IoT: Data collection IoT Sensors Real-time analytics applications
29.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Real-time analytics Amazon Kinesis Streams: Ingest stream Amazon Kinesis Analytics: Time window aggregation Amazon Kinesis Streams: Aggregates stream Amazon Kinesis Firehose: Error stream Amazon S3: Error records Record Producers AWS Lambda: Alert function Amazon DynamoDB: Device thresholds AWS SNS: Notifications
30.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. CREATE OR REPLACE PUMP "STREAM_PUMP" AS INSERT INTO "DESTINATION_SQL_STREAM" SELECT STREAM "device_id", STEP("SOURCE_SQL_STREAM_001".ROWTIME BY INTERVAL '1' MINUTE) as "window_ts", SUM("measurement") as "sample_sum", COUNT(*) AS "sample_count" FROM "SOURCE_SQL_STREAM_001" GROUP BY "device_id", STEP("SOURCE_SQL_STREAM_001".ROWTIME BY INTERVAL '1' MINUTE); Amazon Kinesis Analytics Aggregation 1-minute tumbling window Amazon Kinesis Analytics: Time window aggregation Source stream Destination stream
31.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Real-time analytics Amazon Kinesis Streams: Ingest stream Amazon Kinesis Analytics: Time window aggregation Amazon Kinesis Streams: Aggregates stream Amazon Kinesis Firehose: Error stream Amazon S3: Error records Record Producers AWS Lambda: Alert function Amazon DynamoDB: Device thresholds AWS SNS: Notifications
32.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Streams and AWS Lambda Amazon Kinesis: Stream AWS Lambda: Processor function Streaming source Other AWS services • Number of Amazon Kinesis Streams shards corresponds to concurrent invocations of Lambda function • Batch size sets maximum number of records per Lambda function invocation
33.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Fan-out pattern Fan-out pattern trades strict message ordering vs higher throughput & lower latency Amazon Kinesis: Stream Lambda: Dispatcher function Lambda: Processor function Increase throughput, reduce processing latency Streaming source
34.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Thomson Reuters – Product Insight Solution for usage analysis tracking: Capture, analyze, and visualize analytics data generated by offerings, providing insights to help product teams continuously improve the user experience Throughput: Tested 4,000 requests / second Growing to 10,000 requests / second or 25 Billion requests / month Latency: new events to user dashboards in less than 10 seconds Durable: no data loss since inception
35.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Best practices • Tune batch size when Lambda is triggered by Amazon Kinesis Streams • Higher batch size = fewer Lambda invocations • Tune memory setting for your Lambda function • Higher memory = shorter execution time • Use Kinesis Producer Library (KPL) to batch messages and saturate Amazon Kinesis Stream capacity
36.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Pattern 4: Operations Automation
37.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Automation characteristics • Periodic jobs • Event triggered workflows • Enforce security policies • Audit and notification • Respond to alarms • Extend AWS functionality … All while being Highly Available, Scalable and Auditable
38.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. AWS Ops Automator Amazon CloudWatch: Time-based events AWS Lambda: Event handler AWS Lambda: Task executors AWS SNS: Error and warning notifications Resources in multiple AWS Regions and Accounts Amazon EC2 Instances Tags OpsAutomatorTaskList CreateSnapshotAmazon DynamoDB: Task configuration & tracking Amazon CloudWatch: Logs Amazon Redshift Clusters https://aws.amazon.com/answers/infrastructure-management/ops-automator/
39.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Image recognition and processing Web App Amazon DynamoDB: Image meta-data & tags Amazon Cognito: User authentication Amazon S3: Image uploads AWS Step Functions: Workflow orchestration Start state machine execution 1 Extract image meta-data 2 Amazon Rekognition: Object detection Invoke Amazon Rekognition Generate image thumbnail 3 3Store meta-data and tags 4 https://github.com/awslabs/lambda-refarch-imagerecognition
40.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Step Functions state machine
41.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Enforce security policies RDP from 0.0.0.0/0 RDP from 0.0.0.0/0 CloudWatch Event Bus in another AWS Account New Security Group ingress rule Amazon CloudWatch Events: Rule AWS Lambda: Remediate and alert AWS SNS: Email alert Ingress rule deleted
42.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Autodesk - Tailor Serverless AWS Account Provisioning and Management Service: • Automates AWS Account creation, • Configures IAM, CloudTrail, AWS Config, Direct Connect, and VPC • Enforces corporate standards • Audit for compliance Provisions new Accounts in 10 minutes vs 10 hours in earlier manual process Open source and extensible: https://github.com/alanwill/aws-tailor
43.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Best practices • Gracefully handle API throttling by retrying with an exponential back-off algorithm (AWS SDKs do this for you) • Publish custom metrics from your Lambda function that are meaningful for operations (e.g. number of EBS volumes snapshotted) • Enable X-Ray tracing for your Lambda functions • Document how to disable event triggers for your automation when troubleshooting
44.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Pattern 4: Real-Time Back End
45.
GraphQL API? AWS
AppSync (Preview)
46.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Demo: GraphQL & AWS AppSync
47.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Summary Use DevOps tools to automate your serverless deployments Apply serverless patterns for common use-cases: • Web application • Data Lake Foundation • Stream processing • Operations automation • Real-time back ends (with GraphQL) What will you build with Serverless?
48.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Further Reading Optimizing Enterprise Economics with Serverless Architectures https://d0.awsstatic.com/whitepapers/optimizing-enterprise-economics-serverless-architectures.pdf Serverless Architectures with AWS Lambda https://d1.awsstatic.com/whitepapers/serverless-architectures-with-aws-lambda.pdf Serverless Applications Lens - AWS Well-Architected Framework https://d1.awsstatic.com/whitepapers/architecture/AWS-Serverless-Applications-Lens.pdf Streaming Data Solutions on AWS with Amazon Kinesis https://d1.awsstatic.com/whitepapers/whitepaper-streaming-data-solutions-on-aws-with-amazon-kinesis.pdf AWS Serverless Multi-Tier Architectures https://d1.awsstatic.com/whitepapers/AWS_Serverless_Multi-Tier_Archiectures.pdf
49.
© 2017, Amazon
Web Services, Inc. or its Affiliates. All rights reserved. Serverless Architectural Patterns Danilo Poccia Technical Evangelist danilop@amazon.com @danilop danilop