Next generation intelligent data lakes, powered by GraphQL & AWS AppSync - MAD309 - New York AWS Summit

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Next generation intelligent data lakes, powered by GraphQL & AWS AppSync Ed Lima Sr. Solutions Architect AWS AppSync & AWS Amplify Amazon Web Services M A D 3 0 9 Aaron Bawcom Chief Architect Candid Partners

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Agenda • Data and APIs • GraphQL: A query language for APIs • AWS AppSync: Use cases and capabilities • Data lakes, GraphQL, and AWS AppSync

4. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. “You can have data without information, but you cannot have information without data.” Daniel Keys Moran “Data really powers everything that we do.” Jeff Weiner

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS databases and analytics Broad and deep portfolio, built for builders AWS Marketplace Amazon Redshift Data warehousing Amazon EMR Hadoop + Spark Amazon Athena Interactive analytics Amazon Kinesis Data Analytics Real time Amazon ES Operational analytics Amazon RDS MySQL, PostgreSQL, MariaDB, Oracle, SQL Server Amazon Aurora MySQL, PostgreSQL Amazon QuickSight Amazon SageMaker Amazon DynamoDB Key value, document Amazon ElastiCache Redis, Memcached Amazon Neptune Graph Amazon Timestream Time series Amazon QLDB Ledger database Amazon S3/ Amazon S3 Glacier AWS Glue ETL and Data Catalog AWS Lake Formation Data lakes AWS DMS | AWS Snowball | AWS Snowmobile | Amazon Kinesis Data Firehose | Amazon Kinesis Data Streams | AWS Data Pipeline | AWS Direct Connect Data movement AnalyticsDatabases Business intelligence and machine learning Data lake Amazon Managed Blockchain AWS Blockchain Templates Blockchain Amazon Comprehend Amazon Rekognition Amazon Lex Amazon Transcribe AWS DeepLens 250+ solutions 730+ Database solutions 600+ Analytics solutions 25+ Blockchain solutions 20+ Data lake solutions 30+ solutions Amazon RDS on VMWare

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T InternetMobile/web apps Databases/ data stores Basic API technology stack Failover Load balancers Web/application servers Message buses Workers ?API backend ?API “server”

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T InternetMobile/web apps Databases/ data stores Basic API technology stack ?API backend ?API “server” AWS Fail over Load balancers Web/application servers Message buses Workers

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T No servers to provision or manage Scales with usage Never pay for idle Availability and fault tolerance built in Serverless means…

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T InternetMobile/web apps Basic serverless API technology stack AWS Serverless business logic Serverless APIs Databases/ data stores Fail over Load balancers Web/application servers Message buses Workers

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Queries MutationsTypes Subscriptions GraphQL schema and operations A query language for APIs—and a runtime! type User { id: ID! username: String! firstName: String lastName: String daysActive: Int }

20. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. A query language for APIs Queries query GetPost { getPost(id: ”1”) { id title } } mutation CreatePost { createPost(title: “Summit”) { id title } } subscription OnCreatePost { onCreatePost { id title } } Mutations Subscriptions

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS AppSync Managed serverless GraphQL service Connect to data sources in your account Add data sync, real-time, and offline capabilities for any data source or API GraphQL facade for any AWS service Conflict detection and resolution in the cloud Enterprise security features: IAM, Amazon Cognito, OIDC, API keys

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Multi-auth Type, operation, and field-level authorization Default authorization mode Additional providers: • OpenID Connect • Amazon Cognito user pools • AWS IAM • API key

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Multi-auth Type, operation, and field-level authorization Default authorization mode Additional providers: • OpenID Connect • Amazon Cognito user pools • AWS IAM • API key

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Real-time subscriptions subscription onCreateMessage { createMessage {…} } WebSocket URL and connection payload Secure WebSocket connection (wss://) AppSync Data Sources AWS AppSync Amazon DynamoDB Amazon Elasticsearch Service AWS Lambda Amazon Aurora HTTP Local (Pub/Sub)

48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Overview - Terminology - Data is everywhere - Data management challenges - The ideal solution - Data fabric solution

51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Data is everywhere Acquisitions Business units Geographies External partners Supply chains Products Customer engagement IoT Websites Mobile

52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Data management challenges Most companies only analyze 12%of the data they have By 2020, every person will generate 1.7 megabytes in just a second Gartner estimates that poor-quality data is costing organizations on average $14.2 million annually “Nearly one third of analysts spend more than 40% of their time vetting and validating their analytics data before it can be used for strategic decision-making” Organizations make (often erroneous) assumptions about the state of their data and continue to experience inefficiencies, excessive costs, compliance risks, and customer satisfaction issues as a result Corporate data grows 40% per year (Hollis Tibbetts) Business cost of bad data may be as high as 10% to 25% of an organizations revenue 40% of businesses lose sales due to inaccessible data

53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 54 Asinglesystemtoacquire,store, progress,refine,andmakeprovisioned dataaccessibletosystemsofinsight, dataanalysts,datascientists,and decisionmakers. view Operations •Helpdesk •Workflows •Stewardship Catalog •Glossary •Dictionary •Lineage •Search Universal semantic layer •API •Data warehouse Acquisition •Database push/pull •Streaming •File Promotion •Landing, raw, consolidated, processed, USL, and data science Governance •Policy •Process •Security •Entitlement The ideal solution

54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Data acquisition Refinement Storage Lineage Data fabric solution AccessibilityandsystemIntegration usingcloud-nativetechnologies

55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Conceptual pattern Storage Configuration Metadata Acquisition Router Processors Delivery Userinterface Efficiently acquire data Large-scale data transformation governance Bad ETL/ELT codeAPI access

57. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 58 Problem: Efficiently acquire data Acquire data Cost optimization Operations Scale Flexibility

58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 59 Configuration YAML based with UI RDBMS push/pull API pull AWS Lambda, Amazon SQS Files AWS Transfer for SFTP, AWS Lambda, Amazon SQS Streaming Amazon API Gateway, Amazon Kinesis, AWS Lambda, AWS Service Endpoint Solution: Continuous data acquisition

59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 60 Problem: Large-scale data transformation governance Bad source data Source data systems Source data systems ETL dependency challenges

60. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 61 Solution: Create simplified orchestration of data transformation Create data router Amazon DynamoDB, AWS Lambda, Amazon SQS Create data processors Amazon ECS task, AWS Lambda, AWS Glue, Amazon DynamoDB, Amazon Aurora, Amazon S3, Amazon Redshift

61. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 62 Problem: Bad ETL/ELT code Data transformation challenges Difficult to conduct root-cause analysis Difficult ETL code engineering and operations

62. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 63 Solution: Create modernized ETL logic using AWS AppSync Automated creation of GraphQL schema Automated creation of resolvers to cloud- native storage Amazon DynamoDB, Amazon S3, Amazon Redshift, Amazon Aurora Customized creation of resolvers to external data Simplified ETL business logic Simplified debugging Amazon CloudWatch Higher-quality ETL code

63. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T GraphQL standardized ETL rules: - name: eidrProductUpdate src: s3:/eidr/product extract: query: “query { … }” variables: id: $src.productID transform: | ( { data, src } ) => return { …data, …src } load: mutate: “mutate (…) { … }” variables: input: $transform

64. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 65 Problem: No unified API provided by the organization Different suppliers had different forms of EDI interfaces into the organization Different portals were provided for different types of users to order products Different integrations had different “views” of product master data Lack of consistent order execution lead to penalty fines paid to major customers No external enterprise authorization and authentication structure was available from a consumable API

65. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Candid Partners Proprietary & Confidential 66 Solution: Create a single unified external API for e- commerce transactions Continuous data acquisition Amazon DynamoDB Single GraphQL schema AWS AppSync

Next generation intelligent data lakes, powered by GraphQL & AWS AppSync - MAD309 - New York AWS Summit

Esté a ler uma amostra.

Confira estes a seguir

Next generation intelligent data lakes, powered by GraphQL & AWS AppSync - MAD309 - New York AWS Summit

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a Next generation intelligent data lakes, powered by GraphQL & AWS AppSync - MAD309 - New York AWS Summit (20)

Mais de Amazon Web Services (20)