Em tempo real: Ingestão, processamento e analise de dados
•
0 gostou•940 visualizações
Nesta sessão faremos uma demonstração de controle e defesa de tráfego aéreo utilizando processamento em tempo real. Trataremos das boas práticas para ingestão, armazenamento, processamento e visualização de dados através de serviços da AWS como Kinesis, DynamoDB, Lambda, Redshift, Quicksight e Amazon Machine Learning.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Em tempo real: Ingestão, processamento e analise de dados
Semelhante a Em tempo real: Ingestão, processamento e analise de dados (20)
Mais de Amazon Web Services LATAM
Mais de Amazon Web Services LATAM (20)
Último
Último (20)
Em tempo real: Ingestão, processamento e analise de dados
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Angelo Carvalho, Partner Solutions Architect carvaa@amazon.com Em Tempo Real - Ingestão, Processamento e Análise de Dados
- 2. Agenda 1. Reference Architecture 2. AWS Services 3. Reference Architecture: Demo
- 3. Generation Collection & Storage Analytics & Computation Collaboration & Sharing Big Data Pattern for Batch processing
- 4. Generation Collection & Storage Analytics & Computation Collaboration & Sharing What About Real-Time?
- 6. • Real Time Analytics: • ATC: Detect Collisions! • Air Defense: Find suspicious aircrafts!
- 7. Generation Collection & Storage Analytics & Computation Collaboration & Sharing
- 8. Ingesting Data: Amazon Kinesis Platform
- 9. Amazon Kinesis: Streaming Data Done the AWS Way Makes it easy to capture, deliver, and process real-time data streams Pay as you go, no up-front costs Elastically scalable Right services for your specific use cases Real-time latencies Easy to provision, deploy, and manage
- 10. Amazon Kinesis: Streaming data made easy Services make it easy to capture, deliver and process streams on AWS Kinesis Streams For Technical Developers Kinesis Firehose For all developers, data scientists Kinesis Analytics For all developers, data scientists Build your own custom applications that process or analyze streaming data Easily load massive volumes of streaming data into S3 and Redshift Easily analyze data streams using standard SQL queries
- 13. Amazon Kinesis Streams Build your own data streaming applications Easy administration: Simply create a new stream, and set the desired level of capacity with shards. Scale to match your data throughput rate and volume. Build real-time applications: Perform continual processing on streaming big data using Kinesis Client Library (KCL), Apache Spark/Storm, AWS Lambda, and more. Low cost: Cost-efficient for workloads of any scale.
- 14. Amazon Web Services AZ AZ AZ Durable, highly consistent storage replicates data across three data centers (availability zones) Aggregate and archive to S3 Millions of sources producing 100s of terabytes per hour Front End Authentication Authorization Ordered stream of events supports multiple readers Real-time dashboards and alarms Machine learning algorithms or sliding window analytics Aggregate analysis in Hadoop or a data warehouse Inexpensive: $0.028 per million puts Real-Time Streaming Data Ingestion Custom-built Streaming Applications (KCL) Inexpensive: $0.014 per 1,000,000 PUT Payload Units Amazon Kinesis Streams - GA 2013 Fully managed service for real-time processing of streaming data
- 15. Sending & Reading Data from Kinesis Streams AWS SDK LOG4J Flume Fluentd Get* APIs Kinesis Client Library + Connector Library Apache Storm Amazon Elastic MapReduce Sending Consuming AWS Mobile SDK Kinesis Producer Library AWS Lambda Apache Spark
- 16. • Streams are made of Shards • Each Shard ingests data up to 1MB/sec, and up to 1000 TPS • Each Shard emits up to 2 MB/sec • All data is stored for 24 hours – 7 days • Scale Kinesis streams by splitting or merging Shards • Replay data inside of 24Hr -7days Window Amazon Kinesis Stream Managed Ability to capture and store Data
- 17. Simple Put Interface to put Data in Amazon Kinesis Producers use a PUT call to store data in a Stream. Each record <= 1 MB PutRecord {Data,StreamName,PartitionKey} PutRecords {Records{Data,PartitionKey}, StreamName} A Partition Key is supplied by producer and used to distribute (MD5 hash) the PUTs across (hash key range) of Shards A unique Sequence # is returned to the Producer upon a successful PUT call Putting Data in Amazon Kinesis
- 18. Latency > 200 ms
- 21. Amazon Kinesis Firehose Load massive volumes of streaming data into Amazon S3 and Redshift Zero Admin: Capture and deliver streaming data into S3, Redshift, Elasticsearch and other destinations without writing an application or managing infrastructure Direct-to-data store integration: Batch, compress, and encrypt streaming data for delivery into S3, and other destinations in as little as 60 secs using simple configurations Seamless elasticity: Seamlessly scales to match data throughput w/o operator intervention Capture and submit streaming data to Firehose Firehose loads streaming data continuously into S3 and Redshift Analyze streaming data using your favorite BI tools
- 22. AWS Platform SDKs Mobile SDKs Kinesis Agent AWS IoT Amazon S3 Amazon Redshift • Send data from IT infra, mobile devices, sensors • Integrated with AWS SDK, Agents, and AWS IoT • Fully-managed service to capture streaming data • Elastic w/o resource provisioning • Pay-as-you-go: 3.5 cents/ GB transferred • Batch, compress, and encrypt data before loads • Loads data into Redshift tables base on COPY command Amazon Kinesis Firehose Capture IT & App Logs, Device & Sensor Data, and more Enable near-real time analytics using existing tools Amazon Elasticsearch
- 23. Amazon Kinesis Streams is a service for workloads that requires custom processing, per incoming record, with sub-1 second processing latency, and a choice of stream processing frameworks. Amazon Kinesis Firehose is a service for workloads that require zero administration, ability to use existing analytics tools based on S3, Redshift or Elasticsearch, and a data latency of 60 seconds or higher.
- 24. Latency >= 1 min
- 26. Amazon Kinesis Analytics Analyze data streams continuously with standard SQL Apply SQL on streams: Easily connect to data streams and apply existing SQL skills. Build real-time streaming applications: Perform continual processing on streaming big data with processing latencies in seconds Easy and interactive experience: Complete most stream processing use cases in minutes, and easily progress toward sophisticated scenarios Connect to streaming data sources Run standard SQL queries against data streams Send SQL results to downstream destinations
- 27. Generation Collection & Storage Analytics & Computation Collaboration & Sharing
- 30. AWS Lambda A compute service where you don’t have to think about: • Servers • Being over/under capacity • Deployments • Scaling and fault tolerance • OS or language updates • Metrics and logging …but where you can easily • Bring your own code… even native libraries • Run code in parallel • Create backends, event handlers, and data processing systems • Never pay for idle!
- 31. AWS Lambda – Benefits EVENT-DRIVEN SCALESERVERLESS SUBSECOND BILLING
- 32. AWS Lambda – How It Works DEPLOYMENT AUTHORING MONITORING & LOGGING STATELESS
- 33. AWS Services Integrated with AWS Lambda Amazon S3 Amazon DynamoDB Amazon Kinesis AWS CloudTrail Amazon CloudWatch Logs AWS CloudFormation Amazon SNS Amazon SWF Amazon SES Amazon API Gateway Amazon Cognito
- 34. Generation Collection & Storage Analytics & Computation Collaboration & Sharing
- 37. Why NoSQL? Horizontal scalability Simplicity of design (Unstructured Data) Finer control of availability Less/No limitation in database size Administrative burden of sharding RDBMS
- 38. What is DynamoDB? Non-Relational Managed NoSQL Database Service • Schema less data model • Consistent low latency performance (single digit ms) • Predictable provisioned throughput • Seamless Scalability • No storage limits • High durability and availability (replication between 3 facilities) • Easy Administration – We scale for you! • Low Cost • Cost modelling on throughput and size DynamoDB
- 39. Scalability No limit in terms of throughput (reads/writes per second) No limit in terms of storage DynamoDB automatically partitions data by the hash key Auto-Partitioning occurs when: • Data set growth • Provisioned capacity increase partitions 1 .. N table
- 40. Durability WRITES 3-way replication Quorum acknowledgment Persisted to disk (custom SSD) READS Strongly or eventually consistent No trade-off in latency
- 41. Easy Administration – It’s a service! Making life easier for developers… Developers are freed from: • Performance tuning (latency) • Automatic 3-way multi-AZ replication • Scalability (and scaling operations) • Security inspections, patches, upgrades • Software upgrades, patches • Automatic hardware failover • Improving the underlying hardware …and lots of other stuff
- 43. Latency Single digit miliseconds latency
- 44. Generation Collection & Storage Analytics & Computation Collaboration & Sharing
- 47. Fast, simple, petabyte-scale data warehousing for less than $1,000/TB/Year Amazon Redshift
- 48. Relational data warehouse Massively parallel; Petabyte scale Fully managed HDD and SSD Platforms $1,000/TB/Year; starts at $0.25/hour Amazon Redshift a lot faster a lot simpler a lot cheaper
- 49. The legacy view of data warehousing ... Global 2,000 companies Sell to central IT Multi-year commitment Multi-year deployments Multi-million dollar deals
- 50. … Leads to dark data This is a narrow view Small companies also have big data (mobile, social, gaming, adtech, IoT) Long cycles, high costs, administrative complexity all stifle innovation 0 200 400 600 800 1000 1200 Enterprise Data Data in Warehouse
- 51. The Amazon Redshift view of data warehousing 10x cheaper Easy to provision Higher DBA productivity 10x faster No programming Easily leverage BI tools, Hadoop, Machine Learning, Streaming Analysis in-line with process flows Pay as you go, grow as you need Managed availability & DR Enterprise Big Data SaaS
- 52. Selected Amazon Redshift customers
- 53. Amazon Redshift architecture Leader Node Simple SQL end point Stores metadata Optimizes query plan Coordinates query execution Compute Nodes Local columnar storage Parallel/distributed execution of all queries, loads, backups, restores, resizes Start at just $0.25/hour, grow to 2 PB (compressed) DC1: SSD; scale from 160 GB to 326 TB DS2: HDD; scale from 2 TB to 2 PB Ingestion/Backup Backup Restore JDBC/ODBC 10 GigE (HPC)
- 54. Latency (using firehose) Latency >= 1 min
- 55. Generation Collection & Storage Analytics & Computation Collaboration & Sharing
- 56. Analyzing Data: Amazon Machine Learning
- 58. Data is part of the fabric of the applications Front-end and UX Mobile Back-end and operations Data and analytics
- 59. Three types of data-driven development Retrospective analysis and reporting Amazon Redshift Amazon RDS Amazon S3 Amazon EMR
- 60. Three types of data-driven development Retrospective analysis and reporting Here-and-now real-time processing and dashboards Amazon Kinesis Amazon EC2 AWS Lambda Amazon Redshift, Amazon RDS Amazon S3 Amazon EMR
- 61. Three types of data-driven development Retrospective analysis and reporting Here-and-now real-time processing and dashboards Predictions to enable smart applications Amazon Kinesis Amazon EC2 AWS Lambda Amazon Redshift, Amazon RDS Amazon S3 Amazon EMR
- 62. Machine learning and smart applications Machine learning is the technology that automatically finds patterns in your data and uses them to make predictions for new data points as they become available
- 63. Machine learning and smart applications Machine learning is the technology that automatically finds patterns in your data and uses them to make predictions for new data points as they become available Your data + machine learning = smart applications
- 64. Smart applications by example Based on what you know about the user: Will they use your product?
- 65. Smart applications by example Based on what you know about the user: Will they use your product? Based on what you know about an order: Is this order fraudulent?
- 66. Smart applications by example Based on what you know about the user: Will they use your product? Based on what you know about an order: Is this order fraudulent? Based on what you know about a news article: What other articles are interesting?
- 67. And a few more examples… Fraud detection Detecting fraudulent transactions, filtering spam emails, flagging suspicious reviews, … Personalization Recommending content, predictive content loading, improving user experience, … Targeted marketing Matching customers and offers, choosing marketing campaigns, cross-selling and up-selling, … Content classification Categorizing documents, matching hiring managers and resumes, … Churn prediction Finding customers who are likely to stop using the service, free-tier upgrade targeting, … Customer support Predictive routing of customer emails, social media listening, …
- 68. Latency (for RT predictions) Mostly within 100 MS
- 70. Latencies and Parallelism • Kinesis Streams: Latency > 200 ms • Set IdleTimeBetweenReadsInMillis • Kinesis Firehose: Latency >= 1 min • Set S3 buffer interval • DynamoDB: Single digit miliseconds latency • Don’t set anything
- 71. Latencies and Parallelism • Lambda: Process kinesis stream in parallel: • Create many shards in your stream • Make you lambda call other lambdas • Use many threads • ML: Use real time predictions • Mostly within 100 ms
- 73. Demo Time! • Real Time Analytics: • ATC: Detect Collisions! • Air Defense: Find suspicious aircrafts!
- 74. Remember to complete your evaluations! AWS Summit App