O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Have Your Front End and Monitor It, Too (ANT303) - AWS re:Invent 2018

207 visualizações

Publicada em

Amazon Elasticsearch Service (Amazon ES) is both a search solution and a log monitoring solution. In this session, we address both. We build a front-end, PHP web server that provides a search experience on movie data as well as backend monitoring to send Apache web logs, syslogs, and application logs to Amazon ES. We tune the relevance for the search experience and build Kibana visualizations for the log data. In addition, we use security best practices and deploy everything into a VPC.

  • Seja o primeiro a comentar

Have Your Front End and Monitor It, Too (ANT303) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Have Your Front End and Monitor It, Too Kevin Fallis Senior Specialist Solutions Architect Amazon Web Services A N T 3 0 3
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda  Review lab goals (5 minutes)  Lab setup (5 minutes)  Amazon Elasticsearch Service overview (20 minutes)  Hands-on labs (105 minutes)
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lab goals  Build an application stack that leverages the Amazon Elasticsearch Service  Build visualizations in Kibana  Explore configurations for Filebeats and Logstash, and leverage both for moving and transforming data at scale before publishing to Amazon Elasticsearch Service
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Instructions • Each student will have a prebuilt AWS account • Each one of you will have lab credentials for an AWS account, which you will use for the labs • Please refrain from using personal accounts during the lab • Each account has been validated to manage limits and other concerns • Navigate to the following URL - https://chilp.it/8ed3b89 - (example URL) • Follow the instructions on the static website starting with the Lab Instructions • You will launch an AWS CloudFormation stack that will build out the end-to-end solution
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Web application – Leveraging Elasticsearch
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Monitoring solution – Leveraging Kibana
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Component overview Caching stack Persistence stackNetwork stack Indexers stack Application stackPortal stack Starting with VPC, build out the security, routing and foundation for the solution Add in a caching layer that buffers records for indexers to write to Elasticsearch Deploy an Amazon Elasticsearch Service stack in a private VPC Access your deployment via a portal built on Amazon EC2 and accessed by SSH Create a fleet of Logstash indexers that pull log events from cache and write to Elasticsearch Leverage a fleet of web servers that present an application that uses Elasticsearch for the data
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Things built with a purpose
  12. 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data warehouse | Big data processing AWS databases and analytics Broad and deep portfolio, purpose built for builders Business intelligence & machine learning Data movement AWS DMS | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams Amazon QuickSight Relational databases Amazon RDS Aurora Data lake Amazon S3/ Amazon Glacier AWS Glue Amazon SageMaker Non-relational databases Analytics DynamoDB ElastiCache Neptune Amazon Redshift Amazon EMR Athena Kinesis Data Analytics Amazon ES Real-time Amazon Comprehend
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Text search Natural language Boolean queries Relevance Streaming data High-volume ingest Near real time Distributed storage Data analysis Time-based visualizations Nestable statistics Time-series tools Purpose built for search and analytics Amazon Elasticsearch Service
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Elasticsearch Service Amazon Elasticsearch Service is a fully managed service that makes it easy to deploy, manage, and scale Elasticsearch and Kibana in the AWS cloud
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Benefits of Amazon Elasticsearch Service Easy to use ScalableOpen source Highly available IntegratedSecure Drop-in replacement with no need to learn new APIs or skills Deploy a production- ready Elasticsearch cluster in minutes Resize your cluster horizontally or vertically with a few clicks or a single API call Deploy into your VPC and restrict access using security groups and IAM policies Replicate across Availability Zones with monitoring and automated self-healing Leverage other AWS services for seamless data ingestion, security, auditing, and orchestration
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Elasticsearch Service customers Software & Internet Financial services Biotech & pharma Media & entertainment Telecommunications Travel & transportation Real estate Social media Logistics & operations
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The Elastic Stack (ELK)
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Real-time, distributed, search & analytics engine • Built on top of Apache Lucene • Schema free • Developer friendly RESTful API • Supports structured, unstructured, and geospatial queries • Designed for the cloud from the ground up
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Server-side data processing pipeline • Open-source tool for collecting, parsing, and storing logs • Ingest unstructured data from a variety of data sources including system, access, and application server logs • Offers pre-built filters, so you can readily transform common data types, index them in Elasticsearch • Flexible plugin architecture – can write your own if needed
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Powerful visualization & query tool • Enables visual exploration and real-time analysis of your data in Elasticsearch • Visualizations support histograms, line graphs, pie charts, sunbursts, and more • Powerful geospatial capabilities so that you can seamlessly layer in geographical information on top of your data
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Purpose-built, agent-based data shippers • Lightweight agents built to send data at scale (no JVM) • Perform actions such as • Forward logs to a centralized location • Collect metrics from your servers • Network packet analysis • Health check and status monitoring
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How these fit in the analytics framework • Beats • Logstash COLLECT STORE PROCESS/ ANALYZE VISUALIZE • Elasticsearch • Logstash • Elasticsearch • Kibana
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Network stack AWS Region VPC Service VPCs AvailabilityZone2AvailabilityZone1 Publicsubnet Privatesubnet Publicsubnet Privatesubnet Publicsubnet NAT gateway
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Caching stack AWS Region VPC Service VPCs AvailabilityZone2AvailabilityZone1 Publicsubnet Privatesubnet Publicsubnet EC ENI EC ENI PrivatesubnetPortal Publicsubnet NAT gateway
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Persistence stack AWS Region VPC Service VPCs AvailabilityZone2AvailabilityZone1 Publicsubnet Privatesubnet Publicsubnet EC ENI ES ENI ES ENI EC ENI PrivatesubnetPortal Publicsubnet NAT gateway
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Portal stack AWS Region VPC Service VPCs AvailabilityZone2AvailabilityZone1 Publicsubnet Privatesubnet Publicsubnet EC ENI ES ENI ES ENI EC ENI PrivatesubnetPortal Publicsubnet NAT gateway
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Indexer stack AWS Region VPC Service VPCs AvailabilityZone2AvailabilityZone1 Publicsubnet Privatesubnet Publicsubnet EC ENI ES ENI ES ENI EC ENI Privatesubnet AutoScalinggroup Indexers Portal Publicsubnet NAT gateway
  30. 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Application stack AWS Region VPC Service VPCs AvailabilityZone2AvailabilityZone1 Publicsubnet Privatesubnet Publicsubnet EC ENI ES ENI ES ENI EC ENI Privatesubnet AutoScalinggroup AutoScalinggroup IndexersApplication Portal Publicsubnet NAT gateway
  31. 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Final architecture AWS Region VPC Service VPCs AvailabilityZone2AvailabilityZone1 Publicsubnet Privatesubnet Publicsubnet EC ENI ES ENI ES ENI EC ENI Privatesubnet AutoScalinggroup AutoScalinggroup IndexersApplication Portal Publicsubnet NAT gateway
  32. 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  33. 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Case study: Financial Times Problem • What stories do our readers care about? • What’s hot? • Required a custom clickstream analytics solution • Must be delivered in real time • Operational staff had very little bandwidth to manage custom analytics infrastructure Solution • Stream user data to Amazon Elasticsearch Service • Created custom dashboard for editors and journalists • Lantern – “Shines a light” on reader engagement • Lantern is critical for editorial decisions Benefits • Reliability – Lantern is used throughout the day by journalists and editors. System uptime is critical • Cost savings – Easily able to tune their cluster to meet needs with minimal management overhead
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Logs – 70% of all workloads Troubleshooting • Discover why an install failed • Understand from where your web traffic originates and make scaling decisions • Search for broken links • View network traffic
  36. 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Security Security Information and Event Management (SIEM) • Analytics & visualization tools to detect issues in near real time • High-throughput data ingestion from multiple sources • Find beacons in events that indicate malware is present
  37. 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Streaming IoT • You have a fleet of devices in your factory • Intermittently, each device is sending status information about products on the assembly line • You piece together each device’s status to understand the efficiency of your production activities
  38. 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Geospatial Connecting buyers with sellers • You are an advertising company targeting potential customers near specific restaurants at key hours • You have a transportation service and you are searching for potential clients looking for rides near you
  39. 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Companionship Searching for the “one” • You => Diehard gamer, love playing Fortnite, looking for a companion that does the same • You search for your match based on a finite set of properties that are similar to yours • Your matches come back in a ranked list based on preferences
  40. 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. E-commerce Looking for the perfect IPA • Ratings continually collected and stored for each beer (feedback loop) • You search a catalog of beers (ABV > 7.0) and (popularity = 10) • You choose your beers, and the beer is delivered to your home
  41. 41. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kevin Fallis Senior Specialist Solutions Architect Amazon Web Services kffallis@amazon.com
  42. 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×