The document discusses Amazon Elasticsearch Service (Amazon ES) and how it can be used for log analytics. Amazon ES is a fully managed service that makes it easy to deploy, manage, and scale Elasticsearch and Kibana in AWS. It allows users to ingest and analyze log data in real time to gain valuable insights from machine-generated data. The document provides examples of how various organizations use Amazon ES for infrastructure monitoring, application monitoring, container monitoring, and security information and event management. It also covers best practices for scaling Amazon ES as data volume increases.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Scalable, secure log analytics with
Amazon ES
Carl Meadows
Principal Product Manager
AWS – Search Services
A D B 3 0 2

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Source: TechCrunch survey of popular open source software from April ’17
• Sometimes referred to as the “ELK Stack”
– Elasticsearch, Logstash & Kibana
• Distributed search and analytics engine
built on Apache Lucene
• Easy ingestion and visualization
What is Elasticsearch?

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Machine data driving Elasticsearch growth
Machine-generated data is growing 10x faster than business data… Logs, logs, and more logs
IT & DevOps: Databases,
servers, storage,
networking
Increase in IoT and mobile
devices: Gaming, sensors, web
content
Cloud-based
architectures
Source: insideBigData, “The Exponential Growth of Data,” February 16, 2017

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Application DataServer, application,
network, AWS, and
other logs
Amazon Elasticsearch Service domain
with index(es)
How it works
1. Send data as JSON via REST APIs
2. Data is indexed—all fields searchable, including nested
JSON
3. Queries, via REST APIs, allow fielded matching,
Boolean expressions, include sorting and analysis
1
2
3
Application users, analysts, DevOps, security

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Elasticsearch Service is a fully
managed service that makes it easy to
deploy, manage, and scale Elasticsearch
and Kibana in the AWS Cloud
Amazon Elasticsearch Service

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Cloud
Elasticsearch runs on a cluster of instances
VPC
Data nodes Master nodes
AWS Management Console
AWS Command Line Interface
(AWS CLI)
AWS tools and SDKs
AWS CloudFormation
AWS Identity and
Access
Management (IAM)
Elastic Load Balancing (ELB)
AWS CloudTrailAmazon CloudWatch AWS Database
Migration Service
Amazon Kinesis Data
Firehose
Amazon
CloudWatch
Logs
Amazon ES domain

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Infrastructure monitoring: OS transport
• ELK: Elasticsearch, Logstash, and Kibana
• Beats: lightweight log shippers
• For high-volume writes, use a buffer—Redis in this case
Logstash on
Amazon EC2
Amazon EC2 Amazon ElastiCache
for Redis
Amazon Elasticsearch
Service

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Infrastructure monitoring/SIEM: AWS-centric
• Employing streaming technologies at higher volumes
• Some customers use Amazon CloudWatch Logs
• Many customers use self-managed Kafka with Beats for shipping
Amazon EC2/
Kinesis Agent/
Beats/
etc.
Amazon Elasticsearch
Service
Amazon Managed
Streaming for Kafka
Amazon Kinesis
Data Streams
Amazon EMR

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Metrics
Logs
Container monitoring
• App containers, pods, system, nodes
• Kubernetes events, unavailable pods
• Application logs & metrics
• System logs & metrics
• Cluster capacity, performance, network traffic
• Ad hoc analysis & troubleshooting
VPC
Amazon Elastic Container
Service for Kubernetes
Metricbeat
Fluentd
Elastic network
interface
Elastic network
interface
Elastic network
interface
VPC
Amazon Elasticsearch
Service
Health & performance monitoring

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Log analytics flow
Data producers Collection
Permanent
cold storage
Transformation
Analysis and
reporting

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Collection on the hosts under study
• Simple setup
• Tail log files
• Forward to Kinesis
Data Streams or
Kinesis Data
Firehose
• More complex
• Transform or
enrich data before
shipping
• Send to Kinesis
Data Streams
• Lightweight log
shipper
• Send to Kinesis
Data Firehose
Kinesis Agent KPL Beats

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Create time-based indexes for log analytics
• You use a root string (e.g. logs_).
• Depending on volume, rotate at regular
intervals—normally daily.
• Daily indexes simplify index management.
Delete the oldest index to create more
space on your cluster.
logs_11.26.2018
logs_11.25.2018
logs_11.24.2018
logs_11.23.2018
logs_11.22.2018
logs_11.21.2018
logs_11.20.2018

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
You use the query APIs to retrieve data from
Elasticsearch
Amazon ES Domain
Query
engine
Scoring/
sorting
Ranked
resultsMatches

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The query engine matches requested field values
Field1:value1
Field2:value2
logs_11.28.2018 index
F1 index F2 index
V1
V2
Vn
V1
V2
Vn
ID
Field: value
Field: value
Field: value
Field: value

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
You use aggregations to analyze log data
Amazon ES Domain
Query
engine
Matches
Analysis
engine
(aggregations)
• Histogram
• Numeric – sum,
min., max.
• Terms – bucketing
• Nesting

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Kibana is a real-time visualization tool

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Security
You interact with an endpoint – DNS
Amazon Cognito for authentication and IDPs
Amazon Virtual Private Cloud (Amazon VPC) for restricting to your IP address space
AWS Identity and Access Management (IAM) to control Elasticsearch and Amazon ES APIs
AWS Key Management Service (AWS KMS) for encryption at rest
TLS for encryption in flight
Instance
storage

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Data is stored in indexes, distributed across shards
All docs
1/51/5 1/5 1/5 1/5
Index
ID
Field: value
Field: value
Field: value
Field: value

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Shards are primary or replica
Index
Primary shards
Replica shards
ID
Field: value
Field: value
Field: value
Field: value

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Elasticsearch distributes shards to data nodes
Queries
Updates

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
You set the basic parameters that control scale
Deploy instances based on
storage and compute needs
Add instances for increased
parallelism
Instance type Instance count
Index data (primary and replica
shards) is stored on disk
Shards are the units of work
and storage
Storage Shard count

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Scaling
Storage needed =
Source/day * 1.1 * 2 * 7 * 1.15
• For example: 1 TB daily of source
data needs 17 TB of storage for 7
days
Number of CPUs ~=
1.6 * active shards
• For example: 4 data streams @ 1
TB daily means 40 total shards
(20 primary and 20 replica)
active, so make sure to have 64
total CPUs
• This is almost certainly wrong

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Collection and buffering limit concurrency
• One Kinesis Data Firehose with
a 5 MB buffer
• 5000 1K records per
connection
• Number of connections driven
by flush rate
• Agents on the hosts
connecting directly
• 5000 connections per 5000 1K
records
• Buffering on the hosts helps,
connections driven by host
count

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
T-shirt sizes
T-shirt
size
Data
(per day)
Storage
needed
Active shards
(maximum)
Total shards
(maximum)
Instances
XSmall 10 GB 177 GB 4 300 2x M5/R5.large data
Small 100 GB 1.7 TB 8 600 4x M5/R5.xlarge data
Medium 500 GB 8.5 TB 30 3000 6x I3.2xlarge data
Large 1 TB 17.7 TB 60 3000 6x I3.4xlarge data
XLarge 10 TB 177.1 TB 600 5,000 30x I3.8xlarge data
Huge 80 TB 1.288 PB 3400 25,000 85x I3.16xlarge data

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon ES additional tasks
Deployment Security
• Monitoring metrics
via Amazon
CloudWatch
• Audit via AWS
CloudTrail
• One-click version
upgrades
• One-click security
patches
• Backups
Operations/governance
• Deploy in minutes
• Scale seamlessly
• Node roles: data and
master
• High availability
• Kibana included
• Private networking
• Encryption at rest
and in flight
• HIPAA and PCI
compliance

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Security information and event management (SIEM)
IoT & mobile
Application monitoring & root-cause analysis
Business and web analytics
Amazon ES empowers you with the data to
understand and intelligently react to your business
needs
• End-to-end visibility – Better understanding of
customers’ behavior to improve user experience
and react to demand
• Improve reliability – Increased operational
efficiencies by identifying, solving, and preventing
system failures in real time
• Faster time to value – Accelerate time to market
with application delivery and performance
monitoring
• Security - Improved business confidence with end-
to-end monitoring of data, infrastructure, and
transactions
Build actionable insights

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Central log management system
https://www.youtube.com/watch?v=fSjAfp-uqSs
Case study: Autodesk
Highly distributed organization—no consistent way to
collect and measure metrics.
Small ops team.
Must integrate easily with other AWS services.
Scale – Accommodate current and future requirements.
Must be cost effective with no data lock-in.
TBs of log data to sift through to find and fix customer-
impacting issues.
C h a l l e n g e
B E N E F I T S
Unified log data management solution built on AWS. Single interface
for log analytics across applications. Annotate log records to enable
distributed tracing states.
Streaming application logs via Kinesis Data Firehose to Amazon
S3/Amazon Athena and Amazon ES
10 i3.4xlarge Amazon ES data nodes – 33 TB. Will grow to 110 TB
Kibana, built in within Amazon ES, for near-real-time analytics and
dashboards
S o l u t i o n
All managed services—“manage less to gain more.” Focus on developing awesome products.
Common vocabulary for diagnosing and solving problems. Eliminated silos.
Scalable and cost-effective—i3s delivering great value per TB.
Improving customer experience by reducing the time to find and fix customer issues.

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Wrap-up
• Machine-generated data is growing rapidly, driven by Dev Ops, cloud infrastructure,
and IoT
• Logs contain valuable insights—what your users are doing, whether you have bad
actors, what’s happening at your devices
• Amazon ES enables ingesting and analyzing logs in real time to provide you with the
data and insights you need

29. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Carl Meadows
carlmead@amazon.com