In this session, we cover three common scenarios that include Amazon CloudWatch Logs and AWS Lambda. Learn how to build an Elasticsearch cluster from historical data using Amazon S3, Lambda, and CloudWatch Logs. Discover how to add details to CloudWatch alarm notifications using Amazon SNS and Lambda. Finally, understand how to bring Elastic Load Balancing logs to CloudWatch Logs using S3 bucket triggers from Lambda. Speaker: Leo Zhadanovsky, Principle Solutions Architect. Amazon Web Services Level: 300

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudWatch Logs and AWS Lambda:
A Match Made in Heaven
Leo Zhadanovsky
Principal Solutions Architect, AWS
September 12, 2017

2. What to Expect from the Session
Look at industry trends
impacting monitoring
Learn about Amazon
CloudWatch and Amazon
CloudWatch Logs
Understand several key
monitoring use cases
See Amazon CloudWatch
and AWS Lambda in action

3. Centralize
Centralize logs from
Elastic Load Balancing
(ELB) using S3 bucket
triggers
Customize
Customize alarms from
Amazon CloudWatch to fit
your specific needs
Analyze
Build an on-demand,
scalable Amazon
Elasticsearch Service
cluster to solve a specific
problem or to do analysis
What to expect: Scenario preview

4. Recognize this?

5. Day in the life!
The story you are about to hear is true (mostly)…Only the
names have been changed to protect the innocent…

6. A customer writes a high severity ticket – your Application,
ImportantApp, is down.
John, the on-call developer, is paged through the ticketing
system.
None of your alarms fired.
Blissful ignorance

7. John engages and starts to scan service dashboards.
He does see intermittent availability impact, but doesn’t
know how to assess impact to customers or where to begin
troubleshooting.
He decides to escalate to a manager on call.
Confusion

8. More customer tickets are pouring in.
An escalation manager, Jane, joins the event and starts to
assess the situation and impact.
John and Jane’s CTO happens to notice the problem.
Sends Jane an IM – “Jane, what’s going on with
ImportantApp?”
Stress

9. Jane and John recall a recent issue where certain
customers started to issue “expensive” operations.
John starts log diving on their production hosts.
John identifies a suspect customer. Jane cuts a ticket and
John prepares a configuration change to block the
customer.
False hope

10. The other team engages and indicates they didn’t change
anything.
Jane and John also confirm this when the availability
impact persists after deploying the configuration change.
Out of ideas, John suggests to fail over to the standby – “It
can’t hurt…”
Desperation

11. After the failover, ImportantApp recovers. (Yay!)
Our root cause deep dive finds that a new JDBC version
introduced a memory leak leading to Java heap
exhaustion.
We fix the leak, add new alarms on memory usage, and
tune our service alarms.
Enlightenment

12. Can we do better than that?

13. Day in the life - Reflection
• We have missing alarms, and some of the alarms we
have are not actionable.
• We don’t always have the right logs and interacting with
them can be tedious.
• Our dashboards do not tell us enough about customer
impact or behavior changes.
Monitoring is really (really) hard!

14. Trends in monitoring

15. Trend: Complexity increasing
• Distributed microservices based applications
• Applications are written in different languages and
frameworks
• Workloads are increasingly running on transient
resources such as containers and serverless compute
• Specialization in persistence tier

16. • Small changes are continuously built, tested, and
deployed
• As the scale and design of applications are changing
rapidly, so are the infrastructure needs
• Applications are global and customer behavior is
unpredictable
• Increased role of automation
Trend: Applications are more dynamic

17. • Increased role of applications in business outcomes
(revenue, cost, SLA)
• Rapidly evolving applications are required to gain
competitive advantage
• Increased expectations from customers
Trend: More business impact

18. • Monitoring is no
longer standalone.
It is one aspect of
management and
increasingly
integrated into the
lifecycle of the
application
Trend: Integrated management

19. Amazon CloudWatch
See React Diagnose Resolve

20. Use AWS generated metrics,
logs, and events over time to
understand the behavior of
your system
Publish custom metrics,
logs, and events for your
application specific
telemetry
See React Diagnose Resolve

21. Trigger automatic
notifications based on
your own rules and
metric thresholds
See React Diagnose Resolve

22. Inspect, navigate, zoom, and
correlate across time to
investigate issues
Jump to your logs directly
from your metrics to perform
searches or generate
additional metrics from log
data
See React Diagnose Resolve

23. Easily and automatically
correct issues using common
actions that you control
Define your own custom
actions based on Lambda
functions for more fine-grained
control
See React Diagnose Resolve

24. • Metrics price drop
• More metrics, logs, events from
AWS services: AWS CloudTrail,
AWS Elastic Beanstalk, Amazon
SES
• Simple navigation from metrics to
your logs
• Upgraded metric retention from 2
weeks to up to 15 months
Recent improvements

25. • Support for arbitrary metric percentiles
• collectd output plugin to simplify metric
collection
• Improvements in dashboards (new widgets,
dark theme, Y axis limits)
• Improved Logs console experience
• 1 Second Metrics Resolution
• How to treat missing data
Recent improvements (continued)

26. Not just about what’s inside CloudWatch
• Monitoring is hard (very hard)
• Every enterprise, team and situation has unique needs
• We have a rich partner community
• We give you the tools and flexibility to integrate with
other AWS services

27. Flexible
Egress
Comprehensive
Ingress
Amazon EC2
AWS CloudTrail
Amazon ECS
AWS Lambda
Amazon ES
Amazon S3
AWS Lambda
AMAZON
CLOUDWATCH LOGS
Centralize
Search
Monitor
Stream
Retain
Secure
A log service for arbitrary scale,
availability, durability, and security
AWS Kinesis
Custom

28. Serverless compute: AWS Lambda
COMPUTE
SERVICE
EVENT
DRIVEN
Run arbitrary
code without
managing
servers
Code only runs
when it needs to
run

29. Centralize
Centralize logs from
Elastic Load Balancing
(ELB) using Amazon S3
bucket triggers
Customize
Customize alarms from
Amazon CloudWatch to fit
your specific needs
Analyze
Build an on-demand,
scalable Amazon
Elasticsearch Service
cluster to solve a specific
problem or to do analysis

30. Problem statements
• Log data is scattered on instances and Amazon S3
buckets
• It would be better if it were centralized in CloudWatch Logs
for searching and filtering
• Today CloudWatch provides an agent for instance logs;
what about Amazon S3 delivered logs?

31. Flow of events
LambdaS3ELB CloudWatch
Logs
ELB logs
archived in
Amazon S3
S3 Object Create
Event Notification
to AWS Lambda
AWS Lambda
reads the ELB log
from Amazon S3
and publishes to
CloudWatch Logs

32. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Provision three Amazon EC2 instances running Apache

33. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Create a load balancer

34. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Create a target group with the three Apache servers

35. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Enable ELB log delivery to Amazon S3

36. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Configure the Lambda function to trigger on S3 object create

37. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Read the ELB logs from the S3 bucket by invoking S3 GetObject API

38. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Post the logs into CloudWatch Logs by invoking putLogEvents SDK API

39. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
ELB serving traffic

40. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Verify that ELB logs are delivered to S3

41. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Access the ELB logs from CloudWatch Logs

42. Apache Servers
ELB & Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Search for the HTTP GET calls in the last 24 hours

43. Apache Servers
ELB& Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
Create a filter pattern to extract requests with a latency of more than 1 ms

44. Define a metric filter on the log group
Apache Servers
ELB& Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch

45. Apache Servers
ELB& Lambda
Configured
Lambda Triggered
ELB logs in
CloudWatch
View the metric graph for ELB latency
HighLatencyCount

46. Key takeaways
1. Amazon S3 delivered log data from any source can be
centralized into CloudWatch Logs using Lambda
2. You can search and extract metrics from those logs in
near real time

47. Centralize
Centralize logs from
Elastic Load Balancing
(ELB) using S3 bucket
triggers
Customize
Customize alarms from
Amazon CloudWatch to fit
your specific needs
Analyze
Build an on-demand,
scalable Amazon
Elasticsearch Service
cluster to solve a specific
problem or to do analysis

48. Problem Statements
• When you get an alarm, you want enough information
to decide whether it needs immediate attention or not
• You want to customize the alarm text and format to your
operational needs

49. Flow of Events
Error in Logs
generates an alarm to
an SNS topic
Amazon SNS
topic triggers an
Event Notification
to Lambda
AWS Lambda gets
the filtered logs,
calls SES to email
with those logs
Amazon SESAWS LambdaCloudWatch
Logs
CloudWatch
Alarms

50. Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email
Provision an EC2 instance with Tomcat running on it
Hello World

51. CloudWatch agent sends EC2 instance logs to CloudWatch Logs
Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email

52. Define a filter pattern to extract unauthorized access attempts
Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email

53. Define a metric filter on the log group
Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email

54. Define an alarm with a specific threshold for that metric
Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email
test

55. Configure the Lambda function to trigger on SNS topic message
Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email
Hello World

56. Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email
Get the metric filter information by invoking the describeMetricFilters SDK API

57. Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email
Get the relevant log data by invoking the filterLogEvents SDK API

58. Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email
SES sends an email with the relevant log data

59. Unauthorized access attempts on the Tomcat server
Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email

60. Alarm generated due to unauthorized access attempts
Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email
test
test
test
test

61. View email with the relevant log data about the alarm
Metric Filter
Defined
Alarm & Lambda
Configured
Lambda Triggered SES sends Email
test
test
Hell
o
test

62. Key takeaways
• Alarms can be customized to add specific details about
the issue
• When you see a spike on a metric, you can also get the
logs describing the issue triggering the alarm
• The Lambda function can be extended to add your
specific information to the alarm

63. Centralize
Centralize logs from
Elastic Load Balancing
(ELB) using S3 bucket
triggers
Customize
Customize alarms from
Amazon CloudWatch to fit
your specific needs
Analyze
Build an on-demand,
scalable
Amazon Elasticsearch
Service cluster to solve a
specific problem or to do
analysis

64. Problem Statements
• You want to do log analysis using Amazon Elasticsearch
Service but don’t want to leave the cluster running all the
time
• You want to send data to Amazon Elasticsearch Service,
but don’t want to manage ongoing operations
• Build an on-demand Elasticsearch cluster from historical
data

65. Flow of events
LambdaS3
CloudWatch
Logs
CloudWatch Logs in
a specific timeframe
exported to S3
S3 Object Create
Event Notification
to Lambda
AWS Lambda
transforms and sends
only filtered logs to
Amazon Elasticsearch
Service
Amazon
ES

66. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
Configure flow logging for your VPC

67. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
View the VPC Flow Logs in CloudWatch Logs

68. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
Create an Amazon ES domain

69. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
Read the VPC Flow Logs from the S3 bucket by invoking GetObject API

70. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
Transform the VPC Flow Logs into a JSON document for Amazon ES

71. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
Ingest the logs into Amazon ES by putting to its HTTP endpoint

72. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
Export the VPC Flow Logs to Amazon S3

73. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
VPC Flow Logs exported to Amazon S3

74. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
View the VPC Flow Logs in Kibana

75. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
View different dashboards in Kibana

76. Logs exported to
S3
Lambda
Configured
Logs to
Elasticsearch
Visuals in Kibana
View different dashboards in Kibana

77. Key takeaways
• Send historical data within a timeframe to Amazon
Elasticsearch Service on demand
• This reduces cost, burden of scalability, and operations
time
• Troubleshooting gets easier because you have only
limited and relevant data

78. Recap
• Monitoring is more important than ever, but still too hard
• Amazon CloudWatch is working to make monitoring
easier
• Amazon CloudWatch Logs and AWS Lambda are
powerful tools for tailoring your monitoring for your
business needs

79. Useful Links
• CloudWatch Overview - https://aws.amazon.com/cloudwatch/
• Documentation - https://aws.amazon.com/documentation/cloudwatch/
• CloudWatch Blog - https://aws.amazon.com/blogs/aws/category/amazon-
cloud-watch/
• Lambda functions used in the demo scenarios
Centralize - https://github.com/awslabs/cloudwatch-logs-centralize-logs
Customize - https://github.com/awslabs/cloudwatch-logs-customize-alarms
Analyze - https://github.com/awslabs/cloudwatch-logs-analyze-data

80. Remember to complete
your evaluations!

81. Thank you!