Instrumenting Kubernetes for Observability Using AWS X-Ray and Amazon CloudWatch (DEV303-R2) - AWS re:Invent 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Instrumenting Kubernetes for Observability
Using AWS X-Ray and Amazon CloudWatch
Christoph Kassen
Senior Solutions Architect
D E V 3 0 3

What to expect?
Modern applications
Building modern applications
Deploying to Amazon Elastic Container Service for Kubernetes
(Amazon EKS)
Monitoring
Amazon CloudWatch
Distributed tracing
Hands-on labs

What are modern applications?
Built on containers
and serverless
Microservices
architecture and
distributed

Benefits of modern applications
Improves
fault isolation
Enables continuous
delivery and
deployment
Each component
is autonomous
and independent

Public API
POST /cart
GET /cart
…
Application
/ Logic
Data store
Anatomy of a microservice

Recommender
Catalog
Cart
Order
FrontendEcosystem of microservices

Challenges with microservices
Complexity of deploying a distributed system
Service discovery
Build and deployment automation
Application configuration
Monitoring and operations
Security
Evolvability of the system

Kubernetes with modern applications
Open source Container and
microservices
platform
Hybrid &
portable

Introducing Amazon EKS
Managed Kubernetes Control Plane
Highly Available Master and etcd
Bring-your-own worker nodes (like Amazon Elastic Container Service
[Amazon ECS])
Core Tenets
Platform for enterprises to run production-grade workloads
Provides a native and upstream experience – Certified
Not forced to use additional AWS services, but offer seamless integration
Actively contribute to upstream project

Remaining challenges
Monitoring
Varying log formats across services
Collecting, aggregating, and collating logs from services
Metrics and alarms
Distributed systems
Cross-service interactions
Cascading errors
Request flow
End-user impact

Gain insights across resources and
applications by enabling observability

Getting to answers quickly means collecting and
aggregating as much data as possible
Observability
Metrics Logs Traces

Amazon Web Services (AWS) observability portfolio
Complete visibility of cloud resources
and applications
• Monitor applications
• Respond to performance changes
• Optimize resource utilization
• Get a unified view of operational health
Analyze and debug production,
distributed applications
• Identify performance bottlenecks
• Troubleshoot root cause
• Trace user requests
• For simple & complex applications

Amazon CloudWatch
Use AWS generated metrics,
logs, and events over time to
understand the behavior of your
system
Publish custom metrics, logs,
and events for your application
specific telemetry

Amazon CloudWatch
Trigger automatic
notifications based on your
own rules and metric
thresholds
AnyCompany.com

Automatically correct issues
using common actions that
you control
Define your own custom
actions based on AWS
Lambda functions for more
fine-grained control
Amazon CloudWatch

Tracing connects the dots
Get insights
into individual
operations
Discover
multiple
services
See issues
isolated within
a service
Perform root
cause analysis for
specific issue

Identify
performance
bottlenecks
How does X-Ray help?
Pinpoint specific
service issues
Identify
errors
Identify impact to
users

X-Ray concepts
user
Frontend API
Amazon
DynamoDB table
Amazon Simple
Queue Service
(Amazon SQS)

X-Ray concepts
Trace End-to-end data related a single request across services
Segments Portions of the trace that correspond to a single service
Sub-segments Remote call or local compute sections within a service
Annotations Business data that can be used to filter traces
Metadata Business data that can be added to the trace but not used for filtering traces
Errors Normalized error message and stack trace
Sampling Percentage of requests to your application to capture as traces

X-Ray SDK
Enables you to get started quickly without having to manually instrument your
application code to log metadata about requests
Source on GitHub at https://www.github.com/aws/
Available for Java, .NET, Python, Go, Ruby, and Node.js
Adds filters to automatically capture metadata for calls to:
AWS services using the AWS SDK
Non-AWS services over HTTP and HTTPS
Databases (MySQL, PostgreSQL, and Amazon DynamoDB)
Queues (Amazon SQS)

X-Ray daemon
Receives data from the SDK over UDP and acts as a local buffer; data is
flushed to the backend every second or when the local buffer fills
Available for Amazon Linux AMI, RHEL, Ubuntu, OS X, and Windows
Pre-installed on AWS Lambda
Can be run anywhere as long as AWS credentials are provided (for
example, Amazon Elastic Compute Cloud [Amazon EC2], Amazon ECS, on-
premises, developer machine, and others)

App & X-Ray
SDK
Pod
X-Ray
daemon
UDP
X-Ray API
HTTPS
HTTPS
X-Ray console
App & X-Ray
SDK
X-Ray
daemon
UDP
DevOps Team
HTTPS
X-Ray workflow
Pod Pod Pod

Sampling configuration

X-Ray API
X-Ray provides a set of APIs to enable you to send, filter, and retrieve
trace data
You can send trace data directly to the service without having to use our
SDKs (that is, you can write your own SDKs for languages not currently
supported)
Raw trace data is available using batch get APIs
You can build your own data analysis applications on top of the data
collected by X-Ray

X-Ray API
PutTraceSegments Uploads segment documents to AWS X-Ray
BatchGetTraces Retrieves a list of traces specified by ID
GetServiceGraph Retrieves a document that describes services in your application and their
connections
GetTraceSummaries Retrieves IDs and metadata for traces available for a specified time frame
using an optional filter

X-Ray pricing
Free tier
The first 100,000 traces recorded are free
The first 1,000,000 traces retrieved or scanned are free
https://aws.amazon.com/free/
Additional charges
Beyond the free tier, traces recorded cost $5.00 per million traces
Beyond the free tier, traces retrieved or scanned cost $0.50 per million traces

Key takeaways
Serverless and containers are building blocks of modern applications
AWS X-Ray is powerful tool to visualize and troubleshoot issues
Take a user-centric approach to monitor modern applications
Leverage X-Ray to debug and quantify customer impact
X-Ray works with Amazon EKS and any other Kubernetes cluster and
supports multiple languages and use cases

Service diagram

What are we going to deploy?
Tooling: AWS Command Line Interface (AWS CLI), kubectl, aws-iam-
authenticator
EKS cluster
Sample application “AnyCompany Shop“
Microservices architecture
Node.JS + Express web frontend
Python + Flask based microservices
Amazon DynamoDB, Amazon SQS
GitHub Repo https://github.com/aws-samples/reinvent2018-dev303-code
See README.md for detailed instructions

CloudWatch Logs
AWS Identity and Access Management (IAM) permissions to send logs
FluentD configuration & DaemonSet
Prometheus + Grafana
Prometheus
- Metrics storage
- Automated Kubernetes metrics collection
Grafana dashboards

What you should see

Distributed tracing
X-Ray daemon
Deployed to every EKS node
Enable our applications to send traces to AWS X-Ray
Incoming requests
Calls to supported AWS services
Outgoing HTTP requests

Deploying X-Ray to Amazon EKS
X-Ray daemon
Recieves traces from applications
Forwards traces to AWS X-Ray service
Listening on Port 2000 (TCP and UDP)
Should be present on every node
Deploy as DaemonSet
Configure listening address
Configure IAM policies
AWSXRayDaemonWriteAccess
Configure Pod (or Node)

Instrumenting appliation code
Add the X-Ray SDKs as dependency
Configure the SDK
Get IP address of X-Ray daemon via environment variable
Set up plugins to get Amazon EC2 and container information
Add tracing middleware
Allows tracing incoming requests
Middlewares for Express, Flask, ...
Add tracing to AWS SDKs
Patch libraries to trace outgoing requests

App instrumentation (Node.js)
//Add aws-xray-sdk package to package.json
const AWSXRay = require('aws-xray-sdk');
AWSXRay.config([AWSXRay.plugins.EC2Plugin,AWSXRay.plugins.ECSPlugin]);
const xrayExpress = require('aws-xray-sdk-express’);
app.use(xrayExpress.openSegment('Frontend’));
app.get('/', function(req, res)
…
app.get(‘/static', function(req, res)
app.use(xrayExpress.closeSegment());

Adding business data (Node.js)
//Example showing how to add business data to traces
app.use(function(req, res, next){
if (req.session !== undefined) {
let segment = AWSXRay.getSegment()
// User sessionID as userID
segment.addAnnotation(‘userID', req.sessionID);
}
next();
})

App instrumentation (Python)
from aws_xray_sdk.core import xray_recorder, patch_all
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware
plugins = ('EC2Plugin', 'ECSPlugin') # Plugins
# Configure recorder
xray_recorder.configure(service='recommenderservice',plugins=plugins)
# Add Xray middleware to Flask app
XRayMiddleware(app, xray_recorder)
# Patch clients, SDK‘s such as boto3, requests, ...
patch_all()

Node.JS – HTTP client instrumentation
const axios = require("axios");
const AWSXRay = require('aws-xray-sdk‘);
AWSXRay.captureHTTPsGlobal(require('http')); // Instrument http client globally
const http = require('http');
AWSXRay.capturePromise(); // Make sure we capture promises
const instance = axios.create({ httpAgent: new http.Agent() }); // Get axios instance
X-Ray Branch src/frontend/lib/ws_client.js#L15

Adding tracing to “AnyCompany Shop”
Try to add the instrumentation to the code yourself
If you need a little help: Check out the xray branch in the GitHub
repository
Pre-built containers are also available
Just replace the latest tag with xray to update the image

What it should look like

• Trace List

• Trace Overview

• Trace Detail

Links
Amazon EKS: https://aws.amazon.com/eks/
AWS X-Ray: https://aws.amazon.com/xray/
Amazon CloudWatch: https://aws.amazon.com/cloudwatch/
Blog: Application tracing on Kubernetes with AWS X-Ray

Links
Documentation: http://docs.aws.amazon.com/xray/latest/devguide/
Samples:
.NET: https://github.com/awslabs/aws-xray-dotnet-webapp
Java: https://github.com/awslabs/eb-java-scorekeep/tree/xray
Node.js: https://github.com/awslabs/eb-node-express-sample/tree/xray
Python: https://github.com/awslabs/eb-py-flask-signup/tree/xray
Lambda: https://github.com/awslabs/aws-xray-rekognition-lambda-sample
Alarms & Alerts: https://github.com/aws-samples/aws-xray-cloudwatch-event
Heatmap & Trends: https://github.com/aws-samples/aws-xray-scatter-sample

Please do not forget to clean up the
resources you created
See instructions for more details!

Thank you!
Christoph Kassen

Instrumenting Kubernetes for Observability Using AWS X-Ray and Amazon CloudWatch (DEV303-R2) - AWS re:Invent 2018

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Instrumenting Kubernetes for Observability Using AWS X-Ray and Amazon CloudWatch (DEV303-R2) - AWS re:Invent 2018

Semelhante a Instrumenting Kubernetes for Observability Using AWS X-Ray and Amazon CloudWatch (DEV303-R2) - AWS re:Invent 2018 (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Instrumenting Kubernetes for Observability Using AWS X-Ray and Amazon CloudWatch (DEV303-R2) - AWS re:Invent 2018