Presentation from the developer track at I Love APIs London 2016 featuring Matt McClean, Amazon Web Services.
Developers have been jumping on the microservices bandwagon because of the obvious benefits of faster release cycles and innovation. However, microservices' downside is the increased server costs, operational costs, and performance costs. To reduce this complexity, Amazon Web Services created AWS Lambda - a compute platform that lets you build microservices with no provisioning and servers.
Matt McClean, Solution Architect from AWS, presents how to use AWS Lambda to build your microservices. He covers various architectural patterns and anti-patterns for using AWS Lambda.
2. What to expect from this session
• Evolution of Cloud Compute toward Serverless
• What is AWS Lambda?
• Building a serverless data processing architecture
• Real-time data processing architecture patterns
• Tips and tricks for using AWS Lambda
• Live demo of Apigee + AWS Lambda app
14. SOURCE of
data
Fleet of servers poll for
changes
Listening to source Pushes to queue Pull off queue
Fleet of servers act as workers
to process the data
Auto-scale
worker nodes
to adjust with
load
S3 objects
Amazon Kinesis records
DynamoDB tables
SNS messages
IoT devices
…
Cross-AZ
replication for
high
availability
Load balancer
to distribute
traffic
Data Processing Architecture with Servers
Application code
15. SOURCE of
data
Attach a Lambda function
And that’s it!
Lambda does
Listening/polling
Queuing
Auto scaling
Redundancy
Load balancing
Data Processing Architecture Without Servers
Application code
18. Example – Video Clip Transcode
// Transcode video files uploaded to an S3 bucket
var async = require('async');
var fs = require('fs');
var exec = require('child_process').exec;
var aws = require('aws-sdk');
var s3 = new aws.S3();
// Lambda entry point
exports.handler = function(event, context) {
var inputVideo = '/tmp/myVideo.avi';
var outputVideo = '/tmp/myVideo.mp4';
var inputBucket = event.Records[0].s3.bucket.name;
var inputKey = event.Records[0].s3.object.key;
var outputBucket = inputBucket + '-out';
var outputKey = inputKey.substr(0,inputKey.length-4) + '.mp4';
// Runs the array of functions in series, each passing their results to the next
async.waterfall([
Set up variables
Serialize steps
19. Example – Video Clip Transcode
// Download the object from S3 into a buffer
function download(next) {
s3.getObject({Bucket: inputBucket, Key: inputKey}, next); },
// Write the buffer to a file
function write(response, next) {
fs.writeFile(inputVideo, response.Body, next); },
// Spawn a process to transcode
function transcode(next) {
exec("./ffmpeg -i " + inputVideo + " -vcodec mpeg4 -b:v 1200k “
+ outputVideo, next); },
// Read the file to a buffer
function read(next) {
fs.readFile(outputVideo, next); },
// Upload the buffer to an object in S3
function upload(objectBuffer, next) {
s3.putObject({Bucket: outputBucket, Key: outputKey, Body: objectBuffer},
context.done); },
], context.fail);
};
Get file from S3
Write to disk
ffmpeg transcode
Read from disk
Upload to S3
20. Sample Real-Time Stream Processing Workflow Using Amazon
Kinesis & AWS Lambda
Amazon Kinesis AWS Lambda
Amazon Redshift
Amazon SNS
New records
available
21. Sample Real-Time DB Triggers Workflow Using Amazon
DynamoDB & AWS Lambda
New data
available
Amazon DynamoDB AWS Lambda
Amazon
DynamoDB
Amazon Redshift
22. Sample Real-Time Message Handling Workflow Using
Amazon SNS & AWS Lambda
New message
published
Amazon SNS AWS Lambda
Amazon SNS
Amazon Kinesis
23. Sample CRUD Backend Workflow Using Amazon API
Gateway & AWS Lambda
New API
called
Amazon API Gateway AWS Lambda
Amazon S3
Amazon
DynamoDB
24. More Triggers for AWS Lambda
AWS
CloudFormation
Custom Resources
Amazon SES
Actions
Amazon Cognito
Sync Triggers
… and the list will
continue to grow!
AWS IoT ActionsAWS Lambda
Scheduled Events
Amazon Echo
Skills
Amazon SWF Tasks
26. Using Lambda to Audit CloudTrail Activity
AWS
Lambda
Amazon S3Amazon CloudTrail
Amazon SNS
AWS IAM
27. Using Lambda for Automated Infrastructure Management
AWS
Lambda
Amazon
SNS
Amazon
CloudWatch Alarm
ec2
runInstance
ecs
startTask
beanstalk
updateApp
kinesis
splitShard
Any API call
https://aws.amazon.com/blogs/compute/scaling-amazon-ecs-services-automatically-using-amazon-cloudwatch-and-aws-lambda/
Amazon
CloudWatch Event
28. Using Lambda to Forward AWS Events to Slack
AWS
Lambda
Amazon
SNS
Amazon
CloudWatch Events
Auto Scaling
Slack
https://aws.amazon.com/blogs/aws/new-slack-integration-blueprints-for-aws-lambda/
29. Using Lambda to Deploy Lambda Functions
https://aws.amazon.com/blogs/compute/dynamic-github-actions-with-aws-lambda/
AWS
Lambda
Amazon
SNS
GitHub Repo
lambda
createFn ()
30. Using Lambda in your software delivery pipeline
AWS
Lambda
AWS
CodeCommit
AWS
CodePipeline
https://aws.amazon.com/about-aws/whats-new/2016/01/aws-codepipeline-
adds-support-for-triggering-aws-lambda-functions/
31. A Few Other Tips and Tricks for Extending AWS Lambda
Functionality
Ø Use /tmp space as cache
Ø Run arbitrary binaries
Ø Use Grunt and Jenkins plugins for deploying
Ø Build event-driven Amazon ECS
Ø Try out Serverless Framework