Designing and Implementing a Serverless Media Processing Workflow Using AWS Step Functions - SRV333 - re:Invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SRV333 - Designing and
Implementing a Serverless Media
Processing Workflow Using AWS
Step Functions
A n g e l a W a n g
B r i t t a n y D o n c a s t e r
AWS re:INVENT
N o v e m b e r 2 9 , 2 0 1 7

Part one
Why AWS Step Functions?
Types of states
Workshop overview

λλ
λ DBMS
λ
λ
λ
λ
λ
λ λ
λ
λ
Queue
Modern
app

Modern
app

Step Functions overview
“I want to sequence functions”
“I want to select functions based on data”
I want to run functions in parallel
“I want to retry functions”
“I want to try/catch/finally”
“I want to run code for hours”

Step Functions overview
Visualize in the
console
Define in JSON Monitor
executions

{
"Comment": "A Hello World Example",
"StartAt": "HelloWorld",
"States": {
"HelloWorld": {
"Type": "Task",
"Resource": "${lambdaArn}",
"End": true
}
}}
AWS::StepFunctions::StateMachine

Seven state types
Task A single unit of work
Choice Adds branching logic
Parallel Fork and join the data across tasks
Wait Delay for a specified time
Fail Stops an execution and marks it as a failure
Succeed Stops an execution successfully
Pass Passes its input to its output

What you will build in this workshop
• Multi-step state machine that
leverages several SFN features:
• Lambda task state
• Choice state
• Custom error catching
• Parallel state
• Input/output manipulations
using InputPath and ResultPaths
Task
Choice
Parallel
Fail

Hands-on: Part one (step 0–1C)
• Link to instructions: http://amzn.to/sfn-workshop
What part one will look like when finished:
What you’ll do:
• Run AWS CloudFormation script
• Create state machine using step-easy
tool
• Deploy single-step state machine using
the console
• Test using the console

Part one review

ExtractImageMetadata
ImageTypeCheck
TransformMetadata
Rekognition Thumbnail
StoreMetadata
Start
End
Synchronous task
Task: a single unit of work
Synchronous: Lambda
{
"ExtractImageMetadata": {
"Type": "Task",
"Resource": "arn:aws:lambda:mars-ter …”,
…
}
}

ExtractImageMetadata
ImageTypeCheck
TransformMetadata
Rekognition Thumbnail
StoreMetadata
Start
End
Task: a single unit of work
Asynchronous: “Activity Task”, Polled by workers
{
"ExtractImageMetadata": {
"Type": ”Task",
"Resource": "arn:aws:states:mars-ter …”,
…
}
}
Asynchronous task

API actions on activity task
Register Activity task—returns ARN
Poll For task (by ARN)
Report Success
Report Failure
Report Heartbeat

Part two
Input/output processing
Branching
Custom error handling
Parallel state

State [x]
Input
Output

State [x]"InputPath" "ResultPath" "OutputPath"
Input
Output

InputPath example
{
”s3key”:”apple.png”,
”timestamp”:12324
}
Input

{
}
"InputPath”:“$”
Input
InputPath example

{
}
State [x]"InputPath”:“$”
Input
Input received by state x:
{
}
InputPath example

"InputPath”:“$.s3key”
{
}
Input
{
}
InputPath example

State [y]"InputPath”:“$.s3key”
{
}
”apple.png”
Input
Input received by state y:
{
}
InputPath example

{
}
Input
State [x]
State output:
{“type”: “png"}
"InputPath”
: “$.s3key”
”ResultPath”:“$”
ResultPath example

{
}
Input
State [x]
State output:
"InputPath”
: “$.s3key”
”ResultPath”:“$”{ ”type”:”png}Output
ResultPath example

{
}
Input
State [x]
State output:
"InputPath”
: “$.s3key”
”ResultPath”:“$.metadata”
ResultPath example

{
}
Input
State [x]
State output:
"InputPath”
: “$.s3key”
”ResultPath”:“$.metadata”
{
”timestamp”:12324,
“metadata”: {
“type”: “png"}
}
Output
ResultPath example

Branching logic
“ExampleChoiceState": {
"Type": ”Choice",
”Choices": [{
”Variable": ”$.imageformat",
”StringEquals": ”JPG",
”Next": “ParalellProcessing”
}, …
],
”Default": ”NotSupported",
…
}
Choice state

“ExampleTaskState": {
"Type": "Task",
"Resource": “<Lambda Arn>",
"Next": "Graceful Exit",
"Retry": [
{
"ErrorEquals": [ “ErrorA" ,”ErrorB”],
"IntervalSeconds": 2,
"MaxAttempts": 2,
"BackoffRate": 1.5
} ,
{
“ErrorEquals” : [ “ErrorC” ],
“Interval Seconds” : 5
],
"Catch": [
{
"ErrorEquals": [ "States.ALL" ],
"Next": "Clean Up" }
]
},
Custom error handling

Saga pattern to roll back transactions
credit: http://theburningmonk.com/2017/07/applying-the-saga-pattern-with-aws-lambda-and-step-functions/

“P": {
"Type": "Parallel",
"Branches": [
{
"StartAt": “P1",
"States": {
“P1": {
"Type": "Task",
"End": true } }
},
{
"StartAt": “P2",
"States": {
“P2": {
"Type": "Task",
"End": true } }
}
],
"Next": "NextState"
}
Parallel states

Hands-on: Part two (step 1D–3)
What part two will look like when finished:
What you’ll do:
• Add ResultPath
• Add choice state
• Add custom error handling
• Add parallel state
• Test using the console

Demo part two

Part two review

Part three
Other architecture patterns

Video on Demand—AWS Quick Start
https://docs.aws.amazon.com/solutions/latest/video-on-demand

Use activity task for manual approval
Step Functions
Poll:
taskToken
Amazon
SES
Poller
Approver
Send success:
taskToken
API
Gateway
https://aws.amazon.com/blogs/compute/implementing-serverless-manual-
approval-steps-in-aws-step-functions-and-amazon-api-gateway/

while:
sleep (30)
if status=done:
break
Use Wait + Choice to wait for batch job/async
APIs

Integrate with other AWS services
• Create state machines and activities with AWS CloudFormation
• Call AWS Step Functions with Amazon API Gateway
• Start state machines in response to events, or on a schedule, with Amazon
CloudWatch Events
• Monitor state machine executions with Amazon CloudWatch
• Log API calls with AWS CloudTrail
• Start state machine using AWS Lambda

Hands-on: Part three (step 3–end)
What you’ll do:
• Add a final task state to persist the metadata
• Configure the trigger for the state machine
• Deploy a test web application and test
Optional bonus tasks: (if finished early and want to keep going)
Add in additional error handling/retry logic
Add a manual approval task if inappropriate content is found
A state machine to clean up when images are deleted

Discussion
• Other ideas for extending the state machine?
• Ideas on how you can use Step Functions

Q&A

Learn more
Thu. Nov. 30
2:30–5:00
ARC326: Create a Serverless Image Processing Platform
Thu. Nov. 30
5:30–6:30
SRV306: State Machines in the Wild! How Customers use AWS Step Functions
Meet experts at the Serverless Kiosk in the Expo
Resources: https://aws.amazon.com/step-functions/reinvent

What are you building? #stepfunctions
Serverless image recognition and processing backendEBS snapshot management
https://console.aws.amazon.com/states/

THANK YOU!
h t t p s : / / a w s . a m a z o n . c o m / s t e p - f u n c t i o n s / g e t t i n g - s t a r t e d

Designing and Implementing a Serverless Media Processing Workflow Using AWS Step Functions - SRV333 - re:Invent 2017

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Designing and Implementing a Serverless Media Processing Workflow Using AWS Step Functions - SRV333 - re:Invent 2017

Semelhante a Designing and Implementing a Serverless Media Processing Workflow Using AWS Step Functions - SRV333 - re:Invent 2017 (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Designing and Implementing a Serverless Media Processing Workflow Using AWS Step Functions - SRV333 - re:Invent 2017