SRV328 Designing and Implementing a Serverless Media-Processing Workflow

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A n g e l a W a n g
Solutions Architect, Amazon Web Services
SRV328
Designing and Implementing a Serverless
Media Processing Workflow Using AWS Step
Functions

Part one
Why AWS Step Functions?
Types of states
Workshop overview

λλ
λ DBMS
λ
λ
λ
λ
λ
λ λ
λ
λ
Queue
Modern
app

Modern
app

Step Functions overview
“I want to sequence functions”
“I want to select functions based on data”
I want to run functions in parallel”
“I want to retry functions”
“I want to try/catch/finally”
“I want to run code for hours”

Step Functions overview
Visualize in the
console
Define in JSON Monitor
executions

AWS::StepFunctions::StateMachine
{
"Comment": "A Hello World Example",
"StartAt": "HelloWorld",
"States": {
"HelloWorld": {
"Type": "Task",
"Resource": "${lambdaArn}",
"End": true
}
}}

Seven state types
Task A single unit of work
Choice Adds branching logic
Parallel Fork and join the data across tasks
Wait Delay for a specified time
Fail Stops an execution and marks it as a failure
Succeed Successfully stops an execution
Pass Passes its input to its output

What you will build in this workshop
• Multi-step state machine that
leverages several SFN features:
• AWS Lambda task state
• Choice state
• Custom error catching
• Parallel state
• Input/output manipulations using
InputPath and ResultPaths
Task
Choice
Parallel
Fail

Hands-on: Part one (step 0–1C)
• Link to instructions: http://amzn.to/sfn-workshop
What part one will look like when finished:
What you’ll do:
• Run AWS CloudFormation script
• Deploy single-step state machine using
the console
• Test using the console

Part one review

ExtractImageMetadata
ImageTypeCheck
TransformMetadata
Rekognition Thumbnail
StoreMetadata
Start
End
Synchronous task
Task: A single unit of work
Synchronous: Lambda
{
"ExtractImageMetadata": {
"Type": "Task",
"Resource": "arn:aws:lambda:mars-ter …”,
…
}
}

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ExtractImageMetadata
ImageTypeCheck
TransformMetadata
Rekognition Thumbnail
StoreMetadata
Start
End
Task: A single unit of work
Asynchronous: “Activity Task”, Polled by workers
{
"ExtractImageMetadata": {
"Type": ”Task",
"Resource": "arn:aws:states:mars-ter …”,
…
}
}
Asynchronous task

Part two
Input/output processing
Branching
Custom error handling
Parallel state

Input/Output processing
State [x]
Input
Output

Input/Output processing
State [x]"InputPath" "ResultPath" "OutputPath"
Input
Output

InputPath example
{
”s3key”:”apple.png”,
”timestamp”:12324
}
Input

{
}
"InputPath”:“$”
Input
InputPath example

{
}
State [x]"InputPath”:“$”
Input
Input received by state x:
{
}
InputPath example

InputPath example
"InputPath”:“$.s3key”
{
}
Input
{
}

InputPath example
State [y]"InputPath”:“$.s3key”
{
}
”apple.png”
Input
Input received by state y:
{
}

{
}
Input
State [x]
State output:
{“type”: “png"}
"InputPath”
: “$.s3key”
“ResultPath”:“$”
ResultPath example

{
}
Input
State [x]
State output:
"InputPath”
: “$.s3key”
“ResultPath”:“$”{ ”type”:”png}Output
ResultPath example

{
}
Input
State [x]
State output:
"InputPath”
: “$.s3key”
”ResultPath”:“$.metadata”
ResultPath example

{
}
Input
State [x]
State output:
"InputPath”
: “$.s3key”
”ResultPath”:“$.metadata”
{
”timestamp”:12324,
“metadata”: {
“type”: “png"}
}
Output
ResultPath example

Branching logic
“ExampleChoiceState": {
"Type": ”Choice",
”Choices": [{
”Variable": ”$.imageformat",
”StringEquals": ”JPG",
”Next": “ParalellProcessing”
}, …
],
”Default": ”NotSupported",
…
}
Choice state

Custom error handling
“ExampleTaskState": {
"Type": "Task",
"Resource": “<Lambda Arn>",
"Next": "Graceful Exit",
"Retry": [
{
"ErrorEquals": [ “ErrorA" ,”ErrorB”],
"IntervalSeconds": 2,
"MaxAttempts": 2,
"BackoffRate": 1.5
} ,
{
“ErrorEquals” : [ “ErrorC” ],
“Interval Seconds” : 5
],
"Catch": [
{
"ErrorEquals": [ "States.ALL" ],
"Next": "Clean Up" }
]
},

“P": {
"Type": "Parallel",
"Branches": [
{
"StartAt": “P1",
"States": {
“P1": {
"Type": "Task",
"End": true } }
},
{
"StartAt": “P2",
"States": {
“P2": {
"Type": "Task",
"End": true } }
}
],
"Next": "NextState"
}
Parallel states

Hands-on: Part two (step 1D–3)
What part two will look like when finished:
What you’ll do:
• Add ResultPath
• Add choice state
• Add custom error handling
• Add parallel state
• Test using the console

Part two review

Part three
Other architecture patterns

Use activity task for manual approval
Step Functions
Poll:
taskToken
Amazon
SES
Poller
Approver
Send success:
taskToken
API
Gateway

Use Wait + Choice to wait for batch job/async APIs
while:
sleep (30)
if status=done:
break

Integrate with other AWS services
• Create state machines and activities with AWS CloudFormation
• Call AWS Step Functions with Amazon API Gateway
• Start state machines in response to events, or on a schedule, with Amazon CloudWatch
Events
• Monitor state machine executions with Amazon CloudWatch
• Log API calls with AWS CloudTrail
• Start state machine using AWS Lambda

Hands-on: Part three (step 3–end)
What you’ll do:
• Add a final task state to persist the metadata
• Configure the trigger for the state machine
• Deploy a test web application and test
Optional bonus tasks: (if you finish early and you want to, keep going)
Add in additional error handling/retry logic
Add a manual approval task if inappropriate content is found
A state machine to clean up when images are deleted

Q&A

What are you building? #stepfunctions
Serverless image recognition and processing backendEBS snapshot management
https://console.aws.amazon.com/states/

Please complete the session survey in the
summit mobile app.

Submit Session Feedback
1. Tap the Schedule icon. 2. Select the session
you attended.
3. Tap Session
Evaluation to submit your
feedback.

Thank you!

SRV328 Designing and Implementing a Serverless Media-Processing Workflow

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a SRV328 Designing and Implementing a Serverless Media-Processing Workflow

Semelhante a SRV328 Designing and Implementing a Serverless Media-Processing Workflow (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

SRV328 Designing and Implementing a Serverless Media-Processing Workflow