Building Business Workflows with AWS Step Functions: re:Invent 2018 Recap at the AWS Loft - San Francisco

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Building Business Workflows with AWS Step Functions
Chris Munns
Principal Developer Advocate
AWS Serverless

About me:
Chris Munns - munns@amazon.com, @chrismunns
– Principal Developer Advocate - Serverless
– New Yorker
– Previously:
• AWS Business Development Manager – DevOps, July ’15 - Feb ‘17
• AWS Solutions Architect Nov, 2011- Dec 2014
• Formerly on operations teams @Etsy and @Meetup
• Little time at a hedge fund, Xerox and a few other startups
– Rochester Institute of Technology: Applied Networking and Systems
Administration ’05
– Internet infrastructure geek

https://secure.flickr.com/photos/mgifford/4525333972
Why are we
here today?

DBMSQueue
Modern application

Modern application

Turning functions into apps
“I want to select functions based on data”
“I want to retry functions”
“I want try/catch/finally”
“I have code that runs for hours”
“I want to run functions in parallel”
“I want to sequence functions”

Example Use Case
Suppose I want to
book my vacation to
the Grand Canyon,
…but unwind my reservations if I can’t rent a car.

Coordination by method call

Coordination by function chaining

Coordination by database

Coordination by queues

AWS
Step Functions

Manages the logic of
your application. This
removes extra code that
may be repeated in your
microservices and
functions.
Easy to connect and
coordinate
distributed components
and microservices to
quickly create apps
AWS Step Functions
manages state,
checkpoints and restarts
to make sure your
application executes in
order and as expected.
Build and update
Apps Quickly
Write Less Code Improve
Resiliency
Benefits of AWS Step Functions

New: Richer workflows
Simplify building workloads such as order
processing, report generation, and data analysis
Write and maintain less code; add services in
minutes
More service integrations:
AWS Step
Functions
Amazon
Simple
Notification
Service
Amazon
Simple Queue
Service
Amazon
SageMaker
AWS Glue AWS Batch Amazon Elastic
Container
Service
AWS Fargate
NEW!

Use serverless workflows to build and update apps quickly
Task
Choice
Fail
Parallel
Mountains
People
Snow

Define workflows in JSON
{
"Comment": "Image Processing workflow",
"StartAt": "ExtractImageMetadata",
"States": {
"ExtractImageMetadata": {
"Type": "Task",
"Resource": "arn:aws:lambda:::function:photo-backendExtractImageMetadata-
...",
"InputPath": "$",
"ResultPath": "$.extractedMetadata",
"Next": "ImageTypeCheck",
"Catch": [ {
"ErrorEquals": [ "ImageIdentifyError"],
"Next": "NotSupportedImageType"
} ],
"Retry": [ {
"ErrorEquals": [ "States.ALL"],
"IntervalSeconds": 1,
"MaxAttempts": 2,
"BackoffRate": 1.5 }, ...

Monitor from the console, CLI or API

Run each step in sequence
Vendor A
Vendor B
Vendor C

…and unwind if my plans fail at any point
Vendor A
Vendor B
Vendor C

"HelloWorld": {
"Type": "Task",
…
"Retry": [
{
"ErrorEquals": [”States.TaskFailed"],
"IntervalSeconds": 1,
"MaxAttempts": 2,
"BackoffRate": 2.0
},
… ]
Retry failures

Catch Exceptions
"HelloWorld": {
"Type": "Task",
…
"Catch": [
{
"ErrorEquals": [”AllYourBaseAreBelongToUs"],
"Next": "CustomErrorFallback"
},
{
"ErrorEquals": ["States.TaskFailed"],
"Next": "ReservedTypeFallback"
… ]
{

Coordination must-haves
• Scales out
• Doesn’t lose state
• Deals with errors/timeouts
• Easy to build & operate
• Auditable

Seven state types
Task A single unit of work
Choice Adds branching logic
Parallel Fork and join the data across tasks
Wait Delay for a specified time
Fail Stops an execution and marks it as a failure
Succeed Stops an execution successfully
Pass Passes its input to its output

Task states for any compute
Long poll
Traditional server
Request
Worker requests
tasks from Step
Functions
Step Functions
invokes the Task

Customer use-cases
• Build modern applications
• Data processing
• Automate tasks
• Break up monoliths
• Automate Machine Learning (ML) Jobs

“I want to sequence functions”
AWS Step Functions, we can easily change and iterate on
the application workflow of our food delivery service in
order to optimize operations and continually improve
delivery times. AWS Step Functions lets us dynamically
scale the steps in our food delivery algorithm so we can
manage spikes in customer orders and meet demand.
Mathias Nitzsche, CTO, foodpanda
“

Serverless human approval tasks

• Challenge
• Transcode ~350 news video clips per day into 14
formats each– as quickly as possible
•
Results
• Able to process video segments in parallel
• Reduced processing time from ~20 min to ~2 minutes
• The bigger the source video, the more segments, the
bigger the savings
Thomson Reuters
Serverless split video transcoding Industry: Media & Entertainment
Solution: Big Data / Machine Learning
Architecture: Serverless

How they did it
• Derive keyframe locations within the source
• Split the source at the keyframes
• Process segments (typically 0.5 sec per) in
parallel
• Concatenate segments
• Elapsed time: ~20 min down to ~2 minutes
Workflow

• Challenge
• Ensure that their customers won’t run out of disk space
• Avoid having to over-provision storage allocated to RDS
•
Results
• Reduced storage costs up to 60% per database
• Better customer experience
Outsystems
Saved 60% by creating a database resizing microservice Industry: Internet
Solution: IT Automation
Architecture: Microservices

How they did it
• Developed an automatic resizing process which they
exposed as a microservice behind Amazon API Gateway
• Takes advantage of the Native Backup and Restore
to/from S3 feature available on RDS SQLServer service
• Use Step Functions as the orchestration engine with all
tasks being Lambda functions
• Reduce default database size from 1024 GB to 400GB –
save 60% right away
• Adjust databases on demand as customer needs grow
Workflow

• Challenge
• Update nutrition information on over 100 brands to
comply with updated FDA regulations
•
Results
• Processing time shortened from 36 hours to 10 seconds
• Better productivity: reducing the distance between the
business and development teams identified new
enhancements and improvements that could be quickly
implemented
Coca-Cola
Optimizing nutrition syndication processes
Case Study:
https://aws.amazon.com/solutions/case-studies/bigfinite/
Industry: Retail
Solution: Marketing Automation
Architecture: Serverless

Solution architecture

Workflow

Pricing
• Free Tier
• 4,000 state transitions
• per month
• State Transition, Per 1,000
• $0.025 in most regions
• $0.0271 in Asia Pacific (Seoul)
• $0.0279 in US West (N. California)
• $0.0285 in Asia Pacific (Mumbai)
• $0.0300 in AWS GovCloud (US)
Step Functions counts a state transition each time a step of your workflow is executed.
You are charged for the total number of state transitions across all your state machines,
including retries.

DEMO!

aws.amazon.com/serverless

Chris Munns
munns@amazon.com
@chrismunnshttps://www.flickr.com/photos/theredproject/3302110152/

?
https://secure.flickr.com/photos/dullhunk/202872717/

Building Business Workflows with AWS Step Functions: re:Invent 2018 Recap at the AWS Loft - San Francisco

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Building Business Workflows with AWS Step Functions: re:Invent 2018 Recap at the AWS Loft - San Francisco

Semelhante a Building Business Workflows with AWS Step Functions: re:Invent 2018 Recap at the AWS Loft - San Francisco (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Building Business Workflows with AWS Step Functions: re:Invent 2018 Recap at the AWS Loft - San Francisco