AWS Glue makes it easy to incorporate data from a variety of sources into your data lake on Amazon S3. In this chalk talk, we demonstrate building complex workflows using AWS Glue orchestration capabilities. Learn about different types of AWS Glue triggers to create workflows for scheduled processing as well as event-driven processing. We start with a customer scenario and build it step by step using AWS Glue capabilities.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building Advanced Workflows with AWS Glue
Santosh Chandrachood
SDM
AWS Glue
A N T 3 7 2

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Overview
Building Blocks
Building a usecase
Event driven workflows
Workflow considerations
Monitoring and Tuning
Bring your own workflow engine

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Breakout repeats
Monday, Nov 26
ANT 372 – [CT] Building Advanced Workflows with AWS Glue
10:00 – 11:00 | Aria East, Plaza Level, Orovada 3
Tuesday, Nov 27
ANT 333 – [BS] Building Advanced Workflows with AWS Glue
2:30 – 3:30 | Mirage, Grand Ballroom D, Table 4
Wednesday, Nov 28
ANT 381 – [BS] Building Advanced Workflows with AWS Glue
5:30 – 6:30 | Aria West, Level 3, Starvine 10, Table 5

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fully-managed, serverless extract-transform-load (ETL) service
for developers, built by developers
1000s of customers and jobs
AWS Glue

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OrchestrationData Catalog ETL Engine
Automatic crawling
Apache Hive Metastore compatible
Integrated with AWS analytic services
Discover
Flexible scheduling
Monitoring and alerting
External integrations
Deploy
Apache Spark core
Python and Scala
Auto-generates ETL code
Develop
AWS Glue Components

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Apache Spark is a distributed data processing engine for complex analytics.
AWS Glue builds on the Apache Spark to offer ETL specific functionality.
Spark Core: RDDs
Spark DataFrames Glue DynamicFrames
SparkSQL AWS Glue ETL
Review: Apache Spark and AWS Glue ETL

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building Blocks
Crawlers Jobs TriggersEntities
Schedule ExternalEventsDependencies
Conditions TimeoutRetriesControl

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a usecase
raw
optimize optimized
SLA
reporting
New
variable mins variable mins
Goal: compose jobs in DAG through dependencies
In-practice: time-based workflows

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
New features
External
Conditions
Control
Publish crawler and job notifications into CloudWatch events
CloudWatch events to control downstream workflows
‘ANY’ and ‘AND’ operators in Trigger conditions
Additional job states ’failed’, ‘stopped’, or ‘timeout’
Configure job timeout
Job delay notifications
On-demand cancel

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example event-driven workflow
raw
optimize optimized
SLA
reporting
New

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workflow considerations
• Incremental data processing
• Job bookmarks to keep state
• Job parameters to select new datasets
• Job size
• Unique versus One job per logical units of work
• Multiple small jobs or one big job
• Job parameters
• Initial, Global, In-between jobs
• Use Amazon S3 to pass parameters

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workflow considerations
• Data processing unit
• Number of DPUs
• Adjusting DPUs
• SLA
• Job delays notifications
• Timeouts
• Error handling
• Retry logic
• Integration with 3rd party
• Job re-run

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workflow monitoring—Performance
• How is your dataset
partitioned?
• How is your application
divided into jobs and
stages?
• Data is divided into
partitions that are
processed concurrently
Driver
Executors

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workflow monitoring—Metrics
• Job metrics
• CPU
• Memory
• Network
• Executors, stages
• Data movement
• Use data points to
adjust job parameters

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bring your own workflow engine
Triggers
Top-level AWS
Glue job
CloudWatch
Events
Jobs
Lambda
States
SNS
Config
STEP

17. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Santosh Chandrachood
glue-feedback@amazon.com

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.