Microsoft Azure Data Factory ETL Patterns in the Cloud by Mark Kromer

1. ETL Patterns in the Cloud with
Azure Data Factory
Mark Kromer
Senior Program Manager
Microsoft Azure Data Management
@kromerbigdata

2. ETL Patterns in the Cloud
Important factors for success
1. What is ETL?
• More than Extract, Transform, Load
• Scheduling, Monitoring, Maintenance, Source Control, CI/CD, Operationalize
2. Platform as a Service (ADF) vs. Infrastructure as a Service (IaaS/SSIS)
• Self-managed vs. Provider-Managed
3. ELT or ETL?
• Difference is primarily highly-parsed semantics
• However: In the cloud, common pattern == stage data in low-cost, inexpensive storage
4. Not typically performant to process data in-flight
• Particularly crossing boundaries (on-prem, vnets, data centers, regions)
5. Scale is very important in Cloud ETL
• Cloud projects assume elastic scale. ETL is not immune to this expectation.
6. Flexible Schema is very important in Cloud ETL
1. Assume “Big Data tenets” aka “data chaos”: Your data sources will change shape, size and
volume. Often!

3. Azure Data Factory
Workflow Pipelines/Control Flow

5. Built-in source
control support

7. Quickly get started with building data integration solutions. Avoid building same workflows
repeatedly. Simply instantiate a template. Improve developer productivity along with reducing
development time for repeat processes.
Use Templates to quickly get started

8. Secure data platform enabling Analytics and Insights on Microsoft 365.
Data access @ Scale
Dataset based access rather than
real time API based access
Granular Request/Consent
enabling Data Privacy
Row and column level scoping
with advanced filtering capability
Data Governance & Security
Control and visibility over your
data throughout its entire lifecycle
Microsoft Graph Data Connect

9. ADF Integration Runtime
Activity Dispatch/Monitor
Data Movement
SSIS Package Execution

10. Azure Data Factory Service
Cloud Apps, Svcs & DataOn Premises Apps & Data
UX & SDK

11. Azure (US West)
Public Internet Border
HP Inc (Global)
HP Prod Firewall Border
HP Hadoop Cluster
Integration Fabric On-prem
Data Factory
(Orchestration
micro-service)
443
Storage (Azure)
443
SQL Data
Warehouse
UAM Server
443
ADF Foo On-prem
“IR”
Customer 1
Customer 1 firewall border
Azure Data Factory “Integration Runtime” deployed on premises for
transformation and then moved to cloud

12. SSIS in ADF

14. on premises
in Azure
Deployment via SSMS
Once connected, you can deploy
projects/packages to SSIS PaaS from your
local file system/SSIS on premises

15. Execution via SSMS
You can select some packages to execute
on SSIS PaaS

16. Monitoring via SSMS
on premises
in Azure
You can see package execution
error messages

17. Execute SSIS Packaged in ADF Pipeline

18. ADF Mapping Data Flows

19. What is ADF Mapping Data Flow?
Transform Data, At Scale, in the Cloud,
Zero-Code
Cloud-first, scale-out ELT
Code-free dataflow pipelines
Serverless scale-out transformation
execution engine
Maximum Productivity for Data
Engineers
Does NOT require understanding of Spark /
Scala / Python / Java
Resilient Data Transformation Flows
Built for big data scenarios with
unstructured data requirements
Operationalize with Data Factory
scheduling, control flow and monitoring

20. Code-free Data Transformation At Scale
Does not require understanding of Spark, Big Data Execution
Engines, Clusters, Scala, Python …
Focus on building business logic and data transformation
Data cleansing
Aggregation
Data conversions
Data prep
Data exploration

21. ADF Data Flow Workstream
Stage Data in Azure
(ADLS, Blob, SQL
DB/DW)
Transform Data in
Visual Data Flow
Land Data in Azure
Staging Area (ADLS,
Blob, SQL DB/DW)

22. Build your logical data flows adding data
transformations in a guided experience

23. Microsoft Azure Data Factory Continues to Extend Data Flow
Library with a Rich Set of Transformations and Expression
Functions

24. Debug mode provides row-level context
and visible results in inspector pane

25. Interactive Expression Builder – Build data transform
expressions, not Spark code

26. Deep Monitoring Introspection of Data Transformations

27. Debug Data Flows with Data Preview and Data Sampling

28. Cloud ETL Patterns
with ADF

30. MODEL & SERVE
Azure Analysis ServicesAzure SQL Data
Warehouse
Power BI
Modernize your enterprise data warehouse at scale
A Z U R E D A T A F A C T O R Y
On-premises data
Oracle, SQL, Teradata,
fileshares, SAP
Cloud data
Azure, AWS, GCP
SaaS data
Salesforce, Workday,
Dynamics
INGEST STORE PREP & TRAIN
Azure Data Factory Azure Blob Storage
Azure Databricks
Polybase
Microsoft Azure also supports other Big Data services like Azure HDInsight, Azure SQL Database and Azure Data Lake to allow customers to tailor the above architecture to meet their unique needs.
Orchestrate with Azure Data Factory

31. Lift your SQL Server Integration Services (SSIS) packages to Azure
On-Premise data sources
SQL DB Managed Instance
SQL Server
VNET
Azure Data Factory
SSIS Cloud ETL
SSIS Integration Runtime
Cloud data sources
Cloud
On-premises
Microsoft
SQL Server
Integration Services

32. Author, orchestrate and monitor with Azure Data Factory
Hybrid and Multi-Cloud Data Integration
Azure Data Factory
PaaS Data Integration
DATA SCIENCE
AND MACHINE
LEARNING
MODELS
ANALYTICAL
DASHBOARDS
USING POWER BI
DATA DRIVEN
APPLICATIONS
On-Prem SaaS Apps Public Cloud

33. Nightly ETL Data Flows - Codeless

34. Build Resilient Data Flows with Schema Drift
Handling of Flexible Schemas

35. Data Engineer derives columns using template expression
patterns based on name and type matching. No need to define
static field names.

36. Sink all incoming fields along with new
derived field

37. Slowly Changing Dimension Scenario

38. Load Star Schema DW Scenario

39. Data Lake Data Science Scenario

40. Sponsors