1.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

2.
Dynamic Data Pipelines
in Azure Data Factory
Cathrine Wilhelmsen
Data Saturday Holland · Oct 5, 2019

3.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Building Dynamic Data Pipelines in Azure Data Factory

4.
@cathrinew
cathrinew.net

5.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
You are an
Azure Data Factory
developer

6.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

7.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
…this is getting
pretty tedious…

8.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
There has to be
a better way!

9.
Dynamic Data
Pipelines

10.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Why use dynamic solutions?

11.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Why not use dynamic solutions?

12.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How dynamic should the solution be?

13.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How dynamic should the solution be?

14.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you make dynamic solutions?
1.
2.
3.
4.

15.
Parameters

16.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters

17.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters

18.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service

19.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
pipeline

20.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
data flow

21.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
activity

22.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
pipeline
activity
data flow

23.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@dataset().
@linkedService().
ParameterName
ParameterName
ParameterName

24.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@dataset().
@linkedService().
ParameterName
ParameterName
ParameterName

25.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@pipeline().DataFactory
@pipeline().TriggerTime
ParameterName
and System Variables

26.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@pipeline().DataFactory
@pipeline().TriggerTime
ParameterName
and System Variables

27.
Expressions
and Functions

28.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Expressions
@

29.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Functions

30.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Expressions with Functions
@

31.
Combining Strings

32.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Combining Strings
@concat
@{…}

33.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
String Concatenation
@concat(
' TRUNCATE TABLE dbo. ',
pipeline().parameters.TableName
)

34.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
String Interpolation
TRUNCATE TABLE dbo.
@{pipeline().parameters.TableName}

35.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Combining Strings
String Concatenation String Interpolation

36.
DEMO
Let's parameterize
some pipelines!

37.
ForEach Loops

38.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
ForEach Loops

39.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
ForEach Loops
@item()

40.
Array Items

41.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items

42.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items
colors
@item()

43.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items
@item()
parts
@item()
colors
@item()
sets

44.
Object Items

45.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items

46.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
{ "Name": "colors", "Extension": "csv" }
@item()

47.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
{ "Name": "colors", "Extension": "csv" }
@item().Name

48.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
@item().Name
parts
@item().Name
colors
@item().Name
sets

49.
DEMO
Let's create
some loops!

50.
Lookups

51.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup

52.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output

53.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output
{
"firstRow" :
{
"Name" : "colors",
"Extension" : "csv"
}
}
{
"count" : "2",
"value" : [
{
"Name" : "parts",
"Extension" : "csv"
},
{
"Name" : "sets",
"Extension" : "csv"
}
]
}

54.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: First Row Only
{
"firstRow" :
{
"Name" : "colors",
"Extension" : "csv"
}
}
firstRow

55.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: First Row Only
{
"firstRow" :
{
"Name" : "colors",
"Extension" : "csv"
}
}
firstRow
.Name

56.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: All Rows
{
"count" : "2",
"value" : [
{
"Name" : "parts",
"Extension" : "csv"
},
{
"Name" : "sets",
"Extension" : "csv"
}
]
}
value

57.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: All Rows
{
"count" : "2",
"value" : [
{
"Name" : "parts",
"Extension" : "csv"
},
{
"Name" : "sets",
"Extension" : "csv"
}
]
}
value[1]

58.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: All Rows
{
"count" : "2",
"value" : [
{
"Name" : "parts",
"Extension" : "csv"
},
{
"Name" : "sets",
"Extension" : "csv"
}
]
}
value[1]
Name

59.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output
@{activity('Lookup').output.firstRow.ColumnName}
@{activity('Lookup').output.value}

60.
DEMO
Let's add a lookup to
control our loop!

61.
Controls and
Conditionals

62.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Controls and Conditionals

63.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Branching and Chaining
Success Failure Completion Skipped

64.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
If Condition

65.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Variables

66.
DEMO
Let's create some
if statements!

67.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Controls and Conditionals

68.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Until

69.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Get Metadata

70.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Wait

71.
DEMO
Let's check for
file existence!

72.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Good luck!

73.
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
thank you!