Mais conteúdo relacionado Semelhante a Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland) (20) Mais de Cathrine Wilhelmsen (20) Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)2. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Session Description
Are you tired of creating and updating the same SSIS packages again and again? Is your wrist hurting
from all that clicking, dragging, dropping, connecting and aligning? Do you want to take the next step and
really speed up your SSIS development?
Say goodbye to repetitive work and hello to Biml, the markup language for Business Intelligence projects.
In this session we will look at the basics of Biml. First learn how to use Biml to generate SSIS packages
from database metadata. Then see how you can reuse code to implement changes in multiple SSIS
packages and projects with just a few clicks. Finally, we will create an example project that you can
download and start with to speed up your SSIS development from day one.
Stop wasting your valuable time on doing the same things over and over and over again, and see how
you can complete in a day what once took more than a week!
3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Basics Tools & Code
SQL and SSIS from Metadata
…the next 60 minutes…
5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
You…
…?
SSIS developer
Easily bored
Tired of repetitive work
6. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Work…
…?
Long development time
Many SSIS packages
Frequent requirement changes
7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
job done!
new standards
…yay
Ever experienced this?
8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
It's time for a change!
10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What is Biml?
Business Intelligence Markup Language
Easy to read and write XML language
Describes business intelligence objects:
• Databases, Schemas, Tables, Views, Columns
• SSIS Packages
• SSAS and Azure Data Factory *
11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Why use Biml?
SSIS: Plumbing Biml: Business Logic
12. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
SSIS: Plumbing
Spend time on dragging, dropping, connecting, aligning
Create the same package over and over again with minor changes
Standards, patterns and templates must be defined up-front
Changes must be done in every single package
Higher risk of manual errors
More packages, more time
13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml: Business Logic
Spend time on what is unique in a package
Create a pattern once and reuse for all similar packages
Handle scope and requirement changes quickly and easily
Changes can be applied to all packages at once
Lower risk of manual errors
Longer time to start, but then automate and reuse
14. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Will Biml solve all your challenges?
Biml is a tool for generating SSIS packages
Biml is not a pre-defined Data Warehouse framework
Biml is not a tool for automating deployment
15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How can Biml help you?
Timesaving: Many SSIS packages from one Biml file
Reusable: Write once and run on any platform
Flexible: Start simple, expand as you learn
16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What if you don't have 100+ packages?
Source Control: Keep track of actual logic changes
Simplify: Untangle complex packages for quick edits
SQL Scripts: Use Biml instead of dynamic T-SQL
17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How does SSIS development work?
18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How does Biml development work?
19. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What do you need?
20. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlExpress
Free add-in for Visual Studio
Code editor with syntax highlighting and Biml Intellisense
More frequent updates than BIDS Helper
varigence.com/bimlexpress
21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlStudio
Licensed full-featured development environment for Biml
Visual designer and metadata modeling
Full-stack automation and transformers
varigence.com/bimlstudio
22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax: Root Element
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax: Collections of Elements
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax: Elements
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax: Attributes
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax: Full vs. Shorthand Syntax
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Let's generate
some packages!
29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Add New Biml File from BimlExpress menu…
30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…or right-click on SSIS project to Add New Biml File
31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml files are placed under Miscellaneous
32. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Check Biml For Errors from BimlExpress menu…
33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…or right-click on file to Check Biml For Errors
34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Generate SSIS Packages from BimlExpress menu…
35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…or right-click on file to Generate SSIS Packages
36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
From Biml to SSIS: Control Flow
<Package Name="TruncateLoad" ConstraintMode="Linear">
<Tasks>
<ExecuteSQL Name="Truncate Table" ConnectionName="Staging">
<DirectInput>TRUNCATE TABLE DestinationTable</DirectInput>
</ExecuteSQL>
<Dataflow Name="Load Table">
<Transformations>...</Transformations>
</Dataflow>
</Tasks>
</Package>
37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
From Biml to SSIS: Data Flow
<Transformations>
<OleDbSource Name="Source" ConnectionName="Source">
<ExternalTableInput Table="SourceTable" />
</OleDbSource>
<DerivedColumns Name="Add LoadDate">
<Columns>
<Column Name="LoadDate" DataType="DateTime">
@[System::StartTime]
</Column>
</Columns>
</DerivedColumns>
<OleDbDestination Name="Destination" ConnectionName="Staging">
<ExternalTableOutput Table="DestinationTable" />
</OleDbDestination>
</Transformations>
38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Ok, so we can go from Biml to SSIS…
39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…can we go from SSIS to Biml?
41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Let's convert some
SSIS to Biml!
42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Convert SSIS to Biml in BimlExpress 2018
43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Select Package or project…
44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…import, filter and add to project
45. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
The magic is in the…
46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What is BimlScript?
Extend Biml with C# or VB code blocks
Import database structure and metadata
Loop over tables and columns
Expressions replace static values
Generate, control and manipulate Biml code
47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript Code Blocks
<# … #> Control Block (Variables and logic)
<#= … #> Text Block (Returns string)
<#@ … #> Directive (Compiler instructions)
<#+ … #> Class Block (C# classes and methods)
<#* … *#> Comment Block (Disable BimlScript)
48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript Syntax
<# var con = SchemaManager.CreateConnectionNode(...); #>
<# var metadata = con.GetDatabaseSchema(); #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table in metadata.TableNodes) { #>
<Package Name="Load_<#=table.Name#>"></Package>
<# } #>
</Packages>
</Biml>
49. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript Syntax: Import metadata
<# var con = SchemaManager.CreateConnectionNode(...); #>
<# var metadata = con.GetDatabaseSchema(); #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table in metadata.TableNodes) { #>
<Package Name="Load_<#=table.Name#>"></Package>
<# } #>
</Packages>
</Biml>
50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript Syntax: Loop over tables
<# var con = SchemaManager.CreateConnectionNode(...); #>
<# var metadata = con.GetDatabaseSchema(); #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table in metadata.TableNodes) { #>
<Package Name="Load_<#=table.Name#>"></Package>
<# } #>
</Packages>
</Biml>
51. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript Syntax: Replace static values
<# var con = SchemaManager.CreateConnectionNode(...); #>
<# var metadata = con.GetDatabaseSchema(); #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table in metadata.TableNodes) { #>
<Package Name="Load_<#=table.Name#>"></Package>
<# } #>
</Packages>
</Biml>
52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml vs. BimlScript
Generate, control and
manipulate Biml with C#
XML Language
"Just plain text"
53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How does it work?
54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Yes, but how does it work?
55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Yes, but how does it actually work?
<# foreach (var table in RootNode.Tables) { #>
<Package Name="Load_<#=table.Name#>" />
<# } #>
<Package Name="Load_Customer" />
<Package Name="Load_Product" />
<Package Name="Load_Sales" />
56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
PreviewPane
The power is in the…
57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlExpress Preview Pane
58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Let's generate
many packages!
59. Of course I can create
200 SSIS Packages!
…what do you need
me to do after lunch?
60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Don't Repeat Yourself
Centralize and reuse code
Update once in one file
1. Tiered Biml Files
2. Include Files
3. CallBimlScript
61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files
Multiple Biml Files Working Together
62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What are Tiered Biml Files?
Think of tiers as stacked layers or sequential steps
Tier (Layer) 1
Tier 0
Tier (Layer) 2
Tier (Step) 1 Tier (Step) 2
63. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files
Split Biml code in multiple files to:
• Solve logical dependencies
• Build solutions in multiple steps behind the scenes
Specify the tier per file by using the template directive:
<#@ template tier="2" #>
64. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files
Biml is compiled step-by-step from lowest to highest tier
• Biml files are implicitly tier 0
• BimlScript files are implicitly tier 1
In each tier, objects are added in-memory to RootNode
Higher tiers can call RootNode to use objects from lower tiers
65. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Load Packages
Master Package
Connections
Databases
Schemas
Tables
Packages
66. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Databases
Schemas
Tables
Packages
Connections
Load Packages
Master Package
67. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Connections
Load Packages
Master Package
68. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Connections
Load Packages
Master Package
69. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Connections
Load Packages
Master Package
70. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Connections
Load Packages
Master Package
71. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Connections
Load Packages
Master Package
72. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Connections
Load Packages
Master Package
73. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Master Connections
Load Packages
Master Package
74. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Master Connections
Load Packages
Master Package
75. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Master
76. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How do you use Tiered Biml Files?
1. Create Biml files with specified tiers
2. Select all the tiered Biml files
3. Right-click and click Generate SSIS Packages
1
2
3
77. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Annotations
Store and Pass Metadata Between Biml Files
78. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Annotations
Store custom metadata by attaching tags to Biml objects:
<Annotation Tag="TagName">Custom Value</Annotation>
Higher tier files can get tags from lower tier files
79. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Annotations
Create annotations:
<OleDbConnection Name="Dest">
<Annotations>
<Annotation Tag="Schema">stg</Annotation>
</Annotations>
</OleDbConnection>
Use annotations:
RootNode.OleDbConnections["Dest"].GetTag("Schema");
80. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Include Files
Automated Copy & Paste
81. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Include Files
Include common code in multiple files and projects
Can include many file types: .biml .txt .sql .cs
Use the include directive
<#@ include file="CommonCode.biml" #>
The directive will be replaced by the included file
Works like an automated Copy & Paste
82. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Include Files
83. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Include Files
84. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Include Files
85. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
CallBimlScript
Parameterized Control over Returned Code
86. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
CallBimlScript
Control and limit the code returned
Works like a parameterized include (or stored procedure)
CallBimlScript file specifies accepted parameters:
<#@ property name="Parameter" type="String" #>
Main file calls and passes parameters:
<#=CallBimlScript("CommonCode.biml", Parameter)#>
87. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
CallBimlScript
88. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
CallBimlScript
89. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
CallBimlScript
90. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
CallBimlScript
91. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
CallBimlScript
92. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Wait! When do you use what?
In larger projects, you often see a combination of
Tiered Biml Files, Include Files, and CallBimlScript
Rule of Thumb:
If you reuse code more than 3 times,
refactor so you Don't Repeat Yourself
93. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Include Files
Automated Copy & Paste
Code Included As-Is
CallBimlScript
Parameterized
Control Returned Code
Tiered Biml Files
Solve Dependencies
Multi-Step Builds
94. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Let's put it
all together!
95. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Basics Tools & Code
SQL and SSIS from Metadata
…the past 60 minutes…
96. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Where can you learn more?
Free online training
bimlscript.com
97. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Where can you learn even more?
cathrinew.net/BimlBook
98. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Get things done
Start small
Start simple
Start with ugly code
Keep going
Expand
Improve
Deliver often
100. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
Biml resources and demo files:
cathrinew.net/biml