Mais conteúdo relacionado
Semelhante a Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo) (20)
Mais de Cathrine Wilhelmsen (20)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)
- 2. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
A big thanks to all of our sponsors!
- 3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Basics Tools & Code
SQL and SSIS from Metadata
…the next 60 minutes…
- 5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Once upon a time…
- 6. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Meet Sandra,
the SSIS developer
- 7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Sandra has to create a
new staging project with
100+ SSIS packages
- 8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Sandra gets quite bored of
creating the same package
over and over and over
- 9. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Yay, Sandra has finally
completed her project!
- 10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
But Sandra's manager
has just a few new
requirements…
- 11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Sandra gets to update
every single SSIS package
one… more… time…
- 13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
It's time for a change!
- 15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What is Biml?
Business Intelligence Markup Language
Easy to read and write XML language
Describes business intelligence objects:
• Databases, Schemas, Tables, Views, Columns
• SSIS Packages
• SSAS and Azure Data Factory *
- 16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Why use Biml?
SSIS: Plumbing Biml: Business Logic
- 17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Will Biml solve all your challenges?
Biml is a tool for generating SSIS packages
Biml is not a pre-defined Data Warehouse framework
Biml is not a tool for automating deployment
- 18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How can Biml help you?
Timesaving: Many SSIS packages from one Biml file
Reusable: Write once and run on any platform
Flexible: Start simple, expand as you learn
- 19. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What if you don't have 100+ packages?
Source Control: Keep track of actual logic changes
Simplify: Untangle complex packages for quick edits
SQL Scripts: Use Biml instead of dynamic T-SQL
- 20. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How does SSIS development work?
- 21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How does Biml development work?
- 22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What do you need?
- 23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
- 24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax: Root Element
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
- 25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax: Collections of Elements
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
- 26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax: Elements
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
- 27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax: Attributes
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
- 28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml syntax: Full vs. Shorthand Syntax
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="EmptyPackage1"></Package>
<Package Name="EmptyPackage2"/>
</Packages>
</Biml>
- 29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Let's generate
some packages!
- 30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Ok, so we can go from Biml to SSIS…
- 31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…can we go from SSIS to Biml?
- 33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Let's convert some
SSIS to Biml!
- 34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
The magic is in the…
- 35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What is BimlScript?
Extend Biml with C# or VB code blocks
Import database structure and metadata
Loop over tables and columns
Expressions replace static values
Generate, control and manipulate Biml code
- 36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How does it work?
- 37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Yes, but how does it work?
- 38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript Code Blocks
<# … #> Control Block (Variables and logic)
<#= … #> Text Block (Returns string)
<#@ … #> Directive (Compiler instructions)
- 39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript Syntax
<# var con = SchemaManager.CreateConnectionNode(...); #>
<# var metadata = con.GetDatabaseSchema(); #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table in metadata.TableNodes) { #>
<Package Name="Load_<#=table.Name#>"></Package>
<# } #>
</Packages>
</Biml>
- 40. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript Syntax: Import metadata
<# var con = SchemaManager.CreateConnectionNode(...); #>
<# var metadata = con.GetDatabaseSchema(); #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table in metadata.TableNodes) { #>
<Package Name="Load_<#=table.Name#>"></Package>
<# } #>
</Packages>
</Biml>
- 41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript Syntax: Loop over tables
<# var con = SchemaManager.CreateConnectionNode(...); #>
<# var metadata = con.GetDatabaseSchema(); #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table in metadata.TableNodes) { #>
<Package Name="Load_<#=table.Name#>"></Package>
<# } #>
</Packages>
</Biml>
- 42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript Syntax: Replace static values
<# var con = SchemaManager.CreateConnectionNode(...); #>
<# var metadata = con.GetDatabaseSchema(); #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table in metadata.TableNodes) { #>
<Package Name="Load_<#=table.Name#>"></Package>
<# } #>
</Packages>
</Biml>
- 43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Yes, but how does it actually work?
<# foreach (var table in RootNode.Tables) { #>
<Package Name="Load_<#=table.Name#>" />
<# } #>
<Package Name="Load_Customer" />
<Package Name="Load_Product" />
<Package Name="Load_Sales" />
- 44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
PreviewPane
The power is in the…
- 45. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Let's generate
many packages!
- 46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What about SQL?
- 47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
We can generate SSIS…
- 48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…can we also generate SQL?
- 50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files
Multiple Biml Files Working Together
- 51. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What are Tiered Biml Files?
Think of tiers as stacked layers or sequential steps
Tier (Layer) 1
Tier 0
Tier (Layer) 2
Tier (Step) 1 Tier (Step) 2
- 52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files
Split Biml code in multiple files to:
• Solve logical dependencies
• Build solutions in multiple steps
Specify the tier per file by using template directive:
<#@ template tier="2" #>
- 53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Load Packages
Master Package
Connections
Databases
Schemas
Tables
Packages
- 54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Databases
Schemas
Tables
Packages
Connections
Load Packages
Master Package
- 55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Connections
Load Packages
Master Package
- 56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Connections
Load Packages
Master Package
- 57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Connections
Load Packages
Master Package
- 58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Connections
Load Packages
Master Package
- 59. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Connections
Load Packages
Master Package
- 60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Connections
Load Packages
Master Package
- 61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Master Connections
Load Packages
Master Package
- 62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Master Connections
Load Packages
Master Package
- 63. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Databases
Schemas
Tables
Packages
Load_Customer
Load_Product
Load_Sales
Master
- 64. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Annotations
Store and Pass Metadata Between Biml Files
- 65. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Annotations
Store custom metadata by attaching tags to Biml objects:
<Annotation Tag="TagName">Custom Value</Annotation>
Higher tier files can get tags from lower tier files
- 66. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Annotations
Create annotations:
<OleDbConnection Name="Dest">
<Annotations>
<Annotation Tag="Schema">stg</Annotation>
</Annotations>
</OleDbConnection>
Use annotations:
RootNode.OleDbConnections["Dest"].GetTag("Schema");
- 67. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Let's put it
all together!
- 68. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
New project
- 69. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Of course I can create
200 SSIS Packages!
…what do you need me
to do after lunch?
- 70. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Basics Tools & Code
SQL and SSIS from Metadata
…the past 60 minutes…
- 71. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Where can you learn more?
Free online training
bimlscript.com
- 72. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Where can you learn even more?
cathrinew.net/BimlBook
- 73. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Get things done
Start small
Start simple
Start with ugly code
Keep going
Expand
Improve
Deliver often
- 74. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml on Monday…
…BimlBreak the rest
of the week!
- 75. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
Biml resources and demo files:
cathrinew.net/biml