Biml for Beginners: Script and Automate SSIS development (Hybrid VC)

1. Biml for Beginners: Script and Automate SSIS Development Cathrine Wilhelmsen PASS Hybrid VC · July 30th 2018

2. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Session Description Are you tired of creating and updating the same SSIS packages again and again? Is your wrist hurting from all that clicking, dragging, dropping, connecting and aligning? Do you want to take the next step and really speed up your SSIS development? Say goodbye to repetitive work and hello to Biml, the markup language for Business Intelligence projects. In this session we will look at the basics of Biml. First learn how to use Biml to generate SSIS packages from database metadata. Then see how you can reuse code to implement changes in multiple SSIS packages and projects with just a few clicks. Finally, we will create an example project that you can download and start with to speed up your SSIS development from day one. Stop wasting your valuable time on doing the same things over and over and over again, and see how you can complete in a day what once took more than a week!

3. Cathrine Wilhelmsen @cathrinew cathrinew.net Senior BI Consultant, Inmeta

16. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What is Biml? Business Intelligence Markup Language Easy to read and write XML language Describes business intelligence objects: • Databases, Schemas, Tables, Views, Columns • SSIS Packages • SSAS and Azure Data Factory *

18. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) SSIS: Plumbing Spend time on dragging, dropping, connecting, aligning Create the same package over and over again with minor changes Standards, patterns and templates must be defined up-front Changes must be done in every single package Higher risk of manual errors More packages, more time

19. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml: Business Logic Spend time on what is unique in a package Create a pattern once and reuse for all similar packages Handle scope and requirement changes quickly and easily Changes can be applied to all packages at once Lower risk of manual errors Longer time to start, but then automate and reuse

20. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Will Biml solve all your challenges? Biml is a tool for generating SSIS packages Biml is not a pre-defined Data Warehouse framework Biml is not a tool for automating deployment

21. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How can Biml help you? Timesaving: Many SSIS packages from one Biml file Reusable: Write once and run on any platform Flexible: Start simple, expand as you learn

22. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What if you don't have 100+ packages? Source Control: Keep track of actual logic changes Simplify: Untangle complex packages for quick edits SQL Scripts: Use Biml instead of dynamic T-SQL

26. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>

27. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax: Root Element <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>

28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax: Collections of Elements <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>

29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax: Elements <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>

30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax: Attributes <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>

31. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax: Full vs. Shorthand Syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>

38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What is BimlScript? Extend Biml with C# or VB code blocks Import database structure and metadata Loop over tables and columns Expressions replace static values Generate, control and manipulate Biml code

42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) BimlScript Syntax <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>

43. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) BimlScript Syntax: Import metadata <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>

44. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) BimlScript Syntax: Loop over tables <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>

45. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) BimlScript Syntax: Replace static values <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>

46. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Yes, but how does it actually work? <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>" /> <# } #> <Package Name="Load_Customer" /> <Package Name="Load_Product" /> <Package Name="Load_Sales" />

55. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files Split Biml code in multiple files to: • Solve logical dependencies • Build solutions in multiple steps Specify the tier per file by using template directive: <#@ template tier="2" #>

61. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package

64. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package

65. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package

66. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master

67. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How do you use Tiered Biml Files? 1. Create Biml files with specified tiers 2. Select all the tiered Biml files 3. Right-click and click Generate SSIS Packages 1 2 3

69. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Annotations Store custom metadata by attaching tags to Biml objects: <Annotation Tag="TagName">Custom Value</Annotation> Higher tier files can get tags from lower tier files

70. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Annotations Create annotations: <OleDbConnection Name="Dest"> <Annotations> <Annotation Tag="Schema">stg</Annotation> </Annotations> </OleDbConnection> Use annotations: RootNode.OleDbConnections["Dest"].GetTag("Schema");

Biml for Beginners: Script and Automate SSIS development (Hybrid VC)

Esté a ler uma amostra.

Confira estes a seguir

Biml for Beginners: Script and Automate SSIS development (Hybrid VC)

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a Biml for Beginners: Script and Automate SSIS development (Hybrid VC) (20)

Mais de Cathrine Wilhelmsen (17)

Mais recentes (20)