Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)

Level Up Your Biml:
Best Practices and Coding Techniques
Cathrine Wilhelmsen, Inmeta
@cathrinew | cathrinew.net

© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Abstract
Is your Biml solution starting to remind you of a bowl of tangled spaghetti code? Good! That means you are
solving real problems while saving a lot of time. The next step is to make sure that your solution does not
grow too complex and confusing - you do not want to waste all that saved time on future maintenance!
Attend this session for an overview of Biml best practices and coding techniques. Learn how to improve and
simplify your solution by using some common and some lesser-known Biml features. If standard Biml is not
enough, you can implement custom logic by creating your own C# classes and methods. Finally, see how to
bring everything together in an example project for creating and loading a data mart with facts and
dimensions.
Start improving your code today and level up your Biml in no time!

@cathrinew
cathrinew.net

Free online webinar
events
Free 1-day local
training events
Local user groups
around the world
Online special interest
user groups
Business analytics
training
Get involved
Explore
everything
PASS has
to offer
Free Online Resources
Newsletters
PASS.org

Download the GuideBook App
and search: PASS Summit 2018
Follow the QR code link displayed on session
signage throughout the conference venue and
in the program guide
Session
evaluations
Your feedback is
important and valuable.
Go to passSummit.com
3 Ways to Access:
Submit by 5pm Friday, November 16th to win prizes.

Biml Tools Staging
Biml for Beginners

Code ModelManage
Level Up Your Biml

Don't Repeat Yourself
Refactor and centralize code
Update once for all projects
1. Include Files
2. CallBimlScript
3. C# Code

Include Files
Automated Copy & Paste

Include Files
Works like an automated Copy & Paste
<#@ include file="CommonCode.biml" #>
Code from included file replaces this include directive

Include Files

CallBimlScript
Parameterized Include with Logic

CallBimlScript
Parameterized include (like a stored procedure)
CallBimlScript file specifies accepted parameters:
<#@ property name="Parameter" type="String" #>
Main file calls and passes parameters:
<#=CallBimlScript("CommonCode.biml", Parameter)#>

CallBimlScript

CallBimlScriptWithOutput
Return Code and Object

CallBimlScript file specifies accepted parameters:
<#@ property name="ProjectName" type="String" #>
CallBimlScriptWithOutput file also specifies object:
<# CustomOutput.Project = metadata.Project; #>

Main file defines, passes, and uses output object
in addition to regular parameters:
<# dynamic project; #>
<#=CallBimlScriptWithOutput("Metadata.biml",
out project, "Staging Project")#>
<#=project.SourceConnection#>

use CustomOutput

C# Code
Custom Classes and Methods

C# Code: Inline Blocks
<#+ ... #>
<#+ #>

C# Code: Included Files
<#@ include file="CodeBlock.biml" #>
<#+ #>

C# Code: Separate Code Files
<#@ code file="Code.cs" #>

Two Approaches to Refactoring
1. Identify Identical Code → Include Files
2. Identify Patterns → CallBimlScript

Log Package Execution (Start)
Log Package Execution (Stop)

Log Package
Execution (Start)
Log Package
Execution (Stop)

Log Package Execution
Log Package Execution

Log Package
Execution
Log Package
Execution

Don't Repeat Yourself
If you reuse code more than 3 times,
refactor :)

+10000 XP
Congratulations! You reached level 60!

Annotations and ObjectTags
Store and Pass Metadata Between Biml Files

Annotations and ObjectTags
Store metadata by attaching tags to Biml objects
Annotations: String values
ObjectTags: Objects
Higher tier files can get tags from lower tier files

Annotations
Create annotations directly in Biml:
<OleDbConnection Name="Dest">
<Annotations>
<Annotation Tag="Schema">stg</Annotation>
</Annotations>
</OleDbConnection>
Use annotations in BimlScript:
RootNode.OleDbConnections["Dest"].GetTag("Schema");

ObjectTags
Create ObjectTags in BimlScript:
RootNode.ObjectTag["Filter"]
= new List<string>{"Product","ProductCategory"};
Use ObjectTags in BimlScript:
var FilteredTables = RootNode.ObjectTag["Filter"];

LINQ
Query and Filter Collections

LINQ
One language to query:
SQL Server Databases
XML Documents
Datasets
Collections
Two ways to write queries:
SQL-ish Syntax
Methods

LINQ Methods
Sort
OrderBy, Reverse
Filter
Where, OfType
Get
First, Last
Check Collections
All, Any, Contains
Set Operations
Except, Intersect
Project Collections
Select, SelectMany

LINQ Methods Example
var firstConnection = RootNode.Connections.First()
foreach (var table in RootNode.Tables.Where(…))
if (RootNode.Packages.Any(…))

LINQ Methods Example
foreach (var table in RootNode.Tables.Where(…))
if (RootNode.Packages.Any(…))
But what do you put in here?

Lambda Expressions
"A lambda expression is an anonymous
function that you can use to create delegates
or expression tree types"

Lambda Expressions
column => column.Name == "ColumnID"

Lambda Expressions
The arrow ("goes to") is the lambda operator

Lambda Expressions
Input parameter is on the left side

Lambda Expressions
Expression is on the right side

LINQ and Lambda
Chain LINQ Methods and use Lambda Expressions for
simple and powerful querying of collections:
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => table.Name)

Column Methods
Generate SQL from Biml

GetColumnList()
GetColumnAssignmentList()
GetColumnComparisonList()

GetColumnList
Get columns for SELECT:
[Column1], [Column2], [Column3]

GetColumnAssignmentList
Get columns for UPDATE … SET:
[left].[Column1] = [right].[Column1]
,[left].[Column2] = [right].[Column2]
,[left].[Column3] = [right].[Column3]

GetColumnComparisonList
Get columns for JOIN … ON:
[left].[Column1] = [right].[Column1]
AND [left].[Column2] = [right].[Column2]
AND [left].[Column3] = [right].[Column3]

Column Methods and Lambda
Use Lambda Expressions to filter columns:
GetColumnList(column => column.IsUsedInPrimaryKey)

C# Extension Methods
Custom LINQ-style!

Extension Methods
"Make it look like the method belongs to
an object instead of a helper class"

From this…
public static string
GetName(AstTableNode table) {...}

…to this
public static string
GetName(this AstTableNode table) {...}

From this…
HelperClass.GetName(table)

…to this
table.GetName()

+100000 XP

Solution Overview

Solution Overview
Dimensions
SCD2
Hybrid
SCD1 SCD2
Facts
Trans-
actional

Facts
Source View
Lookup
Dimension Keys
Insert
New Rows
Load Fact
Get Latest Key

Dimensions
Source View
Lookup
Existing Rows
Insert
New Rows
Insert
Updated Rows
Truncate
Update Table
Load Dimension
Update Rows

Solution Overview
Data Warehouse
Facts Dims Update
Staging
Source
Views

Solution Process
1. Model Facts and Dimensions
2. Generate SQL Scripts
3. Modify Source Views
4. Generate SSIS Packages

Facts and Dimensions
Metadata Model in Excel

Metadata Model
Connections Projects
Tables Columns

EXAMPLE
Metadata
Model in Excel

Metadata Model in Biml

Use Biml features to simplify code:
• ScdType on columns
• StaticSource for static and unknown dimension rows
• CloneTable for update tables

ScdType
Define ScdType per column:
<Column Name="DimSK" DataType="Int32"
IsNullable="false" ScdType="SurrogateKey" />
Use instead of custom annotations :)

ScdType Values
SurrogateKey
Key
Update
Historical
Audit
Other
Often IDENTITY
Source IDs
SCD1 Columns
SCD2 Columns
ETL Columns
Not used in SCD logic

StaticSource
Define rows to be inserted when table is created:
<Table Name="Table">
<Sources>
<StaticSource Name="TableRows">...</StaticSource>
</Sources>
</Table>
Use instead of separate T-SQL scripts :)

StaticSource Values
<StaticSource Name="TableRows">
<Rows>
<Row>
<ColumnValues>
<ColumnValue ColumnName="Col1" Value="-1" />
</ColumnValues>
</Row>
</Rows>
</StaticSource>

CloneTable
Create a copy of a table:
<CloneTable Name="NewTable"
TableName="OriginalSchema.OriginalTable"
SchemaName="Database.NewSchema" />
Use instead of defining tables twice :)

Could not drop object 'Schema.Table'
because it is referenced by a
FOREIGN KEY constraint.

Drop and Create
Create Order: Dim1 Dim2 Fact1 Fact2
Fact2 Fact1 Dim2 Dim1Drop Order:

Topological Sort
"A topological sort of a directed graph is a linear
ordering of its vertices such that for every directed
edge uv from vertex u to vertex v, u comes before
v in the ordering"

Topological Sort
"Sorts objects by dependencies.
For example dimensions before fact tables."

Topological Sort
var dependentTables =
DependencyAnalysis.TopologicalSort<AstTableNode> (
RootNode.Tables,
t => t.Columns
.OfType<AstTableColumnTableReferenceNode>()
.Select(c => c.ForeignTable)
);

Topological Sort: Method
RootNode.Tables,
t => t.Columns
);

Topological Sort: Collection to Sort
RootNode.Tables,
t => t.Columns
);

Topological Sort: How to Sort
RootNode.Tables,
t => t.Columns
);

+1000000 XP

Get things done
Start small
Start simple
Solve a problem
Keep going
Expand
Improve
Deliver often

@cathrinew
cathrinew.net
hi@cathrinew.net
cathrinew.net/biml

Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (9)

Semelhante a Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)

Semelhante a Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) (20)

Mais de Cathrine Wilhelmsen

Mais de Cathrine Wilhelmsen (20)

Último

Último (20)

Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)