This document discusses applying agile concepts to data warehousing projects to deliver results faster. It begins with the presenter's background and an overview of Snowflake Computing. It then covers the 12 principles of agile, explaining how each could be applied in a data warehousing context, such as having daily standups with business users, using prioritized backlogs and user stories, delivering working code/reports frequently in short iterations focused on a subject area, and using data vault modeling techniques. Retrospectives are emphasized as important for continuous improvement.

1. KENT GRAZIANO
@KentGraziano | kentgraziano.com
AGILE METHODS AND DATA
WAREHOUSING:
HOW TO DELIVER FASTER

2. Agenda
l My Bio
l Why Agile & DW
l 12 Agile Principles
l Aplying Agile Concepts to DW
l Two week iterations?
l Real World metrics
l Becoming Agile
l Conclusion
© Data Warrior LLC

3. My Bio
› Senior Technical Evangelist, Snowflake Computing
› Blogger: The Data Warrior
› Certified Data Vault Master and DV 2.0 Practitioner
› Oracle ACE Director (BI/DW)
› Data Modeling, Data Architecture and Data Warehouse
Specialist
› 30+ years in IT
› 25+ years of Oracle-­related work
› 20+ years of data warehousing experience
› Former-­Member: Boulder BI Brain Trust
(http://www.boulderbibraintrust.org/)
› Author & Co-­Author of a bunch of books
› Past-­President of Oracle Development Tools User
Group and Rocky Mountain Oracle User Group

4. About Snowflake
Experienced,
accomplished
leadership team
2012
Founded by
industry veterans
with over 120
database patents
Vision:
A world with
no limits on data
First data
warehouse
built for the
cloud
Over 500
customers
since GA

5. © 2016 Snowflake Computing Inc. All Rights Reserved.
Ranked #1 Cloud Data Warehouse!
“Snowflake Hits All the Marks” – Gigaom
“You can tell the data warehouse
pedigree from the development…
With superior performance and
the most hands-off model of
ownership, Snowflake is the
epitome of data warehouse as a
service. The model, cost, features
and scalability have already
caused some to postpone Hadoop
adoption.”
Gigaom Analyst Report: Sector Roadmap: Cloud
Analytic Databases 2017
William McKnight
Read the full report on snowflake.net
Gigaom
Disruption
Vectors

6. Why Agile & DW?
l Perceptions
● DW too slow to produce results
● DW projects “fail”
● DW projects fail to adapt to business changes
l Goal
● Change perceptions
● Deliver value
● Be more adaptable and flexible
© Data Warrior LLC

7. Objectives
l Understand what is meant by “agile”
l Try to apply some agile ideas to data warehouse and business
intelligence efforts
l Answer the question – can we deliver results faster?
© Data Warrior LLC

8. Manifesto for Agile Software Development
http://agilemanifesto.org
We are uncoveringbetter ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and
tools
Working software over comprehensive
documentation
Customer collaborationover contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more
Kent Beck
Mike Beedle
Arie van Bennekum
Alistair Cockburn
Ward Cunningham
Martin Fowler
James Grenning
Jim Highsmith
Andrew Hunt
Ron Jeffries
Jon Kern
Brian Marick
Robert C. Martin
Steve Mellor
Ken Schwaber
Jeff Sutherland
Dave Thomas
© 2001, the above authors
this declaration may be freely copied in any form,
but only in its entirety through this notice.

9. THE PRINCIPLES OF AGILE

10. Principle #1
l Our highest priority is to satisfy the
customer through early and continuous
delivery of valuable software.
● Who is the customer?
● What is “valuable software” in data
warehousing?
● BI reports
● Dashboard interface
● Working ETL code?
● In the context of the customer!
● Once we start – keep going!
© Data Warrior LLC

11. Principle #2
l Welcome changing requirements, even
late in development. Agile processes
harness change for the customer's
competitive advantage.
● User Stories with Prioritized Backlog
● Must be flexible and adaptable in thinking and
design
● Use code generators
● Agile data engineering techniques
● Flexible and performant data platform
© Data Warrior LLC

12. Principle #3
l Deliver working software frequently, from
a couple of weeks to a couple of months,
with a preference to the shorter
timescale.
● Need good scope control!
● Prioritized Backlog
● One subject area at a time
● What is a subject area?
● Agile Data Engineering – Data Vault
© Data Warrior LLC

13. Data Vault Components
Copyright 2011 Dan Linstedt, used by permission

14. Data Vault Extensibility
Adding new components to
the EDW has NEAR ZERO
impact to:
• Existing Loading
Processes
• Existing Data Model
• Existing Reporting & BI
Functions
• Existing Source Systems
• Existing Star Schemas
and Data Marts
(C) LearnDataVault.com
Sprint 1
Sprint 2
Sprint 3

15. Principle #4
l Business people and developers
must work together daily throughout
the project.
● DW MUST have the business involved
● One of the Top 10 reasons for failure
● This applies for BI reports
● Daily interaction would be great!
● But – politics and priorities may interfere!
● At HP GBI/EDW – we used “war” room
● At MSH – daily standups with Biz
© Data Warrior LLC

16. Principle #5
l Build projects around motivated
individuals. Give them the environment
and support they need, and trust them to
get the job done.
● Need people who WANT to be on the project
● Lose the dead weight
● Get training if needed
● Keep units of work small to create an
atmosphere of success
● Don’t try a Big Bang EDW
© Data Warrior LLC

17. Principle #6
l The most efficient and effective method of
conveying information to and within a
development team is face-­to-­face
conversation.
● Daily team huddles
● Co-­located work space
● While face-­to-­face is efficient, still need some
documentation (or meta-­data) for later
● Use a tool like JIRA, VersionOne, Rally, LeanKit
© Data Warrior LLC

18. Principle #7
l Working software is the primary measure
of progress.
●Applied to DW:
● What is “working software?”
● BI reports?
● Tables definitions and working ETL
code?
●Think more broadly – it is not just a data
entry screen
© Data Warrior LLC

19. Principle #8
l Agile processes promote sustainable development. The
sponsors, developers, and users should be able to
maintain a constant pace indefinitely.
● DW Programs last a long time – don’t burn the team
out with unreasonable deadlines
● See P#5 – Motivated individuals
● Good planning and scope control
● No all nighters!
● Smallest valuable unit of work possible (MVP)
● Keep it moving like a production line
● Pick (or develop) a standard, repeatable
methodology
● Study the Agile methods and adopt what works for your team
● Data Vault Modeling Methodology
© Data Warrior LLC

20. Principle #9
l Continuous attention to technical
excellence and good design enhances
agility.
● Bad design + bad architecture = trouble
● Symptom: can’t build a requested data mart
● Set standards and use them!
● Frequent design reviews a must
● Improves team skills – provides cross training
● Over time – better designs, shorter review cycles
● Faster delivery
© Data Warrior LLC

21. Principle #10
l Simplicity-­-­the art of maximizing the
amount of work not done-­-­is essential.
● KISS – Keep it Simple Stupid
● Write less code by hand
● Use code generators! (No syntax errors – ever)
● Oracle SDDM, ERWin, Vertabelo
● Oracle Data Integrator, SSIS
● AnalytxDS, WhereScape RED
● Modifications are easier – just regenerate the code
● Virtualize initial reporting layers/data marts
● Eliminate infrastructure admin & DB tuning
● Agile DevOps!
© Data Warrior LLC

22. Principle #11
l The best architectures, requirements, and
designs emerge from self-­organizing teams.
● Team of smart, motivated people = success
● We succeed (or fail) as a TEAM
● Don’t micro manage or pigeon-­hole staff
● Encourage team work and team thinking
● Staff will gravitate to roles based on skills, interest,
and personality
● Then they have more buy-­in to the process
● Eliminates delays and bottlenecks by having
shared responsibilities (no single point of failure)
© Data Warrior LLC

23. Principle #12
l At regular intervals, the team reflects on how
to become more effective, then tunes and
adjusts its behavior accordingly.
● Retrospectives are a MUST!
● Ask 3 questions:
● What went right?
● What went wrong?
● What will we do different next sprint?
● Related to self-­organizing teams
● Make finding the solution to a problem the team’s
problem
● More buy-­in to the solution
© Data Warrior LLC

24. USING AGILE CONCEPTS

25. Agile Concepts for DW
l Team Huddles (Morning Scrum)
l Extreme Programming
© Data Warrior LLC

26. Team Huddles
l Daily Standup Meeting
● AKA Scrum
● Morning Roll Call (FDD)
l Short meeting (< 15 minutes)
● Every morning, mandatory attendance
● Review assignments, accomplishments, backlogs,
blockers
● 3 Questions
l Immediate feedback and assistance
● Keeps team motivated and on track (P #5)
● Identifies constraints and bottlenecks early in the process
● Eliminates backlogs more quickly via re-­assignments
l Improves team work
l Supports self-­organizing teams (P #11)
© Data Warrior LLC

27. Extreme Programming (XP)
l Programmer works directly with the end user
● At MSH used conference room or developer’s cube
● At HP used Virtual Classroom or NetMeeting
● At DFA used WebEx
l In DW:
● Best with developing BI reports,dashboards,
visualizations
● DW or data mart must already be populated
l Works for ETL as well!
● Used war room with business to get near instant
validation of ETL changes
© Data Warrior LLC

28. Two week iterations?
l Goal is really a few weeks to a few months
(see P#3)
l What is the deliverable?
● A fact table for a star schema
● A dimension table
● A complete star (fact and all dimensions)
● One piece of ETL code that populates a fact
table
● A function needed by the ETL code
● A new report or query
l Who is the customer?
● BI programmer?
● Knowledge worker?
● ETL programmer?© Data Warrior LLC

29. REAL WORLD METRICS

30. HP EDW Examples
l Business found missing report elements
l Solution: modify 3 tables to add 5 new columns in
reporting model (star schema)
l Tasks:
● Document requirements and ETL specs
● Modify Logical & Physical model (w/peer review)
● Rebuild tables in development
● Develop and test ETL
● MTI (Move To Integration) tables and code
● Execute and test ETL
● Modify report in UAT environment & test
l Result: Revised report ready in 18 hours, 44 minutes
● Less than 1 business day
l 2nd case: 6 tables, 16 new columns
● Ready for UAT in 72 hours
l How? War room with Biz & IT Team
© Data Warrior LLC

31. BECOMING AGILE
IN THE REAL WORLD

32. Getting to Agile
l “better than average expertise”
● Expert consulting and mentoring
● Do the work (OTJ)
l At Denver Public Schools
● Took two years before we could try being
more “agile”
● Needed experience in Data Warehousing,
Oracle Designer, OWB, and the “process” of
building, deploying, and maintaining an
Oracle DW
© Data Warrior LLC

33. Getting to Agile
l At HP GBI/EDW it took about a year
● Needed the right team and the right
management support
● Also the right project with willing business
users
l At McKesson over a year setting
standards and training staff
● Built templates in JIRA for repeatable tasks
● Then 3 week sprints
l At DFA – Day 1!
● Experienced team with agile BI/DW coach
● Used modified KanBan
© Data Warrior LLC

34. Getting to Agile
l Cannot instantly become “agile”
l Must develop an agile mindset & culture
l Implement agile tools, processes, and methods
l Get an Agile Coach if necessary
● Specfically an agile BI/DW or Data Coach
© Data Warrior LLC

35. SHAMELESS PLUG: My Intro Book
Available on
Amazon.com
http://www.amazon.com/Better-­
Data-­Modeling-­Introduction-­
Engineering-­
ebook/dp/B018BREV1C/

36. Other References
l Agile Management for Software Engineering:
Applying the Theory of Constraints for Business
Results by David J. Anderson
l CASE Method Fast-­track: A RAD Approach by
Richard Barker & Dai Clegg
l The Goal by Eliyahu M. Goldratt
l The Business of Data Vault Modeling by Dan
Linstedt, Kent Graziano, & Hans Hultgren
l Super Charge Your Data Warehouse by Dan
Linstedt
l Test Automation by Lynn Winterboer
http://www.slideshare.net/AgileDenver/lynn-­
winterboer-­test-­automation
© Data Warrior LLC

37. Conclusion
l Agile concepts can be applied to data
warehouse and BI projects
● Not a purist definition!
● Try to apply the principles – be creative
l Suggested approaches
● Data Vault 2.0!
● Use Snowflake Elastic DW!
● Use team huddles
● Use pair programming to increase quality and cross
training
● Use code generators
● Read about Agile Methods
● Read Oracle CASE Method Fast-­Track
l Be flexible and give it a try
© Data Warrior LLC

38. CONTACT
INFORMATION
KENT GRAZIANO
Snowflake Computing
Snowflake.net
kent.graziano@snowflake.net
@KentGraziano
http://kentgraziano.com