The document provides an overview of analytics for decision making. It discusses how analytics has become increasingly popular due to its results and reliance on massive datasets. However, analytics is only valuable if it continues to provide the right answers. The document outlines some challenges in analytics including unclear needs, unwillingness to pay, and answers that contradict business goals. It proposes a framework for making analytics work that includes starting with a clear business problem, continuous iteration with stakeholders, and being flexible while maintaining integrity of results. The document concludes with some anecdotes highlighting lessons learned.

1. Analytics for
Decision Making
smart people don’t make blind bets
© 2013 Sandro Catanzaro 1

2. About this deck
This deck was prepared for a Meetup, presented in
Boston in August 2013
Some friends suggested it could be useful if shared
more widely
About me
• I’m a mech eng, from back in the day when you would
draw blueprints by hand
• MIT grad - best place to learn about systems – and
rockets
• Worked as consultant for a while (hence the ‘beautiful’
slides)
• Lead the Analytics and Innovation practices at DataXu
– of which I’m co-founder
Funny picture
about me
© 2013 Sandro Catanzaro 2

3. Founded in 2009; Headquartered in Boston, MA;
300+ employees, 14 offices globally, PetaByte scale dataset
Real-time decision system based on MIT research by
Bill Simmons and Sandro Catanzaro
(first used for NASA Mars Mission project)
Easy-to-use software platform enabling brands to use
Big Marketing Data
5th fastest growth US company
Rated #1 algorithmic optimization
technology by Forrester Research
Powering blue-chip brands:
Ford, Lexus, General Mills,
Wells Fargo, Discover
DataXu is
all about applying
data science to
the art of marketing
© 2011-2013 DataXu, Inc. Privileged & Confidential

4. 4
4© 2011-2013 DataXu, Inc. Privileged & Confidential

5. Agenda
PLEASE MAKE MAGIC HAPPEN
MAKING MAGIC WORK
SOME ANECDOTES
© 2013 Sandro Catanzaro 5

6. PLEASE MAKE MAGIC HAPPEN
© 2013 Sandro Catanzaro 6

7. What about Analytics
• Analytics popularity driven
by its results
• MIT Sloan and IBM survey
shows business believes on
its value*
• Science reliance in massive
datasets
– DNA data mining
Search interest trend for “Analytics”
20132011200920072005
Google Search Trends
0%
50%
100%
2010 2011
MIT Sloan – IBM survey
Decision-makers belief analytics is
a source of competitive advantage
© 2013 Sandro Catanzaro 7
*Source: Create Business Value with Analytics – MIT
Sloan Management Review – Fall 2011”

8. Just a trend?
• Declared the new black
– Symptom of shark
jumping?
• Analytics will stay relevant
as long as it continues
providing the right answer
– Or the right answer more
often than not
© 2013 Sandro Catanzaro 8

9. So, there is value on Analytics
• Yes, but..
• What is it?
• And how to make it happen?
© 2013 Sandro Catanzaro 9

10. Analytics is a bridge
Raw data Decision-making
Cleaning
Aggregates
Selection of
what matters
Error
quantification
Model
limits
Insight
As any other bridge, a feat not a science, but of engineering
• Goes to a specific point to another specific point
• Good enough foundations (data and question clarity)
• Can be used by most people who want to cross it
• Has explicit durability assumptions
Engineering = art of tradeoffs with just enough information
Technology
Action
oriented
question
© 2013 Sandro Catanzaro 10

11. Analytics is a system
Cleaning
Aggregates
Selection of
what matters
Error
quantification
Model
limits Insight
Technology
Presentation
Data
Data
stakeholders
Insight
recipients
QA
P&L owner
Brittle
interface
Easier to manage
interface
Input Process Output
© 2013 Sandro Catanzaro 11

12. Types of Analytics
1. Research
– Unclear need, unclear answer, high risk
2. Development
– Clear answer and method, but some risks on
implementation
3. Operations
– Good experience with execution. Can be automated
– QA for Automation becomes a type 1 problem, again
#1 = most fun / most challenging = focus of this slide-set
© 2013 Sandro Catanzaro 12

13. Analytics Leaders rely on
four core competencies …
• Valuable insights take into
consideration
business, domain and
analytics process…
• …with just-good-enough
technology
• The Analytics Manager main
job is to staff a team that can
master them all
Business &
Communication
Domain
knowledge
Analytics &
reasoning
Technology
13

14. … to address a diversity of issues
Dirty data
Too much data
Far away data
Scaling up the team
New toolset dynamics
Thinking takes time
R&D type of risk
Unclear need
Answer not simple enough
Output contradicts
business
Unwillingness to pay
Politics
NIH – fear of change
Little intellectual curiosity
Market noise – new tools
Threat of disruption from
output to input
Too little data
Legal implications
Technical
stakeholders
Business – Domain
stakeholders
Input Process Output
Proving ROI
Reputational risk
Trust and proprietary data
Most challenging issues
© 2013 Sandro Catanzaro 14

15. Reporting vs Analytics
Just a “report writer software” will do it
• Analytics = curated reports
– Curation: identifying what matters, editing out what doesn’t
– Joining the dots
• Analytics is a Bridge
to the right insight
– Not a pile of needles
to replace a pile of hay
– Decisions are made with
just THE ONE needle
– Careful you all detail oriented nation!
© 2013 Sandro Catanzaro 15
Misconceptions

16. Do some analytics for me
• Many problems are not clear
– Domain experts and business managers, even
you, don’t know boundaries of what can be
answered – “catch 22 situation”
– Even worse, the problem existence is unclear
• A solution is identified before the problem is
clear (this is really bad)
– The myth of big data – and the heavy lifting in
data collection
• Analytics are perceived as a “nice to have” by
less technically oriented stakeholders
– Do it on your free time
You are smart, you can come up with
something interesting, can’t you?
Unclear need
Unwillingness to pay
© 2013 Sandro Catanzaro 16
Challenges

17. Make sure the answer is X
• Clear questions and enough data
may still provide the “wrong”
answer
– May not make business sense
– Or not be of the liking of business
• Or, may require complex answers
difficult to digest by intended
audience
• Or, politics may be a stronger
driver for the decision process
Answer not simple
enough
Output contradicts
business
Politics
© 2013 Sandro Catanzaro 17
Challenges

18. Look at the big picture AND focus!
• Good analysts who are willing to see the
big picture are scarce
– Hiring analytics management is harder: Peters
Principle
• Moving too fast or too slow in tool
selection
– In the middle between bleeding alpha and cobol.
Think “second best”
• Analysis “gold polishing”
– Some of us may be perfectionists
• … and finally, there intrinsic risk
that the analysis may not work
Scaling up the team
New toolset dynamics
Thinking takes time
R&D type of risk
© 2013 Sandro Catanzaro 18
Challenges

19. The right data, all in one place
• Data is often in the wrong format or has
rows that need to be ignored
– Cleaning is tedious, often manual
• More often than not, there is superfluous
data that we keep “just in case”
– Bogs down processing, blows out costs
Dirty data
Too much data
Far away data
• Data is rarely
centralized
– May require
collaboration:
“What is in it for me”
© 2013 Sandro Catanzaro 19
Challenges

20. FRAMEWORK
(TO MAKE MAGIC WORK)
© 2013 Sandro Catanzaro 20

21. Where to start
Clear and
present need
“you are lucky”
Need is not
crisp, but challenge
is known
“we just need
Analytics”
Lack of
awareness about
challenge
“Beware”
Crisp and clear statement of the
“question to answer”
Can you tell us how sales correlates
with factors A, B, C, D and E; using
a quarter by quarter dataset, and
data from 2010 to date
Please do some
attribution modeling
good bad
© 2013 Sandro Catanzaro 21

22. The pull model
Starts from the end
Business
problem
Audience
Is this truly
important to you?
Are you willing
to pay for it?
Analysis
output
(analysis
tools)
Processing
(aggregation, p
rocessing
technology)
Dataset
(storage
technology)
Data collection
(data
sources, pick, p
arse cleanup)
© 2013 Sandro Catanzaro 22

23. The pull model
Starts from the end…
© 2013 Sandro Catanzaro 23
…You would cross the “bridge” 3 times
• First time from where you are, to the business stakeholder and problem
• Second time from the business problem, all the way back to the data, so that
every piece “pulls” the next previous one
• With business problem in hand, you will know the minimum analysis
output that provides the answer
• With the analysis in hand, you will know analysis tools needed to
prepare, and also what is the processing and aggregations needed to
support it
• With the aggregations in hand, you will know the minimum dataset
required
• .. All the way to data collection
• Third time from the data, you will come back as you execute the analysis, step
by step to finally arrive to the desired answer

24. Continuous iteration
Crisp
question
statement
Stakeholders
Stakeholders
Stakeholders
Stakeholders
“the contract”
• Share interim results: continue selling on value, continue selling on model belief
• May modify the “crisp question” – ensure you also reframe time, resources, cost
• Whether you’ll share the methodology depends on the audience
• Creates dialog and sense of shared ownership in the answer
Mockup
answer
(Answer
First)
Back of
the
envelope
answer
Complete
answer
Incorporate feedback
Share output
(maybe methodology)
© 2013 Sandro Catanzaro 24
Nothing fancy here
Just risk mitigation /
project management 101…

25. Be flexible
• At each iteration, the question may move…
– Nothing is worse than answering the wrong question
– What is valuable may evolve
• … yet ensure you “get your victory lap” if the question moves
– Thinking evolved thanks to analysis
– Re-evaluate timing, resources
• Practical signs you’re on the right track:
– Experienced stakeholders confirm your findings by ‘gut’
– They ask a LOT of clarifying questions & recommend potential next steps to
your analysis/message
© 2013 Sandro Catanzaro 25

26. Best models: 70% belief - 30% math
• Simplify
• Share with all shareholders – get them to believe in the
answer and model
• Simplify
• Acquire buy in – “prewire meetings”
– Must have on board the P&L owner
– Also recruit influencers, other stakeholders
• Simplify again
– Review and clean up model
– Remove unneeded items, keep just enough to prove your point
– Maybe add a single cherry (non-requested-by-so-cool-to-know insight)
Be careful on
adding accuracy
at the cost of
complexity
© 2013 Sandro Catanzaro 26

27. Risk mitigation
• Understand your project risks and tackle them in order
– The scariest first – even if you don’t solve it, you’ll raise
awareness, understand it better, adjust timeline if needed
– Leave the easiest for last – even if out of sequence
• Ensure you can create full end-to-end passes with increase
accuracy each time
© 2013 Sandro Catanzaro 27
Nothing fancy here
Just risk mitigation /
project management 101…

28. Scaling the team
• Best bet is for introducing geeks, into big
picture thinking
– Smarts beat expertise
– “Spin Doctors” are useful, to a point
• Aim for “T shaped” people
– Broad big picture understanding
– Deep expertise in one area
• Provide context
– Make sure team understands where
does the analysis fit
• Be willing to let go
– Understand costs of potential mistakes
– Responsibility is never delegated
© 2013 Sandro Catanzaro 28

29. Accessing and processing data
• Data accumulates “close” to the data user (formats, quirks)
– Data that is not used is rarely correct, updated on time
• Help from data users is critical, yet them may feel threatened
– Create quid-pro-quo partnerships. Giving credit can be enough
• Avoid changing technologies mid-way a project
– If new tech advantages are leaps and bounds above, parallel run both
until the new replicates results
© 2013 Sandro Catanzaro 29

30. The Truth
• The value of Analytics team is guiding decision-making using
fact-derived insight
• Not easy or in some cases impossible
– Not enough data or tools to arrive to conclusion
– Business pressure to change the answer
– The analysis is not ready in time (pesky bugs?)
• Never ever yield to pressure, and present an answer that
does not follow from analysis, data, facts and correct
mathematical foundation
Analysis that can’t be trusted is worse than no analysis.
Providing such, damages your personal reputation

31. SOME ANECDOTES
© 2013 Sandro Catanzaro 31

32. Make digital attribution
Large banking institution, looking to show
innovation:
“We want to know the value of the ad
sequencing
So, please prepare an attribution model for
us
• Make it say something interesting
• Actionable
• … and that we haven’t heard before”
© 2013 Sandro Catanzaro 32
Beware of unclear demands
Use constructive questioning
to build an idea of “what is it”
Share with actual decision
maker within customer and
get feedback
Incorporate feedback to make
model ownership shared with
customer
Use a mockup made with
fabricated data (for sharing
before analysis is complete)

33. What can you say about
their personal values?
Ad agency executive for a
large CPG account:
“Using the data you are
seeing from our ad
campaign, what can you tell
us about our consumers’
values?”
© 2013 Sandro Catanzaro 33
Adjust expectations about
what is possible, early
If possible, avoid saying NO.
Better to re-interpret question
and provide a possible answer
Other cases are “doable” but
at great cost; if so, charge
enough, and ensure burden is
understood
In some cases no alternative
but saying NO: No intent on
paying, no real usage for
analysis

34. Create wizards: One, two, three, go!
Executive to analytics
manager and recruiting
manager all in one:
“We need you to build our
new Analytics group from
the ground up”
© 2013 Sandro Catanzaro 34
Create a project plan and
secure enough funds for hiring
Don’t over-commit yourself:
you cannot be hiring
manager, star
analyst, trainer, and main
client contact.
Delegate in the order of the
previous list (avoid being
HR, try to keep client contact)
Be prepare to adjust
expectations if you find
yourself over-committed
without wanting

35. Sandro’s Analytics principles
1. Start with the business/domain problem. Never with data
2. Don’t ever ignore humans – especially the one who pays
3. 70% belief – 30% math
4. Teach smart geeks to think big picture & let go
5. Simplify - weight complexity vs. accuracy
6. Quick and good enough wins the day
7. Model and iterate – rinse and repeat
8. Tackle risks from riskiest to least risky
9. Be flexible in scope, but take victory lap and re-budget
10. Never compromise with “the truth”
© 2013 Sandro Catanzaro 35

36. APPENDIX
… Because you always need to move
70% of slides to their appendix…
© 2013 Sandro Catanzaro 36

37. No ROI
• In theory, you should' t care
• In practice, is critical to
understand the value of your
work, and the big picture
– Ask questions, understand
how your answer fits in
• Good fitness means
– Project stability
– Organizational visibility
– Less re-work
Thanks to Vishal Kumar for the graphics – stolen from his blog
“The project has been cancelled – sorry”
© 2013 Sandro Catanzaro 37