SpringOne Platform 2018 Data Driven Decision Making for Product Development Zoe Vance & Denise Yu, Pivotal

1. Data Driven Decision Making
By Denise Yu & Zoe Vance
@deniseyu21

2. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Hello SpringOne Platform!
2

3. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Why is this topic important?
Velocity is a vector: a function of
both speed and direction.
Breaking down, framing and
quantifying problems helps
ensure that we’re always moving
in the right direction.
3


4. Data in service of decision making

5. 5

Tools to help you take a problem, break
it down, quantify it, understand it and
make a decision
Goal

6. Quantitative vs Qualitative
Sales increased 10%
Marketing reached 150 fewer
people this quarter
Development speed increased
15%
Users are enjoying the new
premium features
We sent too any promotional
emails last quarter
Our new processes helped the
team move quickly

7. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Known Knowns
Source:
https://www.filmhousecinema.com/showing/mountain

8. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Known knowns: We all know we have a problem
There’s an issue on the team and everyone knows about it
BUT
there is a lack of consensus on how to tackle it
8


9. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Our pipelines were taking ages to run
9

example Concourse deployment pipeline (concourse-ci.org)

10. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Break the problem down and quantify
1
0

Source: Maria Ntalla, Maya Rosecrance, Sarah Connor,
Jack Newberry, Oliver Stenbom

11. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Keep improving and measuring
11

Source: Maria Ntalla, Maya Rosecrance, Sarah Connor,
Jack Newberry, Oliver Stenbom

12. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Takeaways
1
2

Decision enabled:
Focus attention on a problem, break it down so it’s
addressable and quantifiable
Metrics chosen:
Time overall, time per task

13. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Known Unknowns
Source: Maria Ntalla, Maya Rosecrance, Sarah Connor,
Source: https://www.minube.net/photos/place/695291

14. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Known unknowns: Something is wrong, but
we’re not quite sure what
Situations where the issues are unsurfaced but there may
be some awareness
1
4


15. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
1
5


16. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Assess the value of what you’re trying to learn
1
6

We didn’t really understand the number or what it meant when we hit/didn’t hit it.

17. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
We realised we had different types of users!
1
7

One group of users built new
things with frequently-pulled data
Another group copied data (huge
infrequent pulls from data sources)

18. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Surfaced bigger existential questions and
debate
1
8

What are we building? For who? What do we want to be?
Measure our success against that goal

19. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Takeaways
1
9

Decision enabled:
Type of product we want to be for whom.
Metrics chosen:
Basic starter metrics
Patterns in user behaviors*
* complemented with qualitative information!

20. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Unknown Unknowns
Source: https://ua.wallpapers-fenix.eu

21. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Unknown unknowns: We don’t even know we
have a problem
Surface and acting on nebulous issues
2
1


22. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
How is the team doing?
2
2

Spotify Health Check
Source: labs.spotify.com/
2014/09/16/squad-
health-check-model/

23. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
2
3

Tools & Technologies that we use
Concourse
BOSH
Go
Github

24. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Concourse
BOSH
Go
Github ...
How important for
our success? (1-3)
3
3
1
2
2
4


25. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Concourse
BOSH
Go
Github ...
How important for
our success? (1-3)
3
3
1
2
Where are we
now? (1-3)
1
2
1
2
2
5


26. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Concourse
BOSH
Go
Github ...
How important for
our success? (1-3)
3
3
1
2
Where are
we? (1-3)
1
2
1
2
2
6


27. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Concourse
2
7

Where are we as a
team now? (1-3)
3
What worked?

28. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Takeaways
2
8

Decision enabled:
Spend more time becoming a stronger team
Metrics chosen:
Self-reported assessment

29. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Framework to apply this to your
work

30. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Step 1: Identify a problem (ex: Stinky Fish)
3
0

What are the biggest
pains?
What problem, if we
continue to ignore it, will
become the biggest pains
down the line?

31. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Step 2: Break down and frame the one problem
3
1

Inaccessible
Big
Nebulous
Accessible
Small
Addressable

32. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Step 3: Measure quantitatively to learn
qualitatively
Measurements you choose don’t
need to be perfect or a target. But
they should be good enough to
provoke conversations and
learnings - as you learn more, you
can decide if you need high-fidelity
data or just a lightweight insight.
3
2

Source: http://www.goodwinandgoodwin.com/

33. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/
Based on your learnings,
iterate and improve how
you measure and assess
Step 4: Iterate & Learn
3
3


34. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license:
http://creativecommons.org/licenses/by-nc/3.0/

35. Take a hard problem
Frame it
Break it down
Measure it
Understand it
3
5


36. Thanks to…
Maria Ntalla
Therese Stowell
Ruth Maclean
3
6


37. > Stay Connected.
Talk to us! zvance@pivotal.io & dyu@pivotal.io
#springone@s1p