Main Takeaways: -It’s important to define success metrics before you start building your product or feature -The goal is to validate a hypothesis--what you think users want--and the data might invalidate your hypothesis. This is a good thing! Keep iterating in that direction to identify and test the next hypothesis. -Data is important to point you in the right direction, but it won’t answer “what is the perfect product?” To figure this out, you need to experiment, fail and repeat as fast as possible to find success.

1. www.productschool.com
Data - How to Use it & When by Square and
CallRail Product Leader

2. CERTIFICATES
Your Product Management Certificate Path
Product Leadership
Certificate™
Full Stack Product
Management Certificate™
Product Management
Certificate™

3. Corporate
Training
Level up your team’s Product
Management skills

4. Free Product Management Resources
BOOKS
EVENTS
JOB PORTAL
COMMUNITIES
bit.ly/product_resources
COURSES

5. The business intelligence and analytics platform for
every need
www.gooddata.com
Event is Sponsored by Lucid

6. Data: How to Use it and When
Aaron Szerlip, Product Manager
June, 15th 2021
Katie Salley, Sr. Product Manager

7. Introductions

8. Introductions
Aaron Szerlip
Katie Salley
Podcast: What Would You Say You Do Here?
Website: https://theproductmanagers.com/
Instagram: @theproductmanagerspodcast
Product Management for
10 years at Kabbage,
Cox Communications,
Pindrop, Samsara and
now Square.
Product Manager for 8
years at BetterCloud,
Pindrop, and now CallRail.

9. “If we have data, let’s look at data. If all we
have are opinions, let’s go with mine.”
— Jim Barksdale
CEO, Netscape

10. Lifecycle of Data for a PM

11. Product Insights
Stages of Needing Data
Sources of Data
● CRM
● Customer feedback
● FRs associated with sales pipeline
● Customer requests, interviews
● App store reviews
● Support ticket volume for FRs /
Problem areas
Questions to Answer
● Who is using the product?
● Who hasn’t bought the product?
● Who started using the product but stopped?
● What job was the product hired to do?
● What similar jobs hire other products?
● What do customers think of the product?
● What do customers complain about?
● What do customers love about the product?

12. Product Insights
What do you want to learn?
● Define both product and user goals
● What do customers think of this
feature?
● How does this product do the job
compared to a competitor?
Hypothesis
Validation
How are you going to learn it?
● Interviews
● User experience research
● Surveys
● Live mockup testing
● Feedback
● Actual product usage
Stages of Needing Data

13. Feedback
Product Insights
Hypothesis
Validation
Where does feedback come from?
● Data derived from your product
● Support tickets (volume, categories)
● User Interviews (new customer surveys, NPS)
● CSAT
● Let’s discuss NPS - WSJ found no one says the
score ever goes down...
Stages of Needing Data

14. Feedback
Product Insights
Hypothesis
Validation
Evaluation
Did you accomplish your goals?
● Did you meet your success
metrics/goals?
● Are customers using the feature?
● Is the feature doing it’s job?
● Are customers using the feature the way
it was designed to be used?
● If customers aren’t using the feature,
what are they doing?
● Did you learn anything?
Stages of Needing Data

15. How to Measure?

16. Metrics 101
Good: Ratios make good metrics
e.g. clicks per session
● Pair these with cycle time metrics to give
a more complete view of product health
● Use guardrail metrics to ensure you aren’t
optimizing one metric at the expense of
your product’s likability/usability
Bad: Vanity metrics
e.g. DAU
● They don’t show directional change
● They don’t correlate to product or
business health
Keep in mind...
● Reliable metrics require clean, reliable
data, i.e. GIGO
● Leading vs lagging metrics
● Choose metrics relevant to your business
● Define your target success metrics before
you experiment
Multiple ways to measure
● Netflix: How many hours of video watched
per user per week
● Spotify: How many hours of music listened
to per user per week
● Reddit: Level of engagement, e.g. lurkers vs
infrequent contributors vs highly engaged

17. Types of Metrics
Example metrics depending on what you’re trying to measure:
● Funnel analysis: Click rate --> sign-up rate --> conversion rate -->
abandonment rate --> churn rate
● Customer valuation: Customer Acquisition Cost (CAC) vs
Lifetime Value (LTV), % of paying users, ARPU
● Customer sentiment: NPS, CSAT, app-store reviews
● Customer engagement: open rates, click-through rates,
engagement time, sessions to click ratio
● Feature gap analysis: Feature requests tied to pipeline revenue
● Product performance: Uptime, page load time, qps during peak
traffic
● Support ticket volume Feature requests /complaint categories
● Quantitative vs qualitative

18. Qualitative data helps you understand why.
E.g. A survey showed that users didn’t
understand how to use the new feature
Qualitative
Quantitative
Quantitative data helps you understand what
is happening.
E.g. 99 users clicked on the new feature, but
only stayed on the page a few seconds

19. The Limitations of Data
Data is not a substitute for talking
directly to a customer.
Only customer feedback can reveal the
true jobs to be done that your customers
are willing to hire you for.
Get out of the office and
talk to people!

20. Resources
Lean Analytics
Lean Startup
Measure What Matters

21. Key takeaways
Define success metrics early
Validate your hypothesis
Experiment, fail, repeat (fast)

22. Thank You!
Podcast: What Would you Say You Do Here?
Website: https://theproductmanagers.com/
Instagram: @theproductmanagerspodcast

23. www.productschool.com
Part-time Product Management Training Courses
and
Corporate Training