Data is king! Get ready to understand how a successful analytics team can empower managers from product, marketing, and other areas to make effective, data-driven decisions. Louis Cialdella, a data scientist at ZipRecruiter, shared some case studies and successful strategies that he has used at ZipRecruiter as well as previous experiences. The purpose of this data talk was to enlighten people on how to make sure that analysts can successfully partner with other departments and get them the information they need to do great things.

1. Better Living Through Analytics - Strategies
for Data Decisions
www.productschool.com

2. FREE INVITE
Join 12,000+ Product Managers on
productschool.com/slack-community

3. Louis Cialdella
Tonight’s Speaker

4. Better Living Through
Analytics
Louis Cialdella
August 29th, 2017
4

5. 5
Today’s topics
About me and ZipRecruiter
What does the analytics team do? What are our projects like?
Strategies for successful partnerships
Some things I wish I knew when I started

6. 6
Mostly ZipRecruiter, though
About Me and ZipRecruiter

7. 7
About Me
Columbia University - Computer Science and Machine Learning
Worked at startups in New York, work on the credit risk analytics system at JP
Morgan
Currently a data scientist on the analytics team at ZipRecruiter

8. 8
ZipRecruiter
Short version: We help people find jobs
Employers post jobs, we help them find qualified candidates
#1 job search app on iPhone!
Located in Santa Monica
We’re hiring! For analytics and product positions - Resumes to
louisc@ziprecruiter.com

9. 9
Some challenges we face at ZipRecruiter
How can we match employers with jobseekers in a way that benefits both?
Who should we market our service to?
How can we improve our user experience?
How can we guard against fraud on our platform?

10. 1
0
What we do and how we do it
Analytics at ZipRecruiter

11. 1
1
The Role of analytics at ZipRecruiter
Help business stakeholders make data driven decisions
Other departments have domain knowledge and problems to solve, we
supply statistical skills
Define metrics
Answer vague business questions with well-defined data analysis

12. 1
2

13. 1
3
ZipRecruiter’s Analytics team
Follows the “centralized” model
Independent department which provides data and statistical analysis
Advisory capacity - help other teams understand their data and figure out
which decisions will benefit them the most

14. 1
4
ZipRecruiter’s Analytics team
Follows the “centralized” model
Pros:
Makes it easy to build up institutional knowledge
Can build and share analytics tools
Independent incentive structure
Cons:
Requires skilled analytics-specific leadership
Further from domain experts
My opinion: Given the rapidly changing state of industrial analytics, pros outweigh
cons

15. 1
5
Bringing out the best in each other
Analytics + Product = 😊

16. 1
6
Analytics can help make great products
Product managers have vision, and data analysis can help advise on the best
way to execute it
Two main ways:
Product Optimization:
Example: A/B testing different user experiences
Machine intelligence Integration:
Example: Making recommendations to users, dynamic pricing, fraud
detection

17. 1
7
Successful partnerships
A balancing act:
Product people are incentivized
to do cool things as fast as
they reasonably can
Analytics people incentivized to
be rigorous and careful
Pragmatism vs rigor
Fast vs slow

18. 1
8
Analytics projects
It helps to understand how an analytics project is structured
For a data-driven project to succeed, we need to:
Collect the data
Run experiments, look at historical data, etc
Analyze the data
Build a model, extract insights, etc

19. 1
9
Analytics projects
What insight will
be extracted?
What information
will the model give
us?
What mechanism
collects the data?
Where will it be
stored and
accessed?
What is the
question we want
to answer?
What data will be
used?
What modelling
approach will be
used?
What software will
be used?
Data Collection
Data Analysis
Design Implementation

20. 2
0
Analytics projects
Some anti-patterns:
“Crunch these numbers for me”
Analysts need context
“Do some data stuff with all our historical data”
Analysts benefit from a clear product vision
“We ran this experiment, what does it tell you”
If an analyst did not help design the experiment, the data may not
be useful

21. 2
1
Adventures in Product Optimization
Case study: Free trial optimization

22. 2
2
A typical request
ZipRecruiter has a free trial for our employer subscription product
During the free trial, employers post jobs and get applications
If we knew how the number of applications related to the conversion rate, we
could design our experience to improve this
How can we understand the relationship between number of
applications per job and likelihood of conversion?

23. 2
3
A first look
We have historical examples of how many applications/job were received, as
well as whether or not a user converted
We could compare the response counts of the convert/non-convert groups
to describe the data
But this doesn’t allow us to make predictions - what we want is a model!

24. 2
4
Understanding our data
We have a few years of historical data about:
Hundreds of thousands of free trials
How many jobs each one posted, which jobs received applications
Whether or not they converted
Geographic and industry information
Our model will need to relate the features (geo/industry info, number of
applications) to the output (conversion event)

25. 2
5
Error bars
Just specifying the expected value of the conversion rate at each response
count can give a false impression of precision
We want to quantify our uncertainty
Solution: Add confidence intervals to estimates (via bootstrap)

26. 2
6
Analytics projects
Can we produce a
curve defining the
FT conversion vs
response
relationship?
The data used is
collected in our
SQL (Redshift)
database
What is the
relationship
between FT
conversion and
responses?
Use historical data
Model: Logistic
regression
Software: SQL
and Python
Data Collection
Data Analysis
Design Implementation

27. 2
7
The result
The curve which describes the
relationship between application
count and conversion
Can use this to optimize the free trial
give our customers the best
experience (for example, tuning
the length of time the trial lasts)

28. 2
8
Strategies for success
Things I wish I knew when I
started

29. 2
9
Uncertainty is the only certainty
Analytics helps us understand where there is uncertainty, but it usually can’t
be brought to zero
Nate Silver: “[Some forecasters] see uncertainty as the enemy...this tends to
leave us less prepared when a deluge hits.”
Analysts deal with uncertainty by mitigating it where possible, and
communicating it where not possible
Use confidence intervals and similar techniques, which provide best/worst
cases

30. 3
0
Don’t be afraid to experiment
Experiments have become common in the industry (A/B tests)
But there is a cost associated with running them - when should we run them?
Answer - whenever possible!
Product can often get great insight by running an experiment
Analytics can often provide much more definitive results and mitigate
uncertainty as much as possible

31. 3
1
Understand Everyone’s Incentives
Analysts should report revenue/profit/cost impact, dollars and cents
Product folks can help by making it clear what needle you want to move,
even if it’s a big picture metric
Analysts are responsible for translating their findings into a language that a
business user can use for decision making

32. 3
2
Understand the choices, make a recommendation
Try and help people make data driven decisions by understanding the
choices they want to evaluate
Product - present the strategies you are considering
Analytics - focus on making specific recommendations, rather than simply
conveying the results of number crunching

33. 3
3
Focus on the small picture
It’s tempting to totally overhaul a system and replace it with DEEP
LEARNING THE BIG DATA
But big overhauls are risky
At each step in modelling, you may make bad assumptions - rolling out
incremental improvements allows you to check these assumptions
When trying to improve a process, don’t overhaul it from the very beginning -
start with small improvements to the existing method

34. 3
4
Summary
● ZipRecruiter’s centralized analytics team model has a lot of
advantages
● Analytics + Product = 😊
● Uncertainty is part of the process, but we can do a lot to mitigate it
and communicate it clearly

35. 3
5
About my work? About ZipRecruiter? About working in data science?
Q & A

36. 3
6
Thank You

37. Part-time Product Management Courses in
Silicon Valley, New York, Los Angeles, and
Orange County
www.productschool.com