Main takeaways: - The most valuable thing Product Managers can contribute is discovering the right product to build - the product or feature that is going to meaningfully move the business forward - Common mistakes teams make during the product discovery process - Provide a framework for thinking about the ideal way discovery fits into the overall product development process

1. www.productschool.com
How to Avoid Common Mistakes in
Product by Cake Product Manager

2. FREE INVITE
Join 30,000+ Product Managers on

3. COURSES
Product Management
Learn the skills you need to land a product manager job

4. COURSES
Coding for Managers
Build a website and gain the technical knowledge to lead software engineers

5. COURSES
Data Analytics for Managers
Learn the skills to understand web analytics, SQL and machine learning concepts

6. COURSES
Digital Marketing for Managers
Learn how to acquire more users and convert them into clients

7. COURSES
Blockchain for Managers
Learn how to trade cryptocurrencies and build products using the blockchain

8. Justin Williams
TONIGHT’S SPEAKER
@ProductEvents

9. Product Discovery
Justin Williams

10. Goals Of This Talk
● Give you practical techniques PMs use to figure
out what to build
● Talk through common mistakes and where you
might go wrong
● Give your a framework that ties it all together

11. My Journey

12. My Journey
Dr. Fadem purchases questionable man bag to
show enthusiasm for the new product

13. My Journey

14. Product Discovery Purpose
Product Discovery Answers
● Will they buy/use it? (Value)
● Can they use it? (Usability)
● Can we build it? (Feasibility)
● Can our stakeholders/business support it? (Viability)
● Get evidence that our solution will work
● Describe what will be built in Delivery
Source: Inspired by Marty Cagan

15. Product Discovery
Techniques

16. Product Discovery Techniques
● Testing for Demand. Techniques to answer the question, does anyone
even want this?
● Generating Ideas. Everything starts with ideas - but where do they come
from? While stakeholders and customers are typically the worst sources -
these methods allow you to come up with ideas worth pursuing.
● Validating Solutions. Once we have ideas for solving the problem... how
do we know if we should build it? These methods help determine if your
solution is worth building.
● Framing Discovery. Techniques to get everyone on the same page about
what problem we are solving and what success looks like.

17. Testing For Demand

18. The Landing Page
Create a product pitch page marketing the new offering exactly as if we
were really launching
● Create pitch page + CTA (call to action)
● CTA lands on page explaining we are studying the possibility of adding
this new offering
● Collect email address on this page if they’d like talk with us about it
● Drive a small amount of traffic to the landing page
● Measure conversion vs. expectation

19. The Landing Page

20. The Fake Door
Create a button/link to a potential new feature/product/service and place
it into the existing experience where we believe it should be
● Instead of getting the new feature, the customer gets a page that
explains we are studying the possibility of adding this feature and we
are seeking customers to talk to about it, with the ability to leave their
email address if they’d like to chat with us about it
● Customer should not have any visible indication that this is a test
● We expose the fake door a small percentage of customers and
measure click-through rate vs. expectations

21. The Fake Door

22. The Explainer Video
Create a video demo of the product idea using a prototype
● Create a video showing off our new product concept using a
prototype
● This can be hosted on a landing page or distributed however we
see fit
● Have a CTA that allows users to sign-up for an early version
● Measure conversion vs. expected

23. The Explainer Video

24. Generating Ideas

25. Observation
We shadow or observe users as they do their job or go
through their workflow
● Observe frustrations, struggles, and work-arounds
● We then conduct an interview based on said observations
(contextual inquiry)
● Single best method for quick, extensive learning

26. Observation

27. The Concierge MVP
This technique replaces the product concept with humans
● The idea is to test a product hypothesis without building
anything at all
● Humans deliver the service manually, personally interacting with
customers
● In the process of delivering said service, you’ll learn a ton about
your customers and the value you are offering

28. The Concierge MVP

29. Customer Misbehavior
Review esoteric uses of the product by existing customers
● Customers using our products to solve problems other
than what we planned for and officially support
● Do some homework to figure out what problem they are
trying to solve and why they are using our product to do so
● Build/pivot to support this new use case

30. Customer Misbehavior

31. Customer Misbehavior

32. Customer Misbehavior

33. Customer Misbehavior

34. Experience Sampling
We ask participants the same question repeatedly at predetermined
intervals using an app
● Allows us to understand behaviors and attitudes in real time by
sampling current behavior - which is the best predictor of future
behavior
● Explores current behavior without a product in order to understand
frustrations and identify product opportunities
● Repetition allows for collecting a lot of data in a short amount of time
and recognizing patterns/commonalities

35. Experience Sampling
Source: Don't Listen to Users, Sample Their Experience!

36. Customer Interviews
Dive Deep in 1-1 interviews
● Not focus groups. Not surveys.
● Depth > Breath
● Explore The Switch. Making a purchase or quitting a service.
○ Pushes & Pulls: External / Internal factors moving them
toward the new solution
○ Anxieties & Habits: forces keeping them with the existing
solution

37. Customer Interviews
The Mattress Interview

38. Customer Journey Map

39. Customer Journey Map

40. Customer Journey Map @ Cake

41. Discovery (Design) Sprints
Popularized by Jake Knapp at Google Ventures
● Assemble a group that can dedicate a week to working on a specific
problem
● Start on a Monday and schedule 5x User Tests for Friday
● Build shared understanding of the problem, existing solutions, and
competition
● Go through a series of exercises to get all ideas out on the table
● Pick 1-3 ideas to prototype based on what resonates with the team
● User Test ideas and iterate

42. Design Sprint @ DaVita

43. Validating Solutions

44. The User Test
Simulate the intended experience using a prototype
● Ask representative users to perform representative tasks
● Observe what works and where they run into issues
● Measure: Task Time, Error Rate, Completion %, SEQ
● Prototype serves as spec
● Don’t just test for usability

45. “Easily the most effective thing we do is one-
on-one user interviews. We’re not using retina
tracking or any of that crap. Just asking the
right questions of somebody and watching for
their reactions, having them be able to react to
a prototype, we do this type of research all the
time.”
- Daniel Burka, Design Partner @ GV

46. User Test: Parallel Design
Source: NNG

47. User Testing IRL
User Test Example + Prototype Example

48. User Test: Script
● Interview: start with an interview before the test.
● First Impressions. Do they “get it”?
● Usability Test. Can they complete the task(s)?
● Value Test. Will they give up something?
● Value Discussion. What will it take?
● Content Test. How well does our copy work?

49. The Content Test
Copy is an incredibly important but often overlooked part of our
experience
● Larger companies hire entire teams of UX Writers to work on copy
● Smaller shops can easily test copy by printing out the way we
market/describe/explain a concept and asking users to highlight
things that resonate in one color and things they don’t like in another
● This quickly allows us to understand if our copy is effective and which
messages are working with customers
● Often can be done right after a user test

50. Content Testing @ Cake

51. Content Testing @ Guild

52. Content Testing @ Guild

53. The Live Data Prototype
We build a minimal feature using real code
● We expose this to a small number of customers via an A/B test
● Must be able handle real customer usage
● Must have analytics in place so we can understand performance vs. the
existing feature
● Must work well enough that it doesn't hurt our brand
● We try to write just enough code to validate/invalidate the idea; the goal would
be to only spend 20% or 30% of the time that would be required to write a
production quality feature
● While we are loath to use dev time, we get statistically significant validation,
reusable code, and we can also run user tests on the minimal feature

54. The Live Data Prototype

55. The Live Data Prototype

56. The Wizard of Oz
We use people behind the curtain to simulate a product experience
● We use manual processes in place of automated solutions as a
means of validation and rapid iteration
● The customer believes they’re interacting with an automated
product, but in reality a human is pulling all the levers and
delivering the service
● If users want your service when a human’s behind the curtains,
then you can be confident that they’ll like the automated version
of your product (given that quality of service doesn’t decline).

57. The Wizard of Oz

58. Framing Discovery

59. Opportunity Assessment
An Opportunity Assessment describes:
● What problem are we trying to solve?
● Who are we solving for the problem for?
● How will we know that we have succeeded?
● Who will be involved? (The Discovery Team)
● Is there a time-box?
● Who will we report back to? (Stakeholders)

60. Internal Press Release
An Amazon Technique:
● Create a press release
pretending you have finished the
<product/feature/redesign> and
are about to launch
● The audience is the new
product’s customers
● 1.5 pages max
● Oprah-Speak not Geek-Speak
Structure:
● Heading
● Summary
● Problem
● Solution
● Quote from PM
● Customer Quote
● Closing / CTA
Example: Amazon Catalyst Example

61. Customer Letter
An imagined letter from the perspective of one of your product’s well
defined customer personas
● Good for larger efforts; major redesign, new feature, expansion
product
● The letter is from a very happy customer to the CEO
● Explains why he/she is grateful for the new product
● How it has improved his/her life
● Includes congratulatory response from the CEO to the product
development team explaining how this new product has helped move
the business forward

62. Product Discovery
Mistakes

63. You Listen To What
Customers Say They
Want1

64. Don’t Just Talk To Customers
“The biggest mistake you can
make is talking to
customers.”
“The second biggest mistake
you can make is not talking to
customers.”
- Jeff Bezos

65. Don’t Just Talk To Customers
● Sony’s walkman cassette player was
put on hold when market research
indicated that consumers would
never buy a tape recorder that didn't
have the capacity to record and that
customers would be irritated by
headphones.
● Akio Morita advised against market
research, advocating instead to
“Watch how people live and get an
intuitive sense of what they might
want.”

66. You Don’t Test For
Value2

67. Value Test
Once we’ve done the usability test, we can now have a
meaningful conversation about value (or lack thereof)
● Most people are nice, unwilling to tell you what they really
think, and will generally avoid conflict
● Therefore we force the conservation by asking them to
give up something in order to use the product
● Note: we often have no intention of taking whatever they
give up
Source: Inspired by Marty Cagan

68. Value Test
● Money. Ask user to pre-order the product by typing in their credit card info.
For large purchases, ask to sign a Non-Binding Intent to Buy Agreement.
● Reputation. Ask them to enter FB creds to share on social media. Ask
them to enter email addresses of colleagues which will send them a
product recommendation
● Attention. Ask if we can switch them over to the new version when it
comes it out. Ask them to enter login credentials of whatever product they
would be switching from into a fake migration utility.
● Time. Ask them to sign a beta program in which we will work together to
build out the product with a non-trivial time commitment.
Source: Inspired by Marty Cagan

69. You Outsource
Research3

70. Outsourcing Research
● Usertesting.com, Validately
● The team never builds customer
insights
● Findings are never read or
discounted
● You never get a chance to following
interesting leads
● Obscenely expensive
● Career limiting

71. Affirmation vs.
Information4

72. Affirmation vs. Information
● Using discovery to green-light pre-existing ideas
● Pushing back on people who question if it was informed
by real customer insights
● Not testing on actual customers
● Not testing the things that matter
● Trying out only a single or minimal number of ideas
● Number of ideas tested is similar to the number shipped
Further Reading: Product Discovery Anti-Patterns

73. Going It Alone
5

74. Going It Alone
● UX only or Product-UX only
● Engineers and Data Scientists
are often our best source of
ideas
● Technology, Functionality, &
Design are intertwined
● Allays the “grand reveal” or
“big handoff” issue which
then requires a ton of
documentation
Further Reading: Product Discovery Anti-Patterns
I’m sure we’ll be fine...

75. Putting It All Together

76. Putting It All Together
1. Define Objective/Problem
2. (Optional but useful) Create Discovery Plan
3. Get all the ideas out on the table
4. Pick some ideas to validate as solutions
5. Build / Iterate / Shelve

77. When Do We Stop Iterating In Discovery?
● Use professional judgement
● Have we mitigated key risks and assumptions?
● 5x consecutive user tests that do well across Value, Usability,
Feasibility
● Looking for evidence our solution will work; not statistically
significant proof
● “Every test is a piece of the puzzle. We don’t know if we are
going to get the whole thing after 5 or after 50.” - Marty Cagan

78. Continuous Discovery & Delivery Framework
Objectives
Discovery
● OKRs
● Goals
● Vision
● Strategy
Delivery
Techniques to determine:
✓ Valuable
✓ Useable
✓ Feasible
✓ Viable
✓ Performant
✓ Scalable
✓ Maintainable
✓ Analytics
✓ Automated Tests

79. Q&A
jw@zerofd.com

80. www.productschool.com
Part-time Product Management, Coding, Data, Digital
Marketing and Blockchain courses in San Francisco, Silicon
Valley, New York, Santa Monica, Los Angeles, Austin, Boston,
Boulder, Chicago, Denver, Orange County, Seattle, Bellevue,
Toronto, London and Online