Evaluating input from your customers when the feedback is based on their feelings, emotion, or aesthetic preferences is already a challenging. Anytime you get a spike the amount of user experience feedback you receive—like we did during a beta of a major redesign of one of our products—your project team can be quickly overwhelmed without a clear direction on where to get started. In this presentation I share the system we developed to give us a way to evaluate, score, and prioritize our customer's UX feedback. Want the handout with presenter notes? Grab the PDF at: https://drive.google.com/open?id=0B20rch3MynZ1THA2NE9wTUYycjQ Originally presented at Product Camp Austin (August 19, 2017).

1. www.bestppt.com
How We Went from
Overwhelm to Prioritized
Backlog
BURIED IN UX
FEEDBACK?

2. @winnermint
ABOUT YOUR SPEAKER
2
I’m Stephanie Schuhmacher,
a UX Designer and breakfast taco enthusiast 🌮
BONJOUR!

3. USER FEEDBACK
YOU

4. OUR NEED
AN OBJECTIVE SCORE TO PRIORITIZE
SUBJECTIVE FEEDBACK

6. @winnermint
KNOW YOUR RED ROUTES
6
LET’S GET STARTED
The first two variables in the score are based
on your product’s red routes. These are
frequently used or most critical activities that
your users do.
A simple rule of thumb: red routes are the
key things that users bought your product to
be able to do.

7. @winnermint
KNOW YOUR RED ROUTES
7
How often do users need this function?
FREQUENCY OF USE
How many users need (or have access to) this
function?
NUMBER OF USERS AFFECTED
ONE CONCEPT, TWO
VARIABLES
To determine a red route you need to know:

8. @winnermint
ASSIGNING POINTS TO THE LEVELS
8
Almost universally all users of the product see or use this.
ALL USERS
3
This isn’t something seen or used frequently by all of the
users every day, but it’s often enough that it’s not
uncommon..
SOME USERS
2
Users very infrequently see or do this, or it’s limited to a very
small role (e.g., account administrators).
A FEW USERS
1
NUMBER OF USERS
AFFECTED

9. @winnermint
ASSIGNING POINTS TO THE LEVELS
9
0 We added this one for activities that we have little to no user
data on. These are typically for new features.
UNKNOWN
2 This isn’t something seen or used every day, but it’s often
enough that it’s not uncommon..
SOME OF THE TIME
This is the “People buy our product specifically so they can do
this” activity
ALL OF THE TIME
3
1 A few users do this every once and a while, some users may
never see this, or users do this one or twice over the lifespan
of their use of the product.
INFREQUENTLY
FREQUENCY OF USE

10. @winnermint
RATING ISSUE SEVERITY
10
HOW BAD IS “BAD”?
Next, you’ll next need to choose on how you
want to describe and rank how severe a user
experience problem is.
The scale should range from “This is kind of
annoying” to “OMG, everything is on fire.” You
decide how many steps go in between them.

11. @winnermint
ASSIGNING POINTS TO THE LEVELS
11
1 Minor but irritating problem, or nice-to-have. Cosmetic or
consistency issues. Fix this if you’ve got the time.
LOW
3 It’s difficult for users to get work done and there’s no work-
around to the problem. It’s so bad that the product to be
regarded as “pitiful.”
SERIOUS
It’s easy for a user to accidentally, permanently lose data. The
issue is so bad it prevents people from getting work done. FIX
THIS BEFORE YOU EVEN THINK ABOUT RELEASING.
CRITICAL
4
2 The user can still accomplish tasks, but may need to figure
out a work-around. Slows down the users ability to use the
product.
MEDIUM
SEVERITY OF USER
EXPERIENCE ISSUE

12. @winnermint
MULTIPLY POINTS TO GET YOUR FEEDBACK SCORE
12
USERS AFFECTED
3
FREQUENCY
3
SEVERITY
4
FEEBACK SCORE
36XX =

13. @winnermint
FEEDBACK SCORE AND STORY POINTS ARE BFF’S
13
FEEBACK SCORE
27
STORY POINTS
3

14. @winnermint
PAIR FEEDBACK SCORE AND STORY POINTS TO REVEAL THE MAGIC
14
LOW IMPACT
HIGH EFFORT
LOW IMPACT
LOW EFFORT
HIGH IMPACT
LOW EFFORT
HIGH IMPACT
HIGH EFFORT
IMPACT
EFFORT

15. @winnermint
PAIR FEEDBACK SCORE AND STORY POINTS TO REVEAL THE MAGIC
15
LOW IMPACT
HIGH EFFORT
LOW IMPACT
LOW EFFORT
HIGH IMPACT
LOW EFFORT
HIGH IMPACT
HIGH EFFORT
IMPACT
EFFORT
🙌

16. @winnermint
EXAMPLE ONE
16
9
I was surprised the
hangup button
wasn’t red like it is
on my smartphone.
All users
USERS AFFECTED
3
Every time they make a call (a lot)
FREQUENCY
3
No difficulty finding the button, they
just want a different color.
SEVERITY
1
FEEDBACK
SCORE
9
STORY
POINTS
1

17. @winnermint
EXAMPLE TWO
17
9
All users
USERS AFFECTED
3
Each time they open the app
FREQUENCY
3
It just a nice touch to make the app
feel less formal.
SEVERITY
1
FEEDBACK
SCORE
9
STORY
POINTS
72
“Welcome back,
Stephanie!” is so
much more inviting.

18. @winnermint
THANK YOU AND QUESTIONS
18
THANK YOU!
ENJOY THE REST OF PRODUCT CAMP!
@winnermint
winnermintSTEPHANIE SCHUHMACHER
a.k.a., Winnermint
stephanie@winnermint.com
www.winnermint.com

19. winnermint.com
REFERENCES & FURTHER READING
19
These are some articles that informed and inspired the system we designed:
Sauro, Jeff. “Rating the Severity of Usability Problems.” Measuringu.com, 30 July 2013, https://measuringu.com/
rating-severity/. Accessed 20 Aug. 2017.
Travis, David. “Red route usability: The key user journeys with your web site.” Userfocus, 25 March 2006, http://
www.userfocus.co.uk/articles/redroutes.html. Accessed 20 Aug. 2017.
Travis, David. “How to prioritise usability problems.” Userfocus, 5 October 2009, http://www.userfocus.co.uk/
articles/prioritise.html. Accessed 20 Aug. 2017.
Rosenberg, Carlos. “Turning Usability Testing Data into Action without Going Insane.” UX Planet, 26 Jun. year
unknown, “https://uxplanet.org/turning-usability-testing-data-into-action-without-going-insane-9f364a683076.
Accessed 20 Aug 2017.