Katie Phillips talk on using Diary Studies for customer research including how she used Slack for a study at Australia Post. From Product Anonymous March 2017 event.
Learning from your customers - A diary study with Slack
1. End-to-end validation and optimisation
via Diary Study
Katie Phillips
User Experience Researcher -
Australia Post
2. Agenda
2
1. Project brief
2. Methodologies
3. Recruitment & planning
4. Why Slack?
5. Key Insights & implementation
6. Can you run it?
7. Useful links and resources
3. 3
Project brief
• New application for consumers & businesses to pay for & print Australia Post
labels online
• Application live since October
• New features being deployed frequently
• “International” feature soft-released November
4. Choosing a
research
methodology
4
Key factors
• What is your objective?
• Validating the end-to-end journey of a customer
sending internationally through this application & using
insights to enhance it
• Where are you fidelity-wise?
• In production
• How much time can you commit?
• Endless (aka 2.5 iterations)
• What’s your budget?
• Enough for professional recruitment, incentives, and a
dedicated researcher
Create
order
Get to
MyPost
Business
page
Review &
confirm
Pay Print
labels
Prepare
package
Lodge
parcel
5. Moderated user testing
Pros
• Great insights into usability
• Defects obvious
• Can be observed/live streamed
• Remote options
• Can watch what customers are actually doing,
not just what they say they do
• Any fidelity
6. Cons
• Moderator required (can do self-moderated as well)
• Logistics
Time requirements
• Try for at least 5 participants
• Run time: 45min-1hr each
• Analysis time: 1-3hr per participant
Moderated user testing
7. Diary study
Pros
• Great insight into end-to-end process
• Real world factors
• Lots of raw quotes and photos
• Runs in background
• Helps team understand value of product
8. Cons
• Dependent on what customers
say they do
• Drop outs
• Technology
• Requires very clear instructions
• Prototype must be functional
Time requirements
• Try for at least 8 participants
• Run time: 1-2 weeks
• Analysis time: 2-4 hours per participant
Diary study
10. Diary study
planning &
recruitment
• Recruited 12 small business
participants for a 3 week
study
Tips
• Be tight on recruitment
• Be clear on what they need to do
to receive incentive
• Incentives at the end (or segment)
• Find an easy platform
Recruiting specs
• Demographics (age, gender, location)
• Behavioural (online habits, social, travel)
• Exclusions (colleagues, language
restrictions)
11. Why Slack?
• Free!
• Great desktop and mobile apps
• Private channels
• Easily invite others from your
team to observe
• Keep the team open for as long as
you need
• Group channels (help, participant
chat)
• Bots (Anonymous feedback, Polly
surveys, To Do, SlackBot)
• Easy to moderate & set up
notification preferences
12. Setting up Slack
• Write clear instructions for participants BEFORE the start date
• Set up a private channel for each participant
• Set up a group help channel
• Be prepared to moderate at all hours
• Provide an alternative
16. 16
“If you could let it accept .3kg and
auto add the zero that would be
great - adding the leading zero
stuffs me up each time.”
Insights & resolutions
Resolutions
• Development team allocated 1/2 - 1
iteration to incorporate critical feedback
during study
• Optimisation backlog was defined and
prioritised based on insights
• Instructions page re-designed
Insights
• Confirmation of value
• Form fixes needed (12 usability issues)
• No one reads instructions
• It’s easy to learn
“All of my friends who run online
businesses will be jumping for joy”
17. 17
Insights & resolutions
Resolutions
• Non-interface issues raised with
international product team
• Non-application issues raised with segment
team
• Conducted additional user testing around
language
• Input into website IA
• Insights remained with our team for future
features and projects
Insights
• Plastic pockets aren’t easy to get or use!
• Trouble navigating to application
• Awareness issues
• Language confusion
“Oh, there it is! It's like hidden! That's way
too hidden”
18. Can you run
it?
18
Key factors
• Is it the right method for your product?
• $$ for incentives
• $150-$250 per participant for a 2 week diary-study
• Plus $100-$200 per participant for recruitment
services
• your time if you choose to recruit on your own
• Time
• time requirements as mentioned previously
• development time to incorporate feedback and
changes
19. 19
The real question:
Can you afford NOT to run something?
“If I wasn't participating in this
there is no way in heck I would
even know this is available”
“I'm just assuming I would take this
into my local post office and have
them scan and hopefully they don't tell
me that I can't use them.”
“ It does not like the price I have
entered for the shoes to the US yet it is
the price she paid for them. I have tried
varying the amount but it still remains
highlighted in red no matter what I do”
20. Nielsen Norman Group on choosing methodologies
ethn.io's Remote Research tools guide
Frog’s Design Research bookshelf
IDEO’s Method Cards
20
Useful
resources