A field study was proposed to evaluate Yahoo! Go, a mobile search application. The study involved 9 participants using the app for 1 month. Data collection methods included daily SMS surveys, photos from search contexts, voice diaries, and interviews. This approach provided insights into usage behaviors, content needs for different use cases, and opportunities to improve relevance and the mobile experience.

1. Making the Leap from Web to Mobile
Best Practices in Mobile User Experience Research
Amy Buckner, AnswerLab
Kris Mihalic, Yahoo!
UPA 2009, June 12, Portland

2. Agenda
Topic Minutes
1. Introduction 5 minutes
2. State of Mobile 10 minutes
3. Group Exercise: Mobile Device Task & Discussion 5 minutes
4. Mobile User Research Methods 20 minutes
5. Team Exercise: Develop Research Solution 15 minutes
6. Case Study 15 minutes
7. Mobile Usability Research Challenges & Tips 15 minutes
8. Q&A 5 minutes
3
CONFIDENTIAL

3. Introductions
Amy Buckner
Managing Partner & Co-Founder of AnswerLab
Over 10 years experience leading user research programs for Fortune 500 companies
Expertise in quantitative and qualitative methods, across web and mobile devices
Previous West Coast Director of Professional Services at Vividence (now Keynote) and
Marketing Manager at SmartPlanet
Believes a glass of wine and a beef jerky stick count as dinner
Kris Mihalic, PhD
Head of Mobile User Experience and Design Research, Yahoo!
Seasoned user experience and design researcher for mobile and cross-channel products
and services
Expertise in defining and executing UX and design research strategy in cross-functional
teams
Experience teaching interface design methods, usability engineering, and
telecommunication technologies
Lives on coffee
4
CONFIDENTIAL

4. State of Mobile
5
CONFIDENTIAL

5. There are 4.1 Billion mobile phone subscribers worldwide
Only 2% of US mobile phones are iPhones (~5 Million)
Social networking on mobile phones is growing 196% Y/Y
There are 1.4x more mobile IM users than mobile business
email users in the US
6
CONFIDENTIAL

6. Mobile Market Summary
Globally, there are 800 million cars, 850 million PCs, 1.3 billion fixed landline
phones, 1.4 billion credit cards, 1.5 billion TV sets – and 2.7 billion mobile phones –
in use. (Experian 2008)
eMarketer projects that
worldwide spending on
mobile advertising will
reach a total of $19.1
billion in 2012 – up from
4.6 billion in 2008.
Source: Experian 2008, eMarketer 2008 7
CONFIDENTIAL

7. U.S. Mobile Web Users Growing Rapidly
Source: m:metrics/comScore, 2009 8
CONFIDENTIAL

8. Mobile Audience Size (# of users)
In the U.S., the size of the Mobile Audience is 250 Million
60.00%
52%
50.00%
40.00%
31% 30%
30.00%
20.00%
10.00%
10%
0.00%
SMS Users Mobile Media Users 3G Smartphone
9
Source: m:metrics/comScore, 2009
CONFIDENTIAL

9. U.S. Mobile Internet Usage
Mobile Internet
users are
heavy users
60% access the
Internet from their
phone at least once/
day
48% access the
Internet more than
once per day
Source: TSM|TargetProfile 2007 10
CONFIDENTIAL

10. Device Traffic
Mobile phones dominate mobile
Web traffic with 66%
market share (US).
BlackBerry and Smartphones
(Windows Mobile, Palm, etc.) are
heavy users of mobile Internet –
combined 26% of traffic
iPhone/iPod Touch has shown
tremendous growth, capturing
8% of the mobile Internet market
in the US
Source: m:metrics/comScore, 2009 11
CONFIDENTIAL

11. Widgets Improve Mobile Internet Experience
Source: ‘Widgets Improve the Convenience of the Mobile Internet,’ Forrester, 2008
12
CONFIDENTIAL

12. Group Exercise
13
CONFIDENTIAL

13. Take out your cell phone and
search for times that ‘Star Trek’ is
playing in your neighborhood.
Raise your hand when you have
finished the task.
14
CONFIDENTIAL

14. Discussion Points
How long did it take?
How did you do it? (web, app, SMS?)
What problems, if any, did you have?
15
CONFIDENTIAL

15. Mobile User Research Methods
16
CONFIDENTIAL

16. Overview of Research Methods
Quick and Dirty feedback
Lab Studies
Field Studies
Online Surveys
Basic Survey
Survey with Behavioral Tracking (Keynote on iPhone only)
Quantitative Behavioral Analysis
17
CONFIDENTIAL

17. Quick and Dirty – What are obvious problems?
Advantages
Informal setting with Lo-Fi prototypes
Fast iteration cycles
Brain-damage check
Finds critical usability issues in shortest time
Challenges
Examples
Insider/stakeholder view
Cross-functional
team feedback
Anecdotal, unstructured results
Internal alpha/beta
Small sample size
Difficult to convey real usage scenarios
18
CONFIDENTIAL

18. Lab Studies – Can they use it?
Advantages
Allows for probing by moderator
Can record interactions
Best for prototype testing
Allows for real-time viewing of interaction – team
engagement
Challenges
Examples
Artificial environment
Usability study
Moderator bias; participant ‘pleasing’
Participatory
Small sample size
design sessions
Difficult to test all hardware / network scenarios
19
CONFIDENTIAL

19. Field Studies – What is the context of usage?
Advantages
Assess usage under real conditions
Candid user feedback
Delivers unknown use-cases
Challenges
Examples
Small sample size
Diary study
Difficult to observe and probe
Community study
Participant engagement difficult
Requires robust product
20
CONFIDENTIAL

20. Online Surveys – What do they think of it?
Advantages
Can deliver statistically valid results
Provides qualitative and/or quantitative data
Geographical diversity (remote)
Broad representation of devices
Challenges
Examples
Difficult to observe and probe
Desktop surveys
Out-of-context interaction, esp. with desktop survey
On-device survey
Technical constraints
Survey with
behavioral tracking
21
CONFIDENTIAL

21. Behavioral Analysis – How do they use it?
Advantages
Reflects what people do, not say
Statistically valid results
Shows all behaviors, rather than those confined to a
single task
Challenges
Examples
Probing / deep-dive difficult
Log data mining
Context and intent unknown
Attitudes and perceptions unknown
22
CONFIDENTIAL

22. Types of Prototypes
Paper prototypes
Mocks
Paper and pencil
Interactive prototypes
Apps
Flash
Native, e.g. iPhone
Browser-based
Simple HTML prototype
High-fidelity prototype (with session variables)
23
CONFIDENTIAL

23. Team Exercise & Case Study
24
CONFIDENTIAL

24. Yahoo! Go
Mobile Application
Search
News
Sports
Weather
Other Topics
25
CONFIDENTIAL

25. Y! oneSearch integrated into Y! Go
26
CONFIDENTIAL

26. Business Issue
Drive mobile search uptake – Improved user experience can help
accelerate growth of the service
Improve relevancy – Understanding search intent can contribute
to delivering more relevant results
Product differentiation – New approach to search results page
can deliver higher value to customers
27
CONFIDENTIAL

27. Research Objectives
Understand mobile search behavior in users’ daily lives
Identify content users seek when conducting mobile searches
Assess the context surrounding mobile searches
Evaluate effectiveness of new product
Identify opportunities to improve the user experience
28
CONFIDENTIAL

28. 1. Form into teams of 4.
2. Develop a research plan for
Yahoo! Go.
3. You have 10 minutes.
29
CONFIDENTIAL

29. Team Activity Research Objectives
Develop a research plan for Yahoo!
Understand mobile search behavior in
users’ daily lives
Go
Identify content users seek when
Include: conducting mobile searches
• Recommended method(s)
• Number of participants
Assess the context surrounding mobile
searches
• Profile of participants
Evaluate effectiveness of new product
Identify opportunities to improve the
user experience
30
CONFIDENTIAL

30. Discussion Points
What methods do you recommend & why?
What are your anticipated concerns?
Any potential limitations?
31
CONFIDENTIAL

31. Our Solution: One-Month Field Study (1)
Voicemails &
Daily Mobile Survey Digital Photos
Pocket Card
9 users
Daily SMS mini-
Users sent
survey photos of
Daily voicemails
themselves or
Linked to phone
Laminated surroundings in
number for
pocket card with the context of
voicemail
key questions using oneSearch
32
CONFIDENTIAL

32. ‘Pocket Card’ Questionnaire
33
CONFIDENTIAL

33. Sample Voicemail Diary Messages -- Winchelle
Friday 7:39pm
I used oneSearch today, was looking for a hospital – directions and address to it. I was getting off work
and sitting in my car, trying to find directions. It was about 4:30. I liked that it came up with the website
and with information about that. And the directions on how to go.
I didn’t like that it could not find my work address – which is saved on my Yahoo! directions
(recently gone) saved places.
I think using the actual internet through the PC will help me better because it knows where the address
is. I’m not sure why the address is not coming up in my oneSearch and that sucked.
Saturday 8:06pm
I didn’t use oneSearch today. I looked for info, looked for a store and I was at home. I wanted to know
if the store was open STORE HOURS (9:45am). I used the internet with my phone and used Google to
search. It was not very helpful because I wanted to see what time they opened and I couldn’t find it.
I’ve used oneSearch in the past and I guess the results were the same but I guess it’d be nice to go to
the site for that particular store.
Monday 10:48pm
I used oneSearch to find Expedia, was at work on my lunch break. It was about 12:20. I liked that I can
find it quickly. I guess the website didn’t support my mobile phone though, so I didn’t like that.
Would rather book a flight/hotel at home because its supported.
34
CONFIDENTIAL

34. Our Solution: One-Month Field Study (2)
Pre-Interview Mid-Check Interview Wrap-up Interview
An initial 45-
20-minute check-
Final in-person
minute interview in phone interview of 75
interview minutes
Verified technical
capabilities of
Initial feedback
Clarified
phone voicemail reports
Q&A
Explained
Overall
research impressions of
program the product from
the month-long
usage 35
CONFIDENTIAL

35. Screening Criteria: 9 Experienced Mobile Users
Have an unlimited data plan for
Have a primary mobile device
WAP services through their capable of downloading and
mobile carrier using Y! Go
Use their mobile for text
Live or work within San Francisco
messages (SMS) daily and use
mobile more than three times per
week for activities that require the
Mix of gender, age (23-46+), and
transmission of online data typical method of commuting to
work (4 users primarily drive, 5
Have Cingular, Sprint, T-Mobile, primarily take mass transit)
or Verizon as their mobile carrier
36
CONFIDENTIAL

36. Sliding Scale for Participant Incentives
$100 for the initial in-person interview
$50 for the mid-way phone interview
$150 for the final in-person interview
$3 each day a report is given
37
CONFIDENTIAL

37. Why did we choose this method?
Natural environment key to context
All research activities on the mobile device
Mix of methods
Engage users in program
Breadth of insights – context, usability, impressions
38
CONFIDENTIAL

38. What did we learn? (1)
Perceived value increased with usage
Usage typically driven by lack of computer availability
However, certain scenarios drove phone choice over
computer:
Social setting
Privacy
Convenience
39
CONFIDENTIAL

39. What did we learn? (2)
Mental model based on
computer usage
Speed and relevance of search
results highest area of frustration
Lack of understanding that
results were customized by
widget
40
CONFIDENTIAL

40. Daily SMS Survey Results
*Website not adapted for mobile device 41
CONFIDENTIAL

41. What did we learn? (3)
Use Case Primary Content Wanted
Shopping Business’ number, location, directions, and hours of operation
Entertaining Schedules/Venues of concerts, sport games, movies, and TV shows
Researching Content from news/articles
Dining Restaurants’ number, location, directions, and hours of operation
Traveling Schedules and numbers of transportation (public transit, taxis, etc)
Media Seeking Photos/Ring tones
42
CONFIDENTIAL

42. Summary of Use Case Frequency
Note: The 166 searches were conducted by 9 users
43
CONFIDENTIAL

43. Shopping
Business’ number, location, directions,
and hours of operation
“While driving home from work, I did a search
for Fry’s electronics on the local search and it
came up with all the different Fry’s in the area,
their numbers, as well as directions and all
that. The results were all helpful and fast in
helping me find the one closest.”
“Since we were already out shopping, I wanted
to find where the closest Citibank was to where
we were.”
Pricing/Product comparisons
“I was at Costco and comparing TV prices. I
wanted to know how much these same TV’s
would be at the Circuit City down the street.”
44
CONFIDENTIAL

44. Dining
A restaurant’s number, location,
directions, and hours of operation
“I just woke up and didn’t want to boot my
computer up but I wanted to find out what
time this breakfast place opened so I
could know how soon I could get there.”
“I was in my car and wanted their number
so I could call and place an order to pick it
up on my way home from work.”
Ideas of places to eat
“I was with my wife in the Haight and we
wanted to find nearby restaurants so we
could decide what we wanted to eat that
was within walking distance.”
Reservations & Reviews
“I was in the South Bay with some friends
“I was hoping to make reservations straight
and we wanted to find a good sushi place from the phone, or at least call them to place
nearby.” them.”
“I was out with some friends and curious
about whether the place nearby was any
good.”
45
CONFIDENTIAL

45. Traveling
Transit schedules and numbers
“I wanted to find out when the next MUNI but
the MUNI’s site wouldn’t work on my phone.”
“I wanted the number to call a taxi. I was
outside and tired of waiting for the bus.”
Tourist activities
“I wanted to see what kind of ferry tours we
could do before looking for another museum
that would be fun for my daughter.”
“I was looking for fun things to do while at
Hermosa beach.”
Maps and directions
“I was trying to get a map of the Mt. Shasta area
and I wanted to plan which route to take. It
would be nice if the driving directions had live
updates on traffic and road conditions.”
Flight status / check-in
“I was in the taxi to the airport and wanted to
check my flight’s status.” 46
CONFIDENTIAL

46. Business Implications
Drive mobile search uptake – Distribution through partnerships,
marketing campaigns, opening to third party services and developers
Improve relevancy – Focused on improving results for specific use cases
(e.g. local businesses, flights, etc.)
Product differentiation – Federated search results, integrated user
experience across multiple services (e.g. search results + maps)
48
CONFIDENTIAL

47. Mobile Usability Research Challenges & Tips
49
CONFIDENTIAL

48. Variety of Mobile Devices
Challenges Tips
Difficult to know and understand user
Design team should have a variety
experience across all devices of devices simulators / devices on
hand for testing
Devices vary by model, browser type,
carrier, and input type
Prototype designs should be tested
across as many devices as possible
Moderator may be unable to help
users through usability tasks –
Limit recruiting to only devices that
resetting prototypes, navigating – if have been tested
interface is not well-known
Have a back-up plan with either a
basic device or paper prototypes
50
CONFIDENTIAL

49. Mobile Devices are Small and . . . Well, Very Mobile
Challenges Tips
Create a hot zone on the table, encouraging
Users move the device around while
users to keep the device within a narrow
interacting and explaining, often moving frame
video display out of focus
Consider light source and potential
Reflectors on screens and smudges can reflection on the phone; continually adjust
make video all reflection mid testing; turn-off lights
Small device screens make it difficult to
Use remote-controlled video camera (with
see what users are doing technician in back room) or have a second
technician available in interview room
Large fingers and long fingernails can
Project the mobile screen on a larger
cause unintended device responses monitor in interview room and back room
Creatively screen out participants whose
finger size may distract from findings (if non-
touch screen)
51
CONFIDENTIAL

50. Varied Locations and Use Cases
Challenges Tips
Mobile device usage occurs in a
Create opportunities for feedback in
variety of places context of usage
59% of Americans check email while in
the bathroom*
Conduct field studies on beta
products
Difficult to recreate the true
experience in a lab setting
MOBILE DEVICE ACCESS
67.4%
55.2%
35.9%
15.8% 6.4%
Source: Nielsen Mobile 2008 52
CONFIDENTIAL

51. Varied Carrier Network Coverage
Challenges Tips
Network coverage is inconsistent;
Visit facility with colleagues who
difficult to predict accessibility during have various carriers to confirm
lab testing network availability in lab
Difficult to validate that out-of-town
From out-of-town labs, request:
lab truly has adequate coverage
Reference clients from last mobile
studies
List of carriers confirmed to have
consistent coverage
At a minimum, check carrier
coverage in various cities online
53
CONFIDENTIAL

52. Calling / Data Plans Vary Widely
Challenges Tips
Participants may be charged for
Screen during recruiting for:
mobile web access or sample texts
Unlimited data plans
sent during testing
Willingness to send / receive text
messages during study
Participants may change data plan
between recruiting interview and date
Rescreen again upon arrival for
of study study
54
CONFIDENTIAL

53. Varied User Experience
Challenges Tips
Most mobile device owners only
Pre-test participants to ensure
utilize a small percentage of total appropriate device experience
device capabilities
Mobile address typing
43% of mobile subscribers do not use
Send SMS to recruiter
text messaging on a regular basis*
Spend 2-5 minutes explaining how
Users may have no experience typing prototype phone works
in a web address to access a site or
with sending SMS messages (which
may be critical to your study)
If using a prototype phone for
downloadable app, users may not be
comfortable with it
55
* Source: Nielsen 2008 ‘The Short Code Marketing Opportunity’
CONFIDENTIAL

54. Other Helpful Tips
Tell participants to bring recharger
Simplify language (e.g., SMS vs. text message)
Have a prototype strategy
Back-end SMS text simulation
Easy URL for access
Back-up paper prototypes
Index page for easy access to alternative flows
Have plan for interruption from phone call or text (resetting session variables)
Plan for getting screenshots for your report
iPhone: Press hold button and home button at same time (saves into photo gallery)
Use documented comps
56
CONFIDENTIAL

55. Q&A
57
CONFIDENTIAL

56. Thank You.
Amy Buckner
Managing Partner & Co-Founder, AnswerLab
amy@answerlab.com
Kris Mihalic
Head of Mobile User Experience and Design
Research, Yahoo!
kmihalic@yahoo-inc.com
Twitter: suikris
58
CONFIDENTIAL