Presentation of the MobileHCI 2014 paper: Kjeldskov J. and Skov M.B. (2014) Was it Worth the Hassle? Ten Years of Mobile HCI Research Discussions on Lab and Field Evaluations. Proceedings of Mobile HCI 2014. Toronto, Canada, September 23-26. ACM Press. pp. 43-52. DOI=10.1145/2628363.2628398 Presented in Toronto, Canada, 24 September 2014

1. Was it worth the hassle?
Ten years of Mobile HCI research
discussions on lab and field evaluations
Jesper Kjeldskov & Mikael B. Skov
Center for Socio+Interactive Design / Department of Computer Science
Aalborg University, Denmark

2. 2

3. Background
• At the first mobile HCI workshop (1998) Peter Johnson
called for research on how to evaluate mobile systems
• In 2003 we reported that 71% of mobile HCI evaluations
were done in the lab
• In 2004 we published a paper about evaluation in the
field, asking “is it worth the hassle”?
• The “Hassle” paper caused a bit of a stir...
3

4. The “Hassle” paper
“Quite surprisingly, our study shows that compared to setting up
a realistic laboratory study, evaluators achieved little added value
when taking a usability evaluation of a context-aware mobile
device into the field”
“Expensive time in the field should perhaps not be spent on
usability evaluation”
“Field studies may instead be more suitable for obtaining insight
needed to get the system right in the first place”
4

5. Fast forward 10 years
• We did a literature review of the publications citing the
“is it worth the hassle” paper
• 191 citations (Feb. 2014) - obtained 165 electronically
• 12 were in a language we couldn’t read
• 7 cited for strange reasons (e.g. “mobiles have small displays”)
• 4 did not refer to the paper in the body text
• The remaining 142 were carefully read and classified
5

6. Looking back, and then ahead
Using lab or field
Comparing lab and field
Discussing lab and field
Beyond usability and usability evaluations
Beyond non-wild field studies
Beyond snap-shots of use
6

7. Using lab or field (62)
Lab (23)
• Increasing realism in controlled environments
• From hi-fidelity setups to simple simulations
Field (39)
• Field experiments (17), ethnography (11), and surveys (11)
• From in-depth longitudinal studies to testing outside
Great variation and creativity in developing new techniques
7

8. Comparing field and lab (16)
• Some studies are similar to the Hassle study, but differ
a lot in their experimental setup - making it difficult
to talk about actual replication
• Great difference in what constitutes “lab” and “field”
• Lab: from seated at a table to hi-fidelity simulations
• Field: from simulations to actual real world settings
8

9. “We created two labs at the school, one for field testing
sessions and one for laboratory testing sessions”
“If we want to compare the two settings, the field
evaluation will have to be less realistic”
9

10. Discussing field and lab (64)
• Going beyond practical use and direct comparisons
• Typically quite strong and detailed discussions of
methodology (e.g. ecological validity, and replicability)
• Arguments for and against both lab and field
• Some perhaps more “religious” than objective
10

11. Status in 2014
• A large body of research on lab and field evaluation
• New ways of simulating context and experimenting in-situ
• Have responded quite well to the challenge of building a
body of knowledge and experience with both lab and field
studies outside the common range of usability
• Is it worth the hassle? No definite answer has yet been
found. Perhaps not the right question?
• From if and why to when and how?
11

12. Looking ahead
There are a number of points we would like to put
forward for your consideration
12

13. Move beyond usability and usability evaluations
• The challenge for mobile HCI research is no longer usability
• We are pretty good at designing interfaces for mobiles
• Today’s challenge is about designing services, devices and
interactions that fit well into people’s complex lives, and fit
with all those other technologies out there
• Designing for orchestration or digital ecosystems
13

14. Studies in the field should not be half-tame
• The real value of the fields is that it is real and messy, and
not an amputated version of reality
• A field study that doesn’t take the researcher into an
uncontrolled real world is perhaps not a good one
• When going out of the lab, we ought to make it across the
parking lot all the way into the wild
• Embracing the wilderness rather than trying to tame it
14

15. Do longitudinal studies rather than snapshots
• The legacy from usability engineering has made us
accustomed to “snapshots of use”
• Also true for many field studies (e.g. field experiments)
• Going beyond usability, and embracing the wild, we should
also consider stretching the duration of our studies
• Perhaps even sacrifice direct researcher involvement?
15

16. To-do before 2024
(preferably sooner)
• Move beyond usability and usability evaluations
• Embrace the wilderness - don’t be half-tame
• Do longitudinal studies rather than snapshots of use
16