Laura Bowden's presentation on testing with mobile devices from UXPA Boston 2013.

1. Usability Testing on Mobile
Devices
No More Excuses!
Laura Bowden & Cindy McCracken
Fidelity Investments
Demo
Log in here while you wait!
https://febdesign.webex.com
UXPA Mobile Device Testing
Password: 123456

2. Barriers to Mobile Device Testing
• Geography
• Observation limitations
• Perceived Cost
• Mixed testing & observation platforms
2

3. The Scenario:
• Testing on Smartphone or iPad
– Participants are in the lab
– Observers are all over the place
– You want everyone to view the mobile device
screen
– You want everyone to view the participants’ faces
– You want to record whole shebang to video
3

4. Setup Options
4
Smartphone setup 1: slides 6 - 30
Smartphone setup 2: slides 31 - 44
iPad setup: slides 45 - 54

5. Litmus test to determine Setup 1 or 2
Setup 1 is required if:
• You’re not using an iPhone
• You’re testing behind a “serious” firewall.
• (But you can also use this setup with an iPhone if you want to, which is what
we’re doing today.)
Setup 2 can be used only when:
• You’re testing on an iPhone or iPad (although
there may be similar technology with Android OS)
• You’re not behind a “serious” firewall
5

6. Pros & Cons of Smartphone Setup 1 versus 2
6
Setup 1 Setup 2
Pros Pros
Only dependent on 1 network Look mom, no wires!
Advanced class: Finger dots
with TouchPose
Cons Cons
Wires Jailbreak required for
TouchPose
Hands/fingers can get in the
way of the screen
Can only test iPhones with
Reflector App, not other
smartphones.

7. Smartphone Setup 1
7

8. Smartphone Setup 1
Materials
• 2 laptops, 1 with built in webcam (mixing Macs and PCs is ok)
• 1 tiny camera (we like IPEVO for $69.99)
• 1 smartphone holder “sled” (we had one custom made at a plastics company for
$150 using a photo from the web, here’s a cooler one: http://www.mrtappy.com/index.html)
• Webex w/ VOIP option (no, not GoToMeeting. Webex.)
• Screen capture software (we like Camtasia for Mac, Morae for PC)
• Any smartphone
• An app, website, or prototype
• Patience, fortitude and the spirit of adventure! (You know
how technology breaks down during lab testing sometimes? Multiply that by a
factor of precisely 5, representing the number of additional potential points of
technology failure compared to a regular lab test on a computer.)
8

9. Schematic – Smartphone Setup 1
Phone Image
Capture
Laptop
(w/ built-in webcam)
Smart
phone
Tiny
Camera
cable
sled
Video
Recording
Laptop
(w/ lots of free space)
Running:
-Tiny Camera app
-WebEx (sharing)
-VOIP for audio
Running:
-WebEx (attending)
-Camtasia
Observers
Laptop
(attending
WebEx)
Phone
or VOIP
Lab
9

10. • Are one of your 2 laptops more powerful than the other? Let that one do
the screen recording, and the other one host the phone image.
• Do you know the video format your business partners can use on their
machines? For example, if you’re recording with Camtasia for Mac, you
can’t convert to .wmv without lengthy file conversion resulting in terrible
quality. Pick the right recording software for the job.
• Are all your observers able to log into WebEx? (Some company’s firewalls
get crabby about it, requiring special IT interventions.)
• Are you able to permanently turn off your screensavers on BOTH
machines? Neither of them will have any keyboard action for the whole
sessions, and will eventually go to sleep. 60 minutes of screensaver video
is no fun. You can always download the app called “Caffeine” which will do
that for you on a mac. Its free.
Smartphone Setup 1
Step 1 – Plan for the end result
10

11. Smartphone Setup 1
Helpful Tips Learned from(horrible)Experience:
Tip #1
• To avoid a dozen hours of file transfers, always
record the video on the same machine on which it
will be processed and edited..
Tip #2
• Don’t even think about attempting to copy raw
recordings onto a network “between sessions.”
Make sure you have enough space on your hard
drive for the total amount of video you will end up
with, plus extra, by a lot.
11

12. Follow this order
a. (Put the screen recording computer aside for a minute.)
b. Launch the tiny cam’s driver application on the Image Sharing Computer.
c. Plug tiny cam into USB port of Image Sharing Computer and turn it on.
d. Attach the tiny cam to the sled
e. Attach the smartphone to the sled. (Using a rubber band? No shame!)
f. Fiddle with the cameras application settings so that your image is right-
side up and focused
Smartphone Setup 1
Step 2 – Start plugging things into each other
12

13. 13
b) Launch the tiny cam’s driver application on the Image Sharing Computer.
c) Plug tiny cam into USB port of Image Sharing Computer and turn it on.

14. 14
d) Attach the tiny cam to the sled.

15. 15
e) Attach the smartphone to the sled.
Even with just a rubber band, its pretty
sturdy and can accommodate any
orientation.

16. 16
f) Fiddle with the tiny cam’s application settings so that your image is right-
side up and focused.

17. 17
a) On the image sharing laptop, start your WebEx meeting and phone
call…VOIP is recommended.

18. a) On the phone image sharing laptop, launch a browser & start
your WebEx meeting and phone call.
b) On WebEx, turn on the video cam option next to your name
(this will capture the participant’s face). Now you should see
2 windows: the participant’s face (small window) and the
smartphone screen (larger window). Share your screen.
c) Drag images where you want them. Optional: Minimize the
smaller window with the face image so the participant
doesn’t have to watch themselves. WebEx: Participant menu > Assign
Privileges > Participants Tab > Uncheck Participant List checkbox > Assign button
d) Use static electricity to stick a piece of printer paper onto the
screen of the participant laptop*
Smartphone Setup 1
Step 3 – Fire up WebEx
18

19. 19
a) Put recording machine aside for a second…
b) In WebEx, turn on the video cam option next to your name (this will
capture the participant’s face). Now you should see 2 windows – the face
(small window) and the phone image (large window). Share.
c) Drag images where you want them.

20. 20
c) Optional: Minimize the smaller window with the face image so the participant doesn’t
have to watch themselves. (You need to stop sharing to get the top menu.) WebEx: Participant
menu > Assign Privileges > Participants Tab > Uncheck Participant List checkbox > Assign button

21. 21
c) The participants’ image, minimized. Click the triangle to open it up again.

22. 22
d) Use static electricity to stick a piece of printer paper onto the screen of the participant
laptop so they aren’t distracted by the image*

23. • In theory you can use an external webcam instead and
leave the laptop closed. But that’s just one more item
to carry, and one more potential break-down point.
• And even if you do use an external webcam, you will
find yourself constantly opening and closing the laptop
to make sure the meeting is running and the
screensaver is turned off and the participants’ face is
lined up in the view… whereas the paper is easily
“peeked behind”, completely ignored by the user, and
causes less distraction overall.
Smartphone Setup 1
*Um…Really?
23

24. Smartphone Setup 1
Helpful Tip Learned from(horrible) Experience
Tip #3
• Sometimes you get the reverse of what you
want in each window. You may have to reverse
the plugging-in and turning-on order to get
the right thing on the right screen. It’s a
mystery why it only happens sometimes. And
believe us, it will revert back to the original
way the next time to try it. Embrace the
mystery.
24

25. Once WebEx is running on the Image Sharing machine….
a) Launch a browser on the video recording machine and join
the meeting. Make sure everything looks how you want it to
look.
b) Launch Camtasia (or your favorite screen capture/editing
software).
c) Make sure Camtasia is capturing VOIP audio
d) Start recording your session
Smartphone Setup 1
Step 4 – Fire up the machine recording screen capture
25

26. 26
a) Launch a browser on the video recording laptop and join the meeting. Arrange images.
b) Launch Camtasia (or any screen capture/editing software) on the video recording laptop.
c) Start recording your session – LIVE TRIAL…

27. 27
View behind the paper
What the participant is really looking at:
Tip: Adjust the size of the
participants face larger, and
consider defining a custom
region in Camtasia for less
extraneous stuff displayed on
the recording. Also, clear your
desktop so the background
isn’t cluttered.
What Camtasia recordsView in front of participant
All Views

28. Do you want to see if it really worked?
28

29. • WebEx allows for 2 video inputs simultaneously, GTM does not. So
if you’re dead-set on displaying the participants’ faces alongside the
smartphone image, you must use WebEx.
• GTM
– Pros: Very easy interface and automatic VOIP with every meeting. We
have found it to be more stable than WebEx mid-meeting and if you
use their recording service it will convert to .mov on a mac and .wmv
on a PC.
– Cons: User experience is less consistent while joining a meeting, and it
only allows for single video input so you can’t display users’ faces
• WebEx
– Pros: Supports 2 simultaneous video inputs and provides VOIP.
– Cons: Your company must purchase the VOIP option. User interface is
overcomplicated. It will record to a proprietary format which needs
conversion before loading into a video editor.
Smartphone Setup 1
Why WebEx and not GTM?
29

30. Phones are…meh. There are pros and cons.
• Pros
– Phones in general are pretty much a sure thing. (knock on wood)
• Cons
– A phone tap is one more thing to carry and one more potential failure in
the chain if you connect it wrong. For observers listening through a
phone, a phone’s placement is rarely between the moderator and the
participant – usually that’s where the computer is, and the phone is off to
one side.
VOIP uses the computer’s mic, which is usually
placed more advantageously by
default, between the moderator and the
Smartphone Setup 1
A word about Phone Audio
30

31. Smartphone Setup 1
Helpful tips learned from(horrible)experience
Tip #4
• Do a test recording to understand how loud
your volume will be.
Tip #5
To record without VOIP or a phone tap
• Put the lab phone on speaker so that the observers can hear
you, and just allow the recording laptop to pick up the voices in
the room via its internal mic. Mac audio we recorded like this
has been better than either the VOIP or the phone taps.
31

32. Smartphone Setup 1
Helpful tips learned from(horrible)experience
Tip #6 – If you’re a hard core minimalist.
If you only have 1 laptop, sled & smartphone
• You could run the webex meeting, VOIP and Camtasia on the
same machine that the smartphone is plugged into. But you
risk grinding the most robust machines down to a
halt, requiring restart between participants.
32

33. Smartphone Setup 2
Ask yourself: Are you testing behind a “serious”
firewall? If so, move along - nuthin’ to see here.
Go to slide 43.
33

34. Smartphone Setup 2
Materials
• 2 laptops, 1 w/ built in webcam (don’t try to be a hero by mixing
Mac and PC for this one)
• Reflector App on computer (download for $15.99)
• Webex w/ VOIP option (no, not GoToMeeting. Webex.)
• Screen capture software (we like Camtasia for Mac, Morae for PC)
• An iPhone (no other kind of smartphone)
• An app, website, or prototype
• Belief in a higher power (While the number of elements for this
setup is decreased, the chaos factor higher, due to the requirement of creating
an additional wireless network which must perform for an audience, flawlessly,
for 8 hours straight 2 days in a row.)
34

35. Schematic – Smartphone Setup 2
Image
Capture
Laptop
(w/ built-in webcam)
Smart
phone
Video
Recording
Laptop
(w/ lots of free space)
Running:
-Reflector App
-WebEx (sharing)
-VOIP for audio
Running:
-WebEx (attending)
-Camtasia
Observers
Laptop
(attending
WebEx)
Phone
or VOIP
Lab
35

36. Setup Comparison
Observers
Laptop
(attending
WebEx)
Phone
or VOIP
36
Setup 1 Setup 2

37. • Litmus test: If you need to go to iPhone settings to select your new
network “by hand”, or if you need to say “create network”on your
computer to get the phone to connect: This option isn’t right for you.
• Is 1 of your 2 laptops more powerful than the other? Let that one do the
screen recording.
• Do you know the video format your business partners can use on their
machines? For example, if you’re recording with Camtasia for Mac, you
can’t convert to .wmv without lengthy file conversion resulting in terrible
quality. Pick the right recording software for the job.
• Are all your observers able to log into WebEx? (Some company’s firewalls
get crabby about it, requiring special IT interventions.)
• Are you able to permanently turn off your screensavers on BOTH
machines? Neither of them will have any keyboard action for the whole
sessions, and will eventually go to sleep. 60 minutes of screensaver video
is no fun.
Smartphone Setup 2 (same as Setup 1)
1 – Plan for the end result
37

38. • (Put the video recording computer aside for a minute.)
a) On your laptop, launch the Reflector app
b) On your iPhone, go to the homescreen.
c) Double-tap the button and scroll all the way to the left, past the music
screen, until you see the volume bar (admit it, you didn’t even know that
was there, right?)
d) To the right of the volume bar, an incomprehensible icon should have
appeared. Tap it.
e) On the screen select “dialog” and “turn mirroring on”
f) Say a little prayer.
g) The iPhone screen should magically appear on your computer screen –
PC or mac.
Smartphone Setup 2
Step 2 – Reflect your iPhone screen
38

39. 39
a) On your laptop, launch the Reflector app

40. 40
b) On your iPhone, go to the homescreen.
c) Double-tap the main button and scroll all the
way to the left, past the music screen, until you
see the volume bar (admit it, you didn’t even
know that volume control was there, right?)
d) To the right of the volume bar, an
incomprehensible icon should have appeared.
Tap it.

41. 41
e) On the screen select the new network
and “turn mirroring on.”
f) Say a little prayer.

42. 42
g) Voila! The iPhone screen should magically appear on your computer screen – PC or Mac.
Little icon on your phone
lets you know that you’re
still connected to AirPlay’s
network.

43. a) On the phone image sharing laptop, launch a browser & start
your WebEx meeting and phone call.
b) On WebEx, turn on the video cam option next to your name
(this will capture the participant’s face). Now you should see
2 windows: the participant’s face (small window) and the
smartphone screen (larger window). Share your screen.
c) Drag images where you want them. Optional: Minimize the
smaller window with the face image so the participant
doesn’t have to watch themselves. WebEx: Participant menu > Assign
Privileges > Participants Tab > Uncheck Participant List checkbox > Assign button
d) Use static electricity to stick a piece of printer paper onto the
screen of the participant laptop*
Smartphone Setup 2
Step 3 – Fire up WebEx
43

44. Once WebEx** is running on the Image Sharing
machine….
a) Launch a browser on the video recording laptop and
join the meeting. Make sure everything looks how
you want it to look.
b) Launch Camtasia (or your favorite screen
capture/editing software) on the video recording
laptop.
c) Start recording your session
Smartphone Setup 2
Step 4 – Fire up the machine doing the screen capture
44

45. Networks get dropped. It’s a fact of life.
• When your network gets dropped, sometimes
Reflector app has a “reluctance” to put the iPhone
back onto the network you just made.
• To recover:
– In iPhone Settings: Forget Network.
– On Computer: Quit Reflector App.
– On Computer: Restart Reflector App.
– In iPhone: Follow procedure to mirror display
Smartphone Setup 2
Helpful Tip Learned from(horrible)Experience
45

46. Setup options (I’m sure you can find more, too!)
1. Smartphone Setup 1 can be used identically
for iPad testing setup
2. Wireless Setup
iPad
46

47. Pros & Cons of iPad Setup 1 versus 2
47
Setup 1 Setup 2
Pros Pros
Major: You can see ANYTHING
on the screen
Look mom, no wires!
Cool annotation features
Cons Cons
Camera contraption can be a
little harder to deal with on an
iPad than a smartphone.
Can only share content in a
browser – not an app.
Hands/fingers can get in the
way of the screen
Can’t minimize the
participants’ face
Performance hit on iPad

48. Schematic – Wireless iPad Setup
iPad
(w/ built-in webcam)
Video
Recording
Laptop
(w/ lots of free space)
Running:
-GoToMeeting
-GTM VOIP for audio
-GTM video for face
Running:
-GoToMeeting (attending)
-Camtasia
Observers
Laptop
(attending
WebEx)
Phone
or VOIP
Lab
48

49. iPad
Materials you need
• 1 laptop
• GoToMeeting (no, not WebEx, GoToMeeting.)
• Screen capture software (we like Camtasia for Mac, Morae for
PC)
• iPad 2 or 3 (not iPad 1 because it doesn’t have a camera to capture
participant’s face)
• An app, website, or prototype
49

50. a) (Put the video recording computer aside for a minute.)
b) On your iPad, launch GoToMeeting
– Make sure your call is working by calling into the VOIP
number from another phone
– Enable the participant videocam feature (if you want it)
– Share your screen
c) Unbelievably – that’s it.
iPad setup
Step 2 – Fire up GoToMeeting
50

51. 51
a) On your iPad, launch GoToMeeting. Enable the participant
videocam feature (if you want it). Share your screen

52. 52
Cool Annotation Features!

53. iPad setup
Step 3 – Fire up the machine doing the screen
capture
53
Once GoToMeeting is running on the Image Sharing
machine….
a) Launch a browser on the video recording laptop and
join the meeting. Make sure everything looks how
you want it to look.
b) Launch Camtasia (or your favorite screen
capture/editing software) on the video recording
laptop.
c) Start recording your session

54. • In theory, you can run Display Recorder App
on the iPad during your session to record
audio, video and even users’ face, while
screen-sharing the usability session.
• Word to the wise: This is a serious
performance hit on the iPad, but it’s been
done.
– Tip: Forgo the video option in favor of better
performance on the screen.
Adventure Club for iPad
54

55. • Rather than sitting in the traditional
“moderator” position looking over the
participant’s shoulder, sit across from
them, and watch the session via screensharing
on your own (3rd) laptop.
– Your business partners can IM spontaneous
questions to you.
– You can take your own notes if you need to, from
right there
Uber-Efficiency Mode
55

56. Questions?
56