Text versus voice? A number of approaches have been suggested to optimise mobile survey response rates. Whilst addressing survey design can help for closed questions, what about open-ended ones? The presenters will share the results of an experimental study comparing classical text-input formats in self-administered mobile surveys with an alternative, voice-recognition-based procedure in mobile survey projects. They will report on the relative differences of these two formats on usability and non-response levels, and will provide practical recommendations for mobile survey projects.

1. VOICE RECOGNITION IN SELF-
ADMINISTERED MOBILE SURVEYS
LORENZ GRÄF, SVEN SCHERRER, MARC HAUPT,
MICHAEL BOSNJAK, MARIO CALLEGARO
April 19, 2011 Mobile Study III
1 Globalpark. Manage what matters.

2. AGENDA
1.Background
2.Research Questions
3.Method
4.Results
5.Discussion and Outlook
April 19, 2011 Mobile Study III
2 Globalpark. Manage what matters.

3. AGENDA
1.Background
2.Research Questions
3.Method
4.Results
5.Discussion and Outlook
April 19, 2011 Mobile Study III
3 Globalpark. Manage what matters.

4. USABILITY OF MOBILE SURVEYS: PAST FINDINGS
Subjective Usability Assessment Observed
Post-hoc survey (Web) one week after mobile survey completion Item- Drop-
Indicators for usability score: fluency, simplicity, ease of use NR Out
Single choice
Einfachauswahl untereinander 89,2
Multiple choice
Mehrfachauswahl untereinander 87,3
9%
Fragetyp
Drop-Down menu
Geschlossene Auswahlliste 82,7
Textfield
Textfeld einzeilig 74,7
45% 9%
Image map
Fragetyp mit Bild 87,9
23%
65,00 73,75 82,50 91,25 100,00
Usability score (Range: 0-100 Punkte)
April 19, 2011 Mobile Study III
4 Globalpark. Manage what matters.

5. AGENDA
1.Background
2.Research Questions
3.Method
4.Results
5.Discussion and Outlook
April 19, 2011 Mobile Study III
5 Globalpark. Manage what matters.

6. OVERALL GOAL
Improving mobile input formats requiring rich, non-
standardized information, which is usually captured using
text input in other self-administered survey modes.
Alternative to (typed) text-input formats:
Embedding voice input into self-administered mobile
surveys, similar to CATI or esp. IVR surveys
Today: Subjective usability assessments as indicators of a
potential improvement text versus voice
Next step: Analyzing objective / unobtrusive data on
nonresponse (item-nonresponse, drop-out)
April 19, 2011 Mobile Study III
6 Globalpark. Manage what matters.

7. TECHNICAL IMPLEMENTATION: IPHONE APP
April 19, 2011 Mobile Study III
7 Globalpark. Manage what matters.

8. TECHNICAL IMPLEMENTATION: ANDROID
April 19, 2011 Mobile Study III
8 Globalpark. Manage what matters.

9. RESEARCH QUESTION(S) TESTED
Differences between
iPhone-App-based voice capturing
versus
Android-based voice recognition
versus
Classical (touchscreen-based) text input
on the following dependent variables:
usability assessment immediately following a specific voice/text
input format,
(objective indicators of nonresponse, esp. item-nonresponse and
drop-out,)
retrospective usability assessement Ø 5 days following
participation?
April 19, 2011 Mobile Study III
9 Globalpark. Manage what matters.

10. AGENDA
1.Background
2.Research Questions
3.Method
4.Results
5.Discussion and Outlook
April 19, 2011 Mobile Study III
10 Globalpark. Manage what matters.

11. MINGLE PANEL – MOBILE USERS
N=10518
11 04/11/11 Mobile Study III
Globalpark. Manage what matters.

12. MINGLE PANEL – SMARTPHONE OS
N=3136
12 04/11/11 Mobile Study III
Globalpark. Manage what matters.

13. PARTICIPANTS
13 April 19, 2011 Mobile Study III
Globalpark. Manage what matters.

14. PARTICIPANTS
6,6 %
Sales
14,2 %
Employment
Employed
Status Student
Unemployed
79,2 %
14 April 19, 2011 Mobile Study III
Globalpark. Manage what matters.

15. PROCEDURE
Part 1: Pre-Survey (23.03. – 28.03.2011, 6 Days, time of invitation: 5 PM GMT)
• Device Screening
• Android: Technical requirements check
• iPhone: Instructions App installation
• Pre-commitment
Part 2: Main survey (28.03. – 03.04.2011, 7 Days, time of invitation: 5 PM GMT)
• Test drive: Voice recording / voice recognition
• Three text/voice input items (short: single number, medium: list three items, long answers: elaborated
text/answer)
• Usability assessment following each text/voice input item
Part 3: Post survey (05.04. – 09.04.2011, 5 days, time of invitation: 8 AM GMT)
• Retrospective usability assessment
• Evaluation of Main-Survey Drop-Out
• Preferred input method short / long entries
15 April 19, 2011 Mobile Study III
Globalpark. Manage what matters.

16. PROCEDURE (PARTS 2 AND 3)
Respondi panel members (N= 235)
iPhone App Voice Android Voice
Android Text Input
Capturing Group Recognition Group
Group (n3.1= 49)
(n1.1= 126) (n2.1= 60)
Immediate usability assessment for three corresponding questions.
(n1.2= 87) (n2.2= 36) (n3.2= 38)
Retrospective usability assessment (after Ø 5 days, Web-based).
(n1.3= 79) (n2.3= 36) (n3.3= 34)
April 19, 2011 Mobile Study III
16 Globalpark. Manage what matters.

17. DEPENDENT MEASURES
Immediate usability assessment:
Mean score (transformed to 0 -100 range) of three identically worded, closed-
ended single items evaluating the extent to which (iPhone) voice recording /
(Android VR) voice recognition / (Android text) text input was appropriate to
capture the intended response („How do you evaluate the performance of the
voice- recognition/-capturing /touchscreen keyboard feature?“, scaled from
´very good´ thru ´very poor´).
Retrospective usability assessment (reminder screenshots were displayed):
Mean score (transformed to 0 - 100 range) of the following four indicators:
ease of use, fluency, usability, unintended entries .
Non-response (work in progress):
Item-nonresponse: Aggregated item-nonresponse percentage for all three
questions with experimentally manipulated input formats (per experimental
condition)
Drop-Out: Overall drop-out rates (per experimental condition)
April 19, 2011 Mobile Study III
17 Globalpark. Manage what matters.

18. AGENDA
1.Background
2.Research Questions
3.Method
4.Results
5.Discussion and Outlook
April 19, 2011 Mobile Study III
18 Globalpark. Manage what matters.

19. USABILITY ASSESSMENT; IMMEDIATE
Immediate usability
assessment best and
most homogeneous
for Android-based
text input (!)
April 19, 2011 Mobile Study III
19 Globalpark. Manage what matters.

20. USABILITY ASSESSMENT: RETROSPECT
Retrospective usability
assessment best for
Globalpark´s iPhone-App-
based voice capturing
technology.
April 19, 2011 Mobile Study III
20 Globalpark. Manage what matters.

21. MISC: INPUT FORMAT PREFERENCES
iPhone App Android
100 %
57 %
76 %
50 %
9%
9% 33 %
13 %
0%
Short Entry Long Entry Short Entry Long Entry
Don't Know
Touchscreen
Voice Capt/Rec
Physical Keyboard
April 19, 2011 Mobile Study III
21 Globalpark. Manage what matters.

22. SUMMARY OF RESULTS
Immediate usability assessment:
Android touchscreen > better than > iPhone App VC > better than > Android VR
Android touchscreen ratings most homogeneous (largest agreements)
Retrospective usability assessment:
iPhone App VC > better than > Android touchscreen > better than > Android VR
Android VR shows largest dispersion among ratings (largest disagreement)
Additional result:
The larger the amount of text information to be entered, the more preferred
are voice input formats on the expense of the touchscreen-based version,
but only for iPhone participants (not for Android participants).
April 19, 2011 Mobile Study III
22 Globalpark. Manage what matters.

23. AGENDA
1.Background
2.Research Questions
3.Method
4.Results
5.Discussion and Outlook
April 19, 2011 Mobile Study III
23 Globalpark. Manage what matters.

24. DISCUSSION AND OUTLOOK
Further analyses using data from the current study:
Item-Nonresponse analysis
Drop-Out analysis (partly done, see Appendix)
Quality of content between groups?
Technology and/or selection effects? For instance,
iPhone users are more experienced with Apps and more ´attached´ to their
Smartphone, ´causing´ more favourable post-hoc evaluations of App-based
input formats;
men versus women systematically overrepresented in certain groups,
other systematic differences?
Next steps:
Results consistent over time and in the context of ´real life´ applications?
Please feel free to use VC/VR in your own studies!
April 19, 2011 Mobile Study III
24 Globalpark. Manage what matters.

25. THANK YOU!
April 19, 2011 Mobile Study III
25 Globalpark. Manage what matters.

26. Appendix
April 19, 2011 Mobile Study III
26 Globalpark. Manage what matters.

27. DROP-OUT (WORK IN PROGRESS)
Voice Voice
Regular
Capturing Recognition
Test Drive
9 (9,3%) 5 (10,4%) -
Instructions
Test Drive 1 (1,0%) 0 (0%) 0 (0%)
Single figure 0 (0%) 0 (0%) 0 (0%)
List 0 (0%) 0 (0%) 0 (0%)
Free text 0 (0%) 0 (0%) 0 (0%)
Drop Out on other
0 (0%) 7 (14,6%) 2 (5%)
pages
Drop Out Overall 10 (10,3%) 12 (25%) 2 (5%)
N= 97 (100%) 48 (100%) 40 (100%)
April 19, 2011 Mobile Study III
27 Globalpark. Manage what matters.

28. Comments and Data Samples
Comments Voicecapturing
„Da ich nicht sprechen konnte, konnte ich den Fragebogen nicht zu Ende durchgehen.“
„Ich war krank, böse Erkältung und konnte mich nicht motivieren, zumal mir das Sprechen superschwer fiel.“
Comments Voicerecognition
„Spracherkennung ist bei meinem Handy für solche Felder nicht verfügbar, nur für die Google-Suche“
„Ich hatte nur die englische Spracheingabe. Habe im Internet nach Hilfestellungen gesucht um auf deutsche
Spracheingabe umzustellen, allerdings nichts gefunden.“
„Spracheingabe funktionierte nicht richtig. Spracheingabe ging nur als englischer Text. Alles Einstellungen meines
HTC waren auf Deutsch und ich habe keine Möglichkeit gefunden die Spracheingabe auf Deutsch zu berichtigen.“
Comments Touchscreen Input
„Gute Erhebungstechnik, gerne wieder über smartphone...“
„Würde mich sehr freuen weitere Umfragen mit dem Handy beantworten zu dürfen.“
Data Samples Voicerecognition – Testphrase „Diese Befragung ist sehr interessant“
„Die herbe frau geküsst interessant“
“These are The 5 sense sons”
“disappear.com sia into the sun”
„Die befragung sea of sand“
„wiese befragung ist sehr interessant“
28 04/11/11 Mobile Study III
Globalpark. Manage what matters.

29. Demo - Audio Samples iPhone
Apple Quicktime required
0: Testtrial
mingle61 (optimal)
0: Single Word
mingle 99 (Optimal)
mingle 166 (Background noise)
mingle 173 (Additional info)
2: List
mingle31 / 19 (optimal)
mingle16 (Background noise)
mingle8 (Real List)
3: Open answer
mingle198 (Slight saxon dialect)
mingle187 (Whole sentence)
mingle192 (slightly annoyed)
29 04/11/11 Mobile Study III
Globalpark. Manage what matters.