Self-reporting techniques, such as data logging or a diary, are frequently used in long-term studies, but prone to subjects’ forgetfulness and other sources of inaccuracy. We conducted a six-week self-reporting study on smartphone usage in or- der to investigate the accuracy of self-reported information, and used logged data as ground truth to compare the sub- jects’ reports against. Subjects never recorded more than 70% and, depending on the requested reporting interval, down to less than 40% of actual app usages. They significantly over- estimated how long they used apps. While subjects forgot self-reports when no automatic reminders were sent, a high reporting frequency was perceived as uncomfortable and bur- densome. Most significantly, self-reporting even changed the actual app usage of users and hence can lead to deceptive measures if a study relies on no other data sources. With this contribution, we provide empirical quantitative long-term data on the reliability of self-reported data col- lected with mobile devices. We aim to make researchers aware of the caveats of self-reporting and give recommenda- tions for maximizing the reliability of results when conduct- ing large-scale, long-term app usage studies.

1. INVESTIGATING
SELF-REPORTING BEHAVIOR
IN LONG-TERM STUDIES
Andreas Möller ✽, Matthias Kranz ❖,
Barbara Schmid ✽, Stefan Diewald ✽, Luis Roalter ✽
✽ Technische Universität München, Germany
❖ Universität Passau, Germany

2. Logging
Self-Reporting
DATA COLLECTION

3. Forgetting
Annoyance
Sluggishness
Self-Perception

4. RESEARCH QUESTIONS
Accuracy? Change over time?
Influence of
reporting frequency?
Reliability maximization?
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris

5. BACKGROUND
■ Electronic diaries show higher
compliance
(Hufford & Shields, 2002)
■ Mobile phone as survey tool
(Consolvo et al., 2007)
Consolvo et al., 2007
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris

6. CONTRIBUTIONS
■ Empirical quantitative long-term data
on reliability of information
collected with mobile devices
■ Recommendations for maximizing
result reliability
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris

7. A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
METHODOLOGY
Evaluate
reporting
behavior
Ground truth
Can be gained
in automated
way
Limited effort
Smartphone
usage
GOAL
REQUIRE-
MENTS
SOLUTION
Frequently used apps
Installed by everyone

8. Mail Facebook
Frequently used apps
Installed by everyone
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris

9. A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
PROCEEDING
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 6 + 4
Pre-Questionnaire Reminder Emails Post-Questionnaire
Post-Post-
Questionnaire
Requested Self-Reports
& Logging

10. TASK
■ Answer questionnaire after Facebook or Mail
usage
■ Report as accurate as possible
1. How long did you use the app?
2. How many times did you use the app
without answering a questionnaire?
1 direct report
n indirect reports
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris

11. SELF-REPORTING AND EXPERIENCE
SAMPLING ASSISTANT
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris

12. SERENA
■ App usage logging
■ Server upload
■ Questionnaire triggers
□ Event-based
□ Time-based
□ Manually
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris

13. A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
3 CONDITIONS
Voluntary Interval Event
No trigger Daily trigger
Trigger after
app usage
30 Participants
3,631 Mail usages
3,181 Facebook usages

14. SESSIONS
Voluntary Interval Event
Amount of reported Facebook usages
37.6%
63.8%
54.3%
Indirect reports
Direct reports
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris

15. DURATIONS
Voluntary
Interval
Event
1:29
1:29
1:22
2:52
3:02
3:35
Facebook sessions
Real
Self-Reported
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris

16. OVER TIME
■ Self-report ratio decreases
■ Actual usage decreases
„Answering the questionnaire
changed my Facebook usage habits.“
Voluntary 2.3
Interval 2.2
Event 3.5
5 = strongly agree
A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris

17. A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
DISCUSSION
Subjects
reported max.
70% of usage
Commitment
decreased
Reports may
influence actual
behavior
Subjects
overestimated
session length
Reminder emails
pushed
commitment in
2nd phase
Behavior change
with increasing
burden
Little control:
forgetting
High control:
burden
Trigger influence
lower than
hypothesized
App usage
decreased
Most subjects
would report
max. 4 weeks

18. andreas.moeller@tum.de
https://vmi.lmt.ei.tum.de/serena

19. Please cite this work as follows:
A. Möller, M. Kranz, B. Schmid, L. Roalter, S. Diewald
Investigating Self-Reporting Behavior In Long-Term Studies
In: Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems (CHI 2013), pp. 2931-2940, Paris, France, April-May 2013.
If you use BibTex, please use the following entry:
@inproceedings{chi2013selfreport,
author = {Andreas M"{o}ller and Matthias Kranz and Barbara Schmid and Luis
Roalter and Stefan Diewald},
title = {Investigating Self-Reporting Behavior In Long-Term Studies},
booktitle = {Proceedings of the 2013 ACM annual conference on Human Factors in
Computing Systems},
pages = {2931--2940},
series = {CHI '13},
year = {2013},
isbn = {978-1-4503-1899-0},
location = {Paris, France},
numpages = {10},
publisher = {ACM},
address = {New York, NY, USA},
}