1. Effects of Age and
Think-Aloud Protocol on
Eye-Tracking Data and
Usability Measures
Jennifer Romano Bergstrom @romanocog
Fors Marsh Group @forsmarshgroup
&
Erica Olmsted-Hawala
U.S. Census Bureau
EyeTrackUX | Las Vegas, NV | June 2012

2. Usability Testing
• Technique in user-centered design to
evaluate a product by testing it with actual
users of the product
• Value is getting direct feedback on how
real users work with the product

3. How Usability is Measured
• International Organization for
Standardization (ISO) 9242:11
The extent to which a product can be used by
specified users to achieve specified goals with
effectiveness, efficiency and satisfaction in a
specified context of use.

4. Usability Testing
• Metrics
– Task accuracy
– Task efficiency
– User satisfaction
• Usability professional (test administrator)
identifies problem areas of product based
on user performance & verbalized
feedback

5. Think-Aloud Protocol
• Primary tool used by usability
professionals when conducting usability
tests
• Test administrator
— Listens to what participant says
— Watches what participant does
— Cannot directly observe what a participant
thinks
5

6. Concurrent Think Aloud (CTA)
Protocol
• Participant encouraged to “think aloud”
while working on task
– Running commentary of thought-action
process
– Participant articulates as they work on task
6

7. Retrospective Think-Aloud (RTA)
Protocol
• Participant completes session in silence
– Watches a video replay of session
• Sometime with eye tracking
– Articulates while watching video replay

8. Research on CTA and RTA
• Usability studies have compared effects of
CTA and RTA on usability performance
– Accuracy
– Efficiency
– Satisfaction
– Verbal data quality
• Results vary by study

9. Pros and Cons of Think Aloud
CTA RTA
• Metrics might not be accurate • Metrics “more pure”
• Emotions • Reflection
• Users describing the what • Users describe their own
• Real sense of users’ interpretation or
experience with site understanding of why
• Shorter sessions • Users’ insight into what the
• Real time responses design issues are
• Longer sessions
• Memory is fallible

10. Cognitive Aging

11. Aging: The Facts
• The population is increasing.

12. The Facts
390,000
370,000
Population of US in thousands
350,000
330,000
310,000
290,000
270,000
250,000
2000 2005 2009 2010 2015 2020 2025 2030

13. The Facts
• The population is increasing.
• Older adults are the fastest growing part of
the population.

14. The Facts
120,000
100,000
Population of US in thousands
80,000
18 to 24 years
60,000 25 to 44 years
45 to 64 years
40,000 65 years and over
20,000
0
2000 2005 2009 2010 2015 2020 2025 2030

15. The Facts
Percent Change in US Population and in Older Adults: 2000-2029.

16. The Facts
• The population is increasing.
• Older adults are the fastest growing part of
the population.
• More people are using the Internet.

17. The Facts
90
80
Percent of households with Internet
70
60
50
40
30
20
10
0
1997 2000 2001 2003 2007 2009

18. The Facts

19. The Facts
• The population is increasing.
• Older adults are the fastest growing part of
the population.
• More people are using the Internet.
• Cognitive decline is inherent with age.

20. The Facts
• The population is increasing.
• Older adults are the fastest growing part of
the population.
• More people are using the Internet.
• Cognitive decline is inherent with age.

21. Cognition and Internet Examples
• Spatial skills
– Where things are located
• Short-term memory
– Remembering the purpose
• Working memory
– Remembering information entered earlier to
respond to later questions
• Sustained attention
– Complex Web sites

22. The Facts
• The population is increasing.
• Older adults are the fastest growing part of
the population.
• More people are using the Internet.
• Cognitive decline is inherent with age.
• Web site designers often do not take
cognitive decline into account when
designing Web sites and surveys.

23. Cognitive Aging,
Think-Aloud Protocol
and Usability Performance

24. Challenges of Dual tasks
• Performance for older adults decreases
when they attempt to complete dual tasks

25. Research Questions
• Does the dual task of thinking aloud while
working on finding information impact
usability performance and eye fixation
patterns?
• What think-aloud protocol (CTA / RTA)
should practitioners use when conducting
usability studies with older adults?

26. Our Study
• 2 (CTA / RTA) x 3 (Age) between-subjects
study.
• Outcome Variables
– Accuracy
– Efficiency
– Satisfaction
– Eye-tracking data
– Verbal reports

27. Participants
• 95 participants
• Metro DC area
• Three age groups
– Young adults ages 18-28
– Middle-age adults ages 40-50
– Older adults ages 64-76
• One year of prior experience using Internet
• Reimbursed $40.00

28. Participant Characteristics
Young adults Middle-age adults Older adults
CTA RTA CTA RTA CTA RTA
Gender 3M / 8F 6M / 15F 5M / 7F 9M / 10F 5M / 6F 9M / 12F
Mean Age 22 22 45 46 67 68
Mean Years of
education
15 15 15 14 16 16
Ease of learning
a new Web site*
1.36 1.47 1.83 1.42 2.36 2.38
Ease of
navigating the 1.09 1.05 1.33 1.16 1.72 1.57
Internet*
*Scale: 1 (Not difficult at all) – 5 (Extremely difficult).

29. Methods
• 5 information-gathering tasks
– Typical tasks users come to site to accomplish
– Tasks organized by easy to hard
– Collected accuracy and efficiency measures
– Eye tracking on all tasks
• Satisfaction questionnaire and debriefing at end
• CTA group
– CTA on all tasks
• RTA group
– RTA on last task only
– Gaze video playback
• Test administrator and participant were in separate
rooms

30. Website Tested
• US Census Bureau’s American FactFinder
– Primary website about population, housing,
and economy of the US
– Legacy version of site

31. AFF Website: http://factfinder.census.gov

32. Example Tasks
• First easy task: You want to learn more
about Maryland, and specifically about
how many people live there. How many
people live in Maryland?
• First hard task: You are doing a report on
schooling in the U.S. What percent of the
population in Florida, California and Texas
completed college in 2008?

33. Results
• Task accuracy
• Task efficiency
• User satisfaction
• Eye Tracking
• Verbal reports

34. Young and middle age adults
completed more tasks successfully
than older adults
**
80%
*
70%
60%
50%
40% CTA
30% RTA
20%
10%
0%
Young adults Middle-age adults Older adults
*p < 0.05; **p < 0.01

35. Middle-age adults in CTA had higher
accuracy in hard task than in RTA
80% *
70%
60%
50%
40% CTA
RTA
30%
20%
10%
0%
*p = 0.07 Middle-age adult--1st hard task

36. Young adults completed tasks
faster than older adults
**
4:19
3:50
3:21
2:52
2:24
CTA
1:55
RTA
1:26
0:57
0:28
0:00
Young adults Middle-age adults Older adults
**p = 0.05, *p = 0.10

37. Middle-age adults in RTA completed
hard task faster than in CTA
10:48 *
9:36
8:24
7:12
6:00
CTA
4:48
RTA
3:36
2:24
1:12
0:00
*p = 0.10 Middle age adults--1st hard task

38. No significant age-related
differences in user satisfaction
9
8
7
6
5
CTA
4
RTA
3
2
1
0
Young adults Middle-age adults Older adults

39. Speed / Accuracy Trade Off
• Cognitive demands influence the speed
and accuracy with which adults process
information
• Our findings for the middle-age adults are
consistent with earlier cognitive work.
– Higher accuracy for hard task with CTA
– Slower for hard task with CTA

40. Interaction of Age and Think Aloud
• Older adults’ performance decreases
when they perform dual tasks.
• Our findings for the middle age adults are
consistent with earlier cognitive work.
– Older adults: floor level
– Young adults: ceiling level

41. Eye Tracking
• Total number of fixations per AOIs
– Task 1
• Total number of fixations on main page
– Task 1

42. AOIs
Top banner
Top navigation
Population clock
Left navigation
Whole screen

43. Left navigation: No difference
between CTA and RTA
25
20
15
10
5
0
CTA RTA

44. Left navigation: Older adults had
fewer fixations than young adults
for CTA only
10 *
9
8
7
6
5 CTA
4 RTA
3
2
1
0
*p < 0.05 Young adults Middle-age adults Older adults

45. Top banner: More fixations during
CTA
4
3
2 CTA
RTA
1
0
CTA RTA
p = 0.06

46. Top banner: No significant
differences, but pattern is the same
5
4
3
CTA
2 RTA
1
0
Young adults Middle-age adults Older adults

47. Top navigation: No difference
between CTA and RTA
2
1 CTA
RTA
0
CTA RTA

48. Top navigation: Older adults had
fewer fixations than young adults
for CTA only
4 *
3
2 CTA
RTA
1
0
Young adults Middle-age adults Older adults
*p < 0.05

49. Population clock: No difference
between CTA and RTA
6
5
4
3 CTA
2 RTA
1
0
CTA RTA

50. Population clock: No significant
differences, but CTA pattern is similar
8
7
6
5
4 CTA
3 RTA
2
1
0
Young adults Middle-age adults Older adults

51. Whole screen: No difference
between CTA and RTA
60
50
40
30 CTA
RTA
20
10
0
CTA RTA

52. Whole screen: Middle-age adults
had fewer fixations than young
80
adults for CTA only
70 *
60
50
40 CTA
30 RTA
20
10
0
Young adults Middle-age adults Older adults
*p = 0.05

53. Seconds to complete Task 1: No
significant differences
160
140
120
100
80 CTA
60 RTA
40
20
0
Young adults Middle-age adults Older adults

54. Accuracy for Task 1: No significant
differences
100%
90%
80%
70%
60%
50% CTA
40% RTA
30%
20%
10%
0%
Young adults Middle-age adults Older adults

55. Language Analysis
• Present tense and past tense
verbalizations
• Also examined
– Positive emotion words (e.g., good, helpful)
– Words of assent (e.g., absolutely, cool)
– Negative emotion words (e.g., annoy, awful)
– Negations (e.g., aren’t, can’t)
– Affective processing (e.g., caring, insult)

56. Participants in CTA spoke more present tense;
Participants in RTA spoke more past tense
14%
12%
10%
8%
CTA
6% RTA
4%
2%
0%
Past tense Present tense

57. Young adults in CTA spoke 5%
fewer utterances in present tense
16%
14%
12%
10%
Young CTA
8%
Middle-age CTA
6% Older CTA
4%
2%
0%
Present tense

58. Young adults in RTA follow similar pattern:
fewer utterances of present tense
12%
10%
8%
Young RTA
6%
Middle-age RTA
Older RTA
4%
2%
0%
Present tense

59. Words of Insight and Cognition
• Test administrator observations during sessions
– CTA more immediate what they were doing
– RTA reflections on actions
• Insight words
– Complex
– Meaning
– Prove
• Cognitive processes
– Ambiguous
– Hesitate
– Solution

60. Participants in RTA spoke more
words of insight and cognition
25%
20%
15%
CTA
10% RTA
5%
0%
Insight words Cognitive processes

61. Summary of Usability Results I
• Accuracy – Young and middle-age adults
completed more tasks successfully (higher
accuracy) than older adults.
• Efficiency – Young adults completed tasks
faster than older adults.
• Satisfaction – Age did not affect
satisfaction.

62. Summary of Usability Results II
• Condition effect – Middle-age adults who
completed tasks in silence (RTA)
completed the most difficult task faster and
with higher accuracy than middle-age
adults who thought aloud (CTA) while
completing tasks.

63. Summary of Eye-Tracking Results
• When age was not a factor, there appeared
to be no differences in fixations between CTA
and RTA
• When age was taken into account
– Older had fewer fixations than young in CTA only
• Left navigation
• Top navigation
• Similar pattern for other AOIs
– Middle-age had fewer fixations than young in CTA
only
• Whole screen
– Not related to accuracy or efficiency

64. Implications
• Recruit participants of varying age groups for usability
testing; conduct usability tests with older adults.
• Be aware of cognitive demands and effects that thinking
aloud has on aging adults.
• Consider pros and cons when creating a protocol and
selecting a think-aloud method.
– Use CTA for a richer verbal commentary, particularly with respect to
participants’ emotional reaction to the site.
– Use RTA when interested in verbal feedback related to participants’
insight on the issues they had with the application.
– Use RTA when pure eye-tracking data is necessary.

65. Future Analysis
• Fixation duration
• Sum across all tasks
– Easy vs. hard tasks
• Performance in first 10 seconds
• Correlations
– Education
– Experience
– Cognitive abilities
• Verbalizations
– Phrases
– Number of words, phrases
• Click data
• Identified usability issues

66. Future Research
• Include wider ranges of age to test the
assertion that the age-related effect is
incremental and occurs slowly through a
lifetime
• Alter RTA protocol
• Replicate with different web sites
• Are there specific gaze patterns by age
and think-aloud protocol?

67. Thank You!
Jennifer Romano Bergstrom
Fors Marsh Group
571.858.3795
jbergstrom@forsmarshgroup.com
@romanocog
Proceedings paper from STC re verbalizations is available.
Eye-tracking paper is forthcoming.