“Oh GOSH! Reflecting on Hackteria's Collaborative Practices in a Global Do-It...
Subjective questionnaires
1. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Usability Testing
Subjective Questionnaires
Cristina Cachero
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
2. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Inquiry method: you ask your users about what they do,
but you do not observe it directly.
There are several proposals that have shown enough
internal and external validity, and that can be applied at
different times during the performance test
After all the tasks have been finished (test-level questionnaires)
Before/after each task is performed (task-level questionnaires)
Also, questionnaires may include open questions that
can be codified and analyzed.
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
2
Subjective Questionnaires
3. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Test-level (post-test) questionnaires (1/2):
CSUQ: Computer System Usability Questionnaire [Lewis 1995]: 19 sentences,
all positive. They measure four dimensions: system utility, information quality,
interface quality and general satisfaction. It was devised to be administered off-
line
PSSUQ: Post-Study System Usability Questionnaire: variation of CSUQ to be
administered in person
QUIS: Questionnaire for User Interface Satisfaction. 27 sentences divided in five
groups: general reaction, screen, terminology/system info, learning, system
capabilities.
EUCS : End-user computing satisfaction [Abdinnour-Helm 2006]. 12 items
representing five dimensions: content, accuracy, format, ease of use and
timeliness. All the items are combined into a global EUCS measure. 5-point
scale (Almost never… almost always)
USE: Usefulness, Satisfaction and Ease of Use [Lund 2001]. 30 items divided in
four categories: utility, satisfaction, ease of use and ease of learning. For each
one, seven-point scale items..This project has been funded with support from the European
Commission under the Lifelong Learning Programme
3
Subjective Questionnaires
5. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
5
Post-test SQ: EUCS [Abdinnour-Helm 2006].
6. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
6
Post-test SQ: USE [Lund 2001]
http://hcibib.org/perlman/question.cgi?form=USE
7. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
SUS (Systems Usability Scale) [Brooke 1996] 10
questions, 5 formulated in positive and 5 formulated in
negative. It gives a global measure of the site usability (it
does not distinguish among different components). It is
the most reliable [Tullis and Stetson, 2004]
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
7
Post-test SQ: SUS [Brooke 1996]
The mean SUS value is 66%, the 25
percentile is 57% and the 75
percentile is 77%. This means that we
should get a SUS greater than 80%
(taking into account confidence
intervals) to be reasonably sure of the
global satisfaction of our users.
8. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Activity: Read the original SUS paper, published by
Brooke in 1996. You have the paper available in the
Moodle platform.
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
8
Post-test SQ: SUS
9. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
9
Post-test SQ: SUS: Exercise
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
4. I think that I would need the support of a technical person to be able to use this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system
Strongly Strongly
Disagree Agree
10. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
10
Post-test SQ: SUS: Score?
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
4. I think that I would need the support of a technical person to be able to use this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system
Strongly Strongly
Disagree Agree
11. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
11
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
4. I think that I would need the support of a technical person to be able to use this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system
Strongly Strongly
Disagree Agree
Post-test SQ: SUS. Steps
Reverse items 2,4,6,8,10
Sum positions for each item (0 to 4)
Multiply by 2.5
4
1
1
4
1
2
1
1
4
3
TOTAL
SCORE: 22
SUS SCORE:
22*2.5=55
12. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
12
Post-test SQ: SUS. Threshold value
Which should be
our SUS objective?
13. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
13
Post-test SQ: SUS. Adaptation
The SUS vocabulary can be adapted to the idyosyncrasy
of the particular system
Example: OHIM: system->web site
14. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
14
Post-test SQ: SUS. Calculator
Calculator: http://www.measuringux.com/SUS-scores.xls
15. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Net Promoter Scores: measure obtained through a single
question (loyalty of the user to the app):
How likely is it that you’ll recommend this product to a friend or
colleague? (0..10)
Three segments:
Promoters: Responses from 9 to 10
Passives: Responses from 7 to 8
Detractors: Responses from 0 to 6
Promoter score (-100..+100): %Promoters-%Detractors
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
15
Post-test SQ: Net Promoter Score
16. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Questionnaires that provide rankings to compare your
results against the results from other similar apps
SUPR-Q (Sauro)
WAMMI (www.wammi.org) (SUMI successor)
ACSI (www.TheACSI.org): particularly interesting
form government websites
OpinionLab (www.OpinionLab.com)
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
16
Post-test SQ: Rankings
17. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
17
SQ: Rankings. SUPR-Q (4 sub-scales)
NPS
18. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
18
Post-test SQ: Rankings. SUPR-Q.
SUPR-Q Score: Sum all items + ½ item 10
19. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
19
Post-test SQ: Rankings. SUPR-Q.
Besides the values for each factor, SUPR-Q is backed on a
proprietary DB that allows to generate a ranking of percentiles
E.g. your app belongs to percentil 75 regarding
usability, what means, it is among the best 25%.
The SUPR-Q usability factor has a strong correlation with a SUS
score, r = .96. p < .001, meaning just four questions account for 93
percent of the variation in SUS (.96 squared). These questions are a
good substitute for SUS on websites
Validity and reliability: http://www.suprq.com/
20. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
20
Post-test SQ: Rankings. WAMMI
http://www.wammi.com/samples/index.html
Results are divided into five areas: attractiveness,
controllability, efficiency, helpfulness and
Learnability, plus an overall usability score.
The scores are standardized (from comparison to
their reference database), so 50 is medium and
100 is perfect.
21. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
21
Post-test SQ: Rankings. WAMMI
Sample graphic showing how a give website positions in
reference to average scores in each axis
22. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Circle five words that describe what you think about
this design:
What are the three things you like best and least
about the Web site?
_____________________________________________
______________________________________________
______________________________________________
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
22
Post-test SQ: proprietary (OHIM 1/2)
Cute
Stable
Responsive
Friendly
Helpful
Reputable
Approachable
Reliable
Cluttered
Good
Confident
Trustworthy
Current and cool
Service oriented
Boring
Easy to use
Confusing
Comfortable
Annoying
Informative
Out-of-date
High-tech
Sensitive
Secure
Straightforward
Amateurish
23. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
If you could make one significant change to this Web site, what
change would you make?
(Ask the participant if he is using/knows other IP websites) How do
you find the site in comparison to other IP web sites?
Do you feel this site is current? Why?
If you were to describe this site to a colleague in a sentence or two,
what would you say?
Do you use the current OHIM web site? If Yes then Do you think
that the new OHIM web site clarity of structure is: Better, Same As,
Worse or Don’t know, than the current one?
Do you have any other questions or comments about the Web site or
your experiences with it?
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
23
Post-test SQ: proprietary (OHIM 2/2)
24. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Expectation Rating
After Scenario Questionnaire (ASQ)
Awareness-Usefulness gap
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
24
Task level (Post-Scenario) SQ
25. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Pre-task: How easy/difficult do you thing that the
following task is going to be?
Post-task: How easy/difficult has been for you to
carry out this task?
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
25
Post-Scenario SQ: Expectation Rating
26. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Three post-task questions:
I feel satisfied with the easiness with which I have
completed the task.
I feel satisfied with the time that it has taken me to
complete this task.
I feel satisfied with the support information (online
help, messages, documentation, and so on) that I
have had available while I was completing this task
(only if your system offers online help)
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
26
Post-Scenario SQ: ASQ
27. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Two post-task questions:
‘‘Were you aware of this functionality prior to this study?
(yes or no)
On a 1 to 5 scale, how useful is this functionality to you?
Calculation:
Convert the likert item to a binary scale (e.g. 4 or 5 useful,
rest not useful) and draw the graph.
Differences between awareness and usefulness indicate
which tasks to promote (e.g. by redesigning the visual
hierarchy) in your app.
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
27
Post-Scenario SQ: Awar-Usefuln Gap
28. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Graph sample:
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
28
Post-Scenario SQ: Awar-Usefuln Gap
29. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
More questionnaires are appearing by the day in the
literature. Many of them have been deemed
necessary to cover the idiosyncrasy of new
platforms/domains
E.g. MPUQ: Mobile Phone Usability Questionnaire
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
29
SQ: Other questionnaires
Reliability and Validity of the Mobile Phone
Usability Questionnaire (MPUQ)
30. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Can be post-test or post-task
Reasons why the users are promoters or detractors of your
product
Insights from users gathered from field studies
Complaints about a product sent to the customer service
Why the task was difficult to complete
…
The way of processing this kind of questions is to
turn the open comments into categories, quantify
them and analyze them.
http://www.measuringusability.com/blog/quantify-comments.php
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
30
SQ: Open Questions
31. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
All the questionnaires must be validated with respect to
their reliability, validity and utility. Once validated, they
still need to be revalidated when the context of
application varies.
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
31
SQ: Validation
32. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Lewis95] Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and
instructions for use. International Journal of Human-Computer Interaction, 7, 57–78.
[Lund 2001] Lund, A.M. (2001) Measuring Usability with the USE Questionnaire. STC Usability SIG Newsletter, 8:2
[Chin 88] Chin, J. P., Diehl, V. A., and Norman, K. (1988). Development of an instrument measuring user
satisfaction of the human-computer interface. In: CHI '88. Conference Proceedings on Human Factors in
Computing Systems. New York: ACM, pp. 213–218.
[Brooke 1996] Brooke, J. (1996). SUS: A “quick and dirty” usability scale, In Jordan, P. W., Thomas, B. T., and
Weerdmeester, B. A. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp. 189–194.
[Kirakowski 1996] Kirakowski, J. (1996). The software usability measurement inventory: Background and usage. In
Jordan, P., Thomas, B., and Weerdmeester, B. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp.
169–177.
[Lewis 1991] Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability
studies: the ASQ. SIGCHI Bulletin, 23(1), 78–81.
[Abdinnour-Helm 2006]. Using the End-User Computing Satisfaction (EUCS) Instrument to Measure Satisfaction
with a Web Site.
[Tullis 2008] Thomas Tullis and , William Albert (2008). Measuring the User Experience: Collecting, Analyzing, and
Presenting Usability Metrics (Interactive Technologies). Morgan Kauffman.
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
32
References
33. Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
These slides are made available under the license Creative
Commons Attribution-NonCommercial-NoDerivs CC BY-
NC-ND. More information about license:
http://creativecommons.org/licenses/by-nc-nd/3.0/.
These slides were created under Leonardo da Vinci
Partnerships Project 2012-1-PL1-LEO04-28181 GUI
USABILITY AND ACCESSIBILITY: EXCHANGING
KNOWLEDGE AND EXPERIENCES (http://usability-
accessibility.org/).
This project has been funded with support from the European
Commission under the Lifelong Learning Programme
33
Attributions
Notas do Editor
In order to compute the overall value of the scale, the first step is to make all the items consistent (in this case, making lower values imply more negative feelings). For this purpose, we must reverse the items 2, 4, 6, 8 and 10. If the user strongly agreed (5) to the sentence ‘I found the system unnecessariliy complex’, reversing the item is equivalent to saying that the user strongly disagrees to the opposite of that sentence (the positive feeling). Since she strongly disagrees, she does not add to the SUM score.
In order to be on the safe side, we should aim at getting SUS scores greater than 80% (including confidence intervals).
In order to be on the safe side, we should aim at getting SUS scores greater than 80% (including confidence intervals).
In order to be on the safe side, we should aim at getting SUS scores greater than 80% (including confidence intervals).
Figura extraida del libro Measuring Web Usability
[Lewis 1991] Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability
studies: the ASQ. SIGCHI Bulletin, 23(1), 78–81.
One type of analysis that can be very valuable is to look at the difference between
participants’ awareness of a specific piece of information or functionality and the
perceived usefulness of that same piece of information or functionality once they
are made aware of it. For example, if a vast majority of participants are unaware of
some specific functionality, but once they notice it they find it very useful, you
should promote or highlight that functionality in some way.
To analyze awareness–usefulness gaps, you must have both an awareness and a
usefulness metric. We typically ask participants about awareness as a yes/no
question—for example, ‘‘Were you aware of this functionality prior to this study?
(yes or no).’’ Then we ask: ‘‘On a 1 to 5 scale, how useful is this functionality to you?
(1 = Not at all useful; 5 = Very useful).’’ This assumes that they have had a couple of
minutes to explore the functionality. Next, you will need to convert the rating-scale
data into a top-2-box score so that you have an apples-to-apples comparison. Simply
plot the percent of participants who are aware of the functionality next to the
percent of those who found the functionality useful (percent top-2-box). The difference
between the two bars is called the awareness–usefulness gap (see Figure 6.29).
One type of analysis that can be very valuable is to look at the difference between
participants’ awareness of a specific piece of information or functionality and the
perceived usefulness of that same piece of information or functionality once they
are made aware of it. For example, if a vast majority of participants are unaware of
some specific functionality, but once they notice it they find it very useful, you
should promote or highlight that functionality in some way.
To analyze awareness–usefulness gaps, you must have both an awareness and a
usefulness metric. We typically ask participants about awareness as a yes/no
question—for example, ‘‘Were you aware of this functionality prior to this study?
(yes or no).’’ Then we ask: ‘‘On a 1 to 5 scale, how useful is this functionality to you?
(1 = Not at all useful; 5 = Very useful).’’ This assumes that they have had a couple of
minutes to explore the functionality. Next, you will need to convert the rating-scale
data into a top-2-box score so that you have an apples-to-apples comparison. Simply
plot the percent of participants who are aware of the functionality next to the
percent of those who found the functionality useful (percent top-2-box). The difference
between the two bars is called the awareness–usefulness gap (see Figure 6.29).
You can then further
analyze the data by generating a confidence interval to understand what percent of all users likely
feel this way (see Chapter 3).
You can then further
analyze the data by generating a confidence interval to understand what percent of all users likely
feel this way (see Chapter 3).
You can then further
analyze the data by generating a confidence interval to understand what percent of all users likely
feel this way (see Chapter 3).