Paper Prototyping @ EIA 2013

1. Paper Prototype
Evaluation

2. Evaluation
• Well, before addressing the evaluation of
paper prototypes, perhaps we should consider
reasoning upon what we should
evaluate, right?
– Let’s then take a look into concepts such as
usefulness, usability and user experience

3. Usefulness
• Usefulness is a crucial quality of any product
or service
– Concerns the degree to which a product enables a
user to achieve his or her goals, and is an
assessment of the user’s willingness to use the
product or service at all
• Without usefulness, other measures make little or no
sense, because it will just sit on the shelf
• We name this the fundamental quality of a
product or service

4. Usability
• According to the ISO standard
– Usability is the extent to which a product or
service can be used by specific users to achieve
predefined goals in a specified context of use
– The standard defines three main usability
dimensions:
• Effectiveness,
• Efficiency, and
• Satisfaction

5. Usability
• Anyway, when looking into usability, we
normally account for:
– Effectiveness
– Efficiency
– Satisfaction
– Learnability
– Memorability
• We name these the pragmatic qualities of a
product or service

6. Usability
• Common reason for the delivery of less usable
products are
– System focused process
– Poorly integrated teams
– Design and development mismatches
– And of course, designing usable products is
difficult
• That’s what we are pushing paper prototyping in this
moment of the process

7. User Experience
• According to its ISO standard, user experience
is a person’s perception and responses that
result from the use or anticipated use of a
product or service

8. User Experience
• User experience subsumes usability and
includes includes
– emotions, beliefs, preferences, perceptions, physic
al and psychological responses, behaviors and
accomplishments that occur before, during and
after use
• We name these the effected of the
hedonic, or pleasure related, qualities of a
product or service

9. Evaluation
• As stating in the beginning, before addressing
the evaluation of paper prototypes, we should
consider reasoning upon what we should
evaluate
– And now that we briefly cover
usefulness, usability and user experience, I believe
it’s fair to say that we should target going as far as
usability

10. Evaluation
• In general, product and service evaluation can
be formative or summative
• Paper prototype evaluation is mostly
formative

11. Evaluation
• Paper prototype evaluation focus on…
– The most significant issues preventing users from
accomplishing their goals
– What works and what do users find frustrating
– What are the most common errors or mistakes
users are making
– Assessing the improvements being made from one
design interaction to the next
– Identifying issues that are expected to remain
even when the product is launched

12. Evaluation
• Today we look into
– Inspection methods
• These are methods where an expert evaluator inspects
a product or service
– Testing methods
• These are methods where products and services are
evaluated by testing them on real users

13. Inspection methods
• There are also several but we will address:
– Cognitive walk-through
– Heuristic evaluation

14. Cognitive walk-through
• The purpose is to verify if the paper prototype
actually allows the fulfillment of the selected
user stories
– This is a within team activity that ensures that
your prototype complies with the identified user
stories

15. Cognitive walk-through
• Designers and developers of the product or
service then walk through the steps as a
group, asking themselves a set of questions at
each step
– Data is gathered during the walk-through, and
afterwards a report of potential issues is compiled
– Finally the evaluated proposition is redesigned to
address the issues identified

16. Heuristic evaluation
• This is a type of evaluation ideally carried out
by an expert.
– It specifically involves evaluators examining the
design and judging its compliance with recognized
principles, such as Jakob Nielsen’s 10 usability
heuristics
• These evaluation methods are now widely taught and
practiced in the new media sector, where products and
services are often designed in a short space of time on
a budget that may restrict the amount of money
available to provide for other types of interface testing

17. Testing methods
• There are several but we will address:
– Co-discovery
– Wizard of Oz

18. Co-discovery
• Two users attempt to perform tasks together
while being observed
– They are to help each other in the same manner
as they would if they were working together to
accomplish a common goal using the product
– They are encouraged to explain what they are
thinking about while working on the tasks

19. Co-discovery
• The designers and developers should refrain
from explaining the design decisions and
rather focus on getting the most of the pair of
users tacking with your prototype
– Note taking is fundamental and you should run
this kind of test until no significant additional
information is feed back into the design process

20. Wizard of OZ
• This is a testing approach built upon a paper
device instead of using a working technological
artifact
– This king of testing involves systematic observation
under controlled conditions to determine how well
people can use a product or service
– Rather than showing users a rough draft and asking,
Do you understand this?, this kind of testing involves
watching people trying to use something for its
intended purpose

21. Wizard of OZ
• Setting up such a test involves asking the test
subjects to recreate a set of user stories after
being introduced to the underlying scenario
– For example, to test the attachment function of an
e-mail program, a scenario would describe a
situation where a person needs to send an e-mail
attachment, and ask him or her to undertake this
task
– The aim is to observe how people function in a
realistic manner, so that developers can see
problem areas, and what people like.

22. Testing methods
• This kind of evaluation should be repeated
until no significant added value comes from
bring in an additional subject
– The usual number is 5, but other claim otherwise

23. Pre and post-tests
• When applying testing methods, pre-test and
post-test questionnaires are also used to
gather feedback on the product being tested
– A common questionnaire is the 25 years old
System Usability Scale

24. System Usability Scale

25. Evaluation protocol
• But these methods are useless without an
adequate evaluation protocol
• While designing your evaluation protocol, you
should take into account…
– A mixed selection of inspection and testing
methods;
– The user stories supported by your prototype; and
– The affordances of your paper prototype.

26. Evaluation protocol
• Examples are provided in the companion blog
but please note that…
– In all cases, evaluation protocols should be piloted
to ensure that once they are being applied, you
are actually focusing in assessing the prototype
and not on solving evaluation protocol issues

27. Now it’s up to you to
make it happen

28. paperprototypingateia
.wordpress.com