Presented to the Badges for Lifelong Learning grantees of the DML Competition, February 18, 2013.

1. Formative Evaluation for
Educational Product
Development
*Or, how to tell if this thing you made is worth its salt
Vanessa Gennarelli
Learning Lead, P2PU
Master’s Candidate, Harvard University
@mozzadrella

2. Why Should I Test My Product?
Can folks use it?
Evaluate usability and nomenclature.
Do users like it?
Measure appeal and engagement.
Does this audience get it?
Test comprehension and recall.

3. What is Formative Evaluation?
Testing an educational product with users
during development.
The results inform the direction of the product.
Process can be conducted at any time.
Speedy stuff--time is of the essence.

4. Who Uses Formative Evaluation?

5. Who Uses Formative Evaluation?

6. Who Uses Formative Evaluation?

7. That’s peachy, Vanessa.

8. That’s peachy, Vanessa.
Can I do this myself?
Do I have time for formative evaluation?
Is there an easy way to get started?

9. The answer is “yes”...
And you will crush it.

10. Methods: Interviews
A structured protocol of questions with a target user.
Benefits:
Learn the vocabulary and logic of your users.
Deep investigation of appeal, engagement and
comprehension.
Limitations:
Recruitment, research and analysis is time-intensive.

11. Methods: Interviews
Sample questions:
Appeal:
What badge would you want to share with a friend?
Which would you apply for? Not apply for?
Comprehension:
[Stop activity]: What just happened?
What does this badge mean?

12. Methods: Think Aloud Protocol
As a user walks through your product they narrate what they
notice, think, feel, look at, and decide.
Benefits:
Observe hesitations, hiccups and trouble spots.
Observe positive and negative reactions.
Limitations:
Interviewer needs to stay quiet, can be awkward.

13. Methods: Think Aloud Protocol
Sample Questions:
You have completed a project and are applying for
a badge. Apply for a badge, describing what you
notice and think at each step of the way.
Select a badge you’d like to apply for. Describe and
explain each action you make.

14. Example: P2PU Course UX
Usertesting.com: asked folks to create a course
3 respondents
Video recording of users--where do they get stuck?

15. Example: P2PU Course UX
Results informed our new Course UX
Full report: http://www.slideshare.net/VanessaGennarelli/
design-research-course-creation

16. Methods: Focus Groups
A structured protocol with 2+ more targeted users.
Benefits:
Feedback from several users at once.
Limitations:
In a group setting, bias can be an issue.
Recruiting several users can be difficult.

17. Methods: Focus Groups
Sample questions:
Appeal: What did you like most? Least?
Engagement: What would you like to know
more about?
Comprehension: What do you understand the
difference between [Badge X] and [Badge Y]
to be?

18. Methods: Questionnaires
Series of questions that may have standardized answers.
Benefits:
Lightweight in regards to collection and analysis.
Limitations:
Tricky to ask precise and reliable questions.
No observational data, questionnaires are all self-
reported data.

19. Methods: Questionnaires
Sample Questions
Appeal:
Rate how much you like/dislike [Badge X] on a
1-5 scale (1=least, 5=most)
Comprehension:
Rate the clarity of [Badge X] on a 1-5 scale
(1=least clear, 5=most clear)
What are 3 things you learned about from [X]?

20. Methods: Click-testing
Veryifyapp asks users to click on their preferred design,
annotate the screen, or test the labels on your page
Benefits:
Great for expectation questions “Where would you
expect to find...”
Testing navigation & nomenclature
Inexpensive & fast
Limitations:
Adult testers only

21. Example: P2PU Home Page
Users were asked to highlight where they could click to
create a course
538 viewers, 62 responses

22. Example: P2PU Home Page
Results informed our new P2PU.org home page
Full report: http://www.slideshare.net/VanessaGennarelli/
user-research-p2puorg-home-page

23. Sample Project Schedule
Identify research areas--what are you evaluating?
1
Decide upon tools and methods.
Recruit participants from target user base.
NB: You may need a release.
Conduct research.
Analyze results & present to project team.
Target timeframe: 1 week.

24. Mad Props
Karen Brennan, Assistant Professor, Harvard Graduate School
1
of Education
Mindy Brooks, Director of Education and Research, Sesame
Workshop
Brad Felix, Chief Products Officer, Sangari Global Education
Allison Gevarter, Interactive Producer, CloudKid Media
Ilona E. Holland, Lecturer, Harvard Graduate School of
Education

25. Ask things!
Vanessa Gennarelli
Learning Lead, P2PU
Master’s Candidate, Harvard University
@mozzadrella