1. The OLDS MOOC Evaluation
Plan and an Experiment
in Recognising Learning using
Badges
Slides by Simon Cross and Rebecca Galley
Institute of Educational Technology, The Open University
Presented by Simon Cross as part of:
Cross, S. and Mor, Y. (2013) OLDS MOOC: design,
evaluation and badges, eLearning Community Event:
MOOCs at the Open University: past, present and future,
13 March 2013, OU, Milton Keynes
For more on the event visit: http://cloudworks.ac.uk/cloud/view/8043

2. Presentation overview
1. OLDS MOOC Evaluation Plan
(themes, structure, questions)
2. OLDS MOOC Badge Strategy
(what are badges, what are they
for, and how could we use them)
3. Use and user comments
(examples of twitter posts that
comment on role of badges +
Bonus slide on expectations)

3. September 2012
Questions on
Previous Experience Registration
of MOOCs
Questionnaire Prior understanding
and knowledge
OLDS
n=35 n=1400+
MOOC
MOOC ‘Evaluation
Panel’ recruited
before start for Expectations and
Evaluation
Strategy
longitudinal study Motivations
and to ensure we Questionnaire: one
are likely to interview week prior to start
some people at end
who ‘dropped out’ n=60+
n=15
MOOC STARTS
Weekly manual Weekly capture and
recording of use archive of all tweets
statistics from made across the
Cloudworks and channels
Bibsonomy
Recording of data
Feedback form relating to badge
submissions approvals
n=25 at present n=80+ and rising
MOOC ENDS
MOOC Experience
Quesitonnaire - open
to all
Opens March
Telephone interviews
with Panel and others
thought to be
interesting cases
Opens March
April 2013

4. MOOC Evaluation Themes
Motivations Quality / Quantity
Expectations Badges / Reward
Groups & Support MOOC Makers

5. Expectations and Motivation in MOOCs
A review of pre-course survey results and
user posts will provide insight in to what
people expected of the MOOC and why
they were engaging with it.
What variety of expectation is
there?
How does this compare to the
reality of the MOOC experience?
Who and when could we use this
type of MOOC structure/format?
How is content and structure
informed by the aims and intentions
of the designers?

6. Use of Badges and approach to reward
Nine badges have been developed in
conjunction with a badging strategy. A
method of creating and awarding badges
added to Cloudworks.
What role did the badges have in
motivating participants and
preventing withdrawal?
What can we learn about using
badges to support learners and
promote learning?

7. … September 2012
1. What are badges?
2. What are they for?
3. How could we use them?

8. … 1. What is a badge?
My working definition:
“A badge is visual public symbol that
communicates to others a particular
quality, achievement or affiliation
possessed by the owner. It is one of
a variety of ways that such recognition
can take a material form.”

9. … 2. What are they for?
The Role of the badge
for receiver and issuer…
As a motivator
As an engager
As a meaning maker
As a signifier of learning objectives
As a valuer
As a symbol of identity
As a means of association
As an empower
As an entrencher
http://www.olds.ac.uk/blog/moocbadgingandthelearningarc

10. http://www.olds.ac.uk/blog/moocbadgingandthelearningarc

11. … 3. How can we use them?
Additional Cloudworks technical
development undertaken:
•Badge unload functionality
•Define approval criteria
•Apply for badge
•Approval (two processes)
•Display on personal profile
•Aligned to Mozilla badging

12. … 3. How can we use them?
Additional Cloudworks technical
development

13. OLDS MOOC Badges
Achievement
Endeavour and
Progress
Practice
http://www.olds.ac.uk/blog/olds-moocbadgingstrategy

15. OLDS MOOC Participant Profile
From a sample of registrations
http://www.olds.ac.uk/blog/moocprofiles

16. Study planning
[Am] interested in gaining badges, and working with colleagues in a
local study group.
[My personal aims]: 1. completing one week of the MOOC and applying
for a badge (maybe not getting the badge but at least applying for one
week:)
I like the Badges, works as motivation for me. [Will] chose to get
[the] Endeavour [ones] due [to] time constraints. Maybe Achievement.

17. Badges and Participation Following
Contributing / Doing
Unique views
Badge
1310 1220 340
200
100
1 2 3 4
Week 1 Week 3
1. Introduce yourself cloudscape 3. Week 3 Cloudscape
2. Dream (Project) Bazaar cloudscape 4. Sharing LD representations Cloud (Activity 6-7)

18. A bit of fun ... even a surprise
@[B]: Earned my Week 1Badge from #oldsmooc ! Unexpectedly
pleased to have a reward for efforts! What fun and motivates …
Wooh [I] too have got a badge from #oldsmooc_w1 how #awsome
Just applied for my #oldsmooc week 1 badge. Not that it matters but ....
if I get it I will print it out (in colour) and pin it on my lapel
First #oldsmooc badge...what fun, feel like a Girl Guide again
Thursday motivation - got my 3 weeks #oldsmooc badge”
@[T] Well done and snap! Delighted to get my 3 weeks badge too - likin'
those badges more than expected!

19. Confidence, collecting, something to show
Hooray for my first @openbadges!
... and badges def. do work. I'm bagsing as many as I can to show that
I'm learning loads… I love #openbadges. See my #oldsmooc badges [x]
Guiding reflective learning
Impressed #oldsmooc badge linked to evidence of reflective blogging
#openbadges > me too, I got a badge :-)
@[R] I love the #oldsmooc badges http://t... - keep up the reporting on
your learning, love it

20. Reason to ask for peer review
Facilitator: OER developer badge is crowdverified, you need to get your
peers to approve your application. don't be shy to ask!
Have applied for OER developer badge, pls verify http://t...
Hi Folks Just applied 4 my #oldsmooc collaborator badge http://t.co/[x]
Could you do the honours?

21. Supporting sense of community
and checking progress
Just applied for Resource Gatherer Badge...anyone else? thanks to
@[D]'s reminder I have got another #oldsmooc badge http://t.co/[X]...
I got another #oldsmooc badge!! #oldsmooc_wk4 looks puzzling
though …See how we go :)
Yay! #cloudworks #badges now link to "Who has been awarded
this badge?”
… Just need to figure out my #mozilla backpack now :)

22. ‘Own Path’ participants
I haven't qualified for a badge yet as I'm pursuing a somewhat
singular path through #oldsmooc -- I'm OK with that
“Oh - and a final word: don't let badge envy ruin your day :)”
“I’m starting to really enjoy OLDSMOOC, I’ve accepted
that I’m no longer on the approved route but wandering
about reading, thinking about stuff and chatting with
like-minded people is great learning anyway. Having
said that, I got my Week 1 badge today!”

23. Concluding remarks
• Promoting deeper and more expansive learning?
• Promoting richer learning experience?
• Alternatives to summative and formative assessment?
• Appeal to broader range of study motivations / practice?
• Help students better value their study?
• Help students better communicate their achievements?

24. Concluding remarks
Design considerations
• Overjustification
• Reflective practice / Affirmation
• Status / Reputation
• Overlearning
• Goal setting / Identity
• Community endorsement
• Whenever
• Fun and enjoyment
• Collecting …

25. What do you want to get out of it?
Immersion in an online social environment As a bonus, here is a summary
Brushing up on skills or knowledge of just some of the reasons
Learning new skills, knowledge, participation skills
participants gave for wanting to
International working with many people from many different countries
Local working with colleagues do the MOOC. More to follow in
Structured format for thinking about and designing learning the full project evaluation
Being challenged
Taking time out to reflect on practice
New networks
Trying a MOOC / a (c)MOOC / new technologies
Achieving objectives Just 8 of 39 mentioned completion
Completing the course
Completing the subject of the course - a learning project
Exploring or gaining exposure to new resources / design tools / teaching tools or technology
Experiencing a MOOC and seeing how it works
Evaluating the potential of the MOOC
Documenting experience to publish, for own reflection or for students
Research case study or for research data
Team work skills required / seeing how team working is made to work
Peer discussion of issues around the subjects with peers
Peer feedback on own learning projects and ideas
Collaboration with others on developing a course
Sharing and aggregating tools
Complementing formal study (post-graduate / graduate courses)
Gain badges
Group working with local colleagues
Learning how to exploit technology for teaching
Overview of field
Extending existing learnt knowledge in area previously studied
Cutting edge ideas and research

26. This presentation was prepared on the eve of the final week
of the OLDS MOOC. A full evaluation will take place between
March and May 2013 so watch out for more detailed
analysis and review at:
OLDS MOOC Blog: http://www.olds.ac.uk/blog
And personal reflection at:
Simon’s Blog: http://latestendeavour.wordpress.com/
Please use an appropriate reference or acknowledgement when
citing this presentation.
For further information please contact:
simon.j.cross@open.ac.uk
rebecca.galley@open.ac.uk
yishay.mor@open.ac.uk