Report on a graduate course in emerging technologies where students conducted peer reviews on web-based projects of classmates, on criteria OTHER THAN those of the instructor.

1. BADGING / MICRO-CREDENTIALING –
CLASSROOM PEER REVIEW,
FACILITATED AND EXTENDED BY
TECHNOLOGY
“Badging” as a way to encourage review,
ownership, professional-networking,
lateral learning, & reflection
By: Amy McQuigge (amy.mcquigge@esc.edu)
& Eileen O’Connor (eileen.oconnor@esc.edu)
SUNY CIT 2013 Empire State College

2. Agenda
 Introduction to course where badging was
implemented
 Overview of the badging concept &
approaches
 Data and findings within the course
 Concluding comments / Q&A

3. Badging – within an online course
• Course & the student population
– Learning with Emerging Technologies: Theory & Practice
– Working adults take this introductory master’s courses within
the Masters of Arts in Learning & Emerging Technology
• The multiple purposes for badging – (many follow-up
studies are possible for analyzing the results)
– Peer review & awarding; extending course evaluation
– “Lateral” learning; encouraging review & reflection; building
community & understanding
– Modeling an emerging approach that these students might use
in their work
• Designing learning systems that use emerging tech
• Having students consider further applications

4. Discussion
boards
Media postings
w/ d-boards
Second Life – w
d-board follow-
up
Badge voting
Blog sharing on
learning
Research /
resource findings
via Google Drive
Final
presentation /
review
Course interactions –
community highly prized
course objective; a
collaborative project in the
last module

5. The “badging” components & process
within each module

6. The integration of the badge process
• Web-accessible assignments geared to course
requirements :
– additional criteria were posted for the badging;
– Informal headings / descriptive / “stylistically” reinforcing the
non-graded aspect of these peer-given awards;
– Additional category of creative / inventive was created for
badge 4
• Postings for the class
– after votes were tallied results were emailed & put in class blog
– final presentation of summary badges in a virtual meeting
• The peer-voting was within Google Forms
-- Data was then gathered; results were shared
anonymously by instructor
• Formal presentation of badges in Second Life at
end of course

7. Instructor review vs. peer review
Assessment
points
Instructor (for
credit)
Formal assessment with
criteria; aligned w/
course objectives
Reports
Annotated
Bibliographies
Sandbox w/
technologies w/
rubrics
Peers (for badge
& not credit)
5 badges – peer review
opportunities; encouraging
creativity
Holistic criteria
Informal
categories
Open ended

8. Example of the instructor rubric – specific & detailed;
focus is on sandbox-level skills (not design or
instruction of e-mediated environment)

9. Badge criteria – holistic & informal;
employed a deliberately different tone,
language, & expectations
BADGE SCALE – Prezi / Presentation; Website; YouTube; YouTube
No go (1) Pewter (2) Bronze (3) Gold (4)
Won’t even make the
grade for the
assignments minimum
criteria
Minimally acceptable
for the assignment but
nothing noteworthy in
this aspect
Interesting & useful;
solid display of
expertise on this
criteria
Wow, I am learning and
taking notes here – a
great job; I’ll have my
friends visit here too
-- Used the scale above for the first 3 badges; added “willingness to try new things” to
the categories above for Badge 4 (Facebook)
-- Wanting to encourage students to think outside the box; some concerns that the
evaluation was not “fair” to the different skill levels

10. All badge-prompts encouraged new areas of thinking

11. Created in Google Drive Forms –
summarized and tallied

12. TOTAL COUNT OF BADGE VOTES CAST
WITHIN THE 4 INDIVIDUAL AREAS
• The course asked for 3 reviews minimum; 9 students in the course; thus
there should be at least 27 votes per badge
• Badge 2 – some student-participation issues
TOTAL COUNT OF BADGE VOTERS BY THE DIFFERENT AREAS
BADGE 1 BADGE 2 BADGE 3 BADGE 4
Prezi/PPT Website YouTube Facebook
51 22 33 28

13. Average votes per student per category
(Cat 2 & 3 have multiple criteria)
Values
Row Labels
Average of
aesthetics
Average of
Category2
Average of
Category3
Average of “try
new” category
total
average
Std1 3.3 3.1 3.1 2.7 3.1
Std2 3.5 3.5 3.7 3.5 3.6
Std3 3.4 3.4 3.3 3.2 3.3
Std4 3.2 3.2 2.8 3.0 3.0
Std5 3.5 3.6 3.5 4.0 3.7
Std6 3.3 3.3 2.9 2.0 2.9
Std7 3.4 3.5 3.6 3.8 3.6
Std8 3.4 3.1 3.3 3.0 3.2
Std9 2.5 2.4 2.4 3.0 2.6
Grand Total 3.3 3.2 3.2 3.2 3.2

14. Std1 Std2 Std3 Std4 Std5 Std6 Std7 Std8 Std9
Average of aes 3.3 3.5 3.4 3.2 3.5 3.3 3.4 3.4 2.5
Average of cat2 3.1 3.5 3.4 3.2 3.6 3.3 3.5 3.1 2.4
Average of cat3 3.1 3.7 3.3 2.8 3.5 2.9 3.6 3.3 2.4
Average of try new 2.7 3.5 3.2 3.0 4.0 2.0 3.8 3.0 3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
AverageVotes(4highestvalue)
Student Average Votes per Category (several criteria in Cat 2 & 3)
“Try new features”
category was only
added in the 4th
badge event

15. Std1 Std2 Std3 Std4 Std5 Std6 Std7 Std8 Std9
Average of aes 3.3 3.5 3.4 3.2 3.5 3.3 3.4 3.4 2.5
Average of cat2 3.1 3.5 3.4 3.2 3.6 3.3 3.5 3.1 2.4
Average of cat3 3.1 3.7 3.3 2.8 3.5 2.9 3.6 3.3 2.4
Average of try new 2.7 3.5 3.2 3.0 4.0 2.0 3.8 3.0 3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
AxisTitle
Average of the votes for each student by category (cat 2 & 3
have multiple criteria) – results aligned w/ prior student
background & experience
Std 6 & 9 were from more
“traditional” institutions &
environments; Std 5 & 2 had prior
experience with more innovative tech

16. Comparison – instructor vs. peer ranking, mindful that
the criteria are different for both deliberations
Instructor ranking, on “regular”
course assignments & criteria
for the entire course Student # Peer ranking
1 Std8 4
2 Std2 2
2 Std3 3
2 Std6 7
3 Std5 1
3 Std9 8
4 Std4 6
4 Std7 2
5 Std1 5
Note: instructor ranking would be similar to students on the criteria
considered; however, the peer assessment was on criteria beyond the
course requirements

17. Conclusions
• Students participation was often beyond the
minimum requirements;
• Students were very collegially engaged, however,
the entire course reinforces this approach –
further study needed to isolate the role of the
badging process itself;
• Peer review mapped quite closely to instructor
review & ranking;
• Using peer review (with badging) allowed for
course & learning expansion in informal ways

18. Considerations for adaptation / further study
• Adapt concept & process to your needs
– Consider how to integrate badges to encourage more examination & reflection
within your courses
• You will still need to work beyond the LMS – this is extra step; however, integrating web
2.0 technologies often means extending beyond standard LMS features
• Emphasize ad explain the differences between peer review vs. instructor review
• Consider course objectives, audience, and the audience expectation – but don’t be afraid
to integrate and evaluate new approaches for reflection and community building
– Consider the level of granularity you need in the badges
– Introduce badges early and complete a “cycle” (through to picking-up the
badges) within the course
• When to issue the badges – can be associated with a course / motivational – engagement
factors
• For external presentation of works, perhaps
• Consider how to use badges – as rewards? as competitions? to encourage
reflection & review?
• Consider when to distribute badges – during the course or at the end?
• Badges can address emerging understandings about learning:
– Is professional interactions and lateral learning important?
– Is additional, peer-level credentialing of value for learning within a course?