FINAL_Tenorio_masters_thesis

Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 1
Digital Badges as Micro-Credentials:
Using Digital Badges in an Online Learning System
by
Lisa Tenorio
A Field Project and Thesis Submitted to the
DIGITAL MEDIA AND LEARNING PROGRAM
In Partial Fulfillment of the Requirements
For the Degree of
MASTER OF EDUCATION
University of San Francisco
December 2014

Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS i
Table
of
Contents

List
of
Tables
.........................................................................................................................................
iii

List
of
Figures
.......................................................................................................................................
iv

Chapter
1
Introduction
.......................................................................................................................
1

Statement
of
the
Problem
............................................................................................................................
2

Background
and
Need
...................................................................................................................................
4

Purpose
of
the
Project
...................................................................................................................................
7

Project
Objectives
...........................................................................................................................................
7

Definition
of
Terms
........................................................................................................................................
8

Summary
..........................................................................................................................................................
10

Chapter
2
Literature
Review
..........................................................................................................
11

Student
Perceptions
of
the
Value
of
Credentials
................................................................................
12

Student
Perceptions
of
Digital
Badges
and
Alternative
Credentials.
.....................................................
14

Employer
Perceptions
of
the
Value
of
Credentials
............................................................................
17

The
Impact
of
Credential
Source
on
Employer
Perceptions.
.....................................................................
19

The
Impact
of
the
Labor
Market
on
Employer
Perceptions.
......................................................................
21

The
Impact
of
Employers’
Personal
Experience
and
Beliefs.
....................................................................
22

Conclusion
.......................................................................................................................................................
23

Badge
Design
&
Implementation
.............................................................................................................
24

Case
Study
One:
IP&T
EdTec
Course.
...................................................................................................................
25

Case
Study
Two:
Computer
Science
Student
Network
(CS2N).
................................................................
27

Case
Study
Three:
TRAKLA2
and
Daechschen).
..............................................................................................
31

Case
Study
Four:
RL
Hit
List.
...................................................................................................................................
33

Conclusion
.......................................................................................................................................................
35

DIGITAL BADGES AS MICRO-CREDENTIALS ii
Chapter
3
Project
Development
....................................................................................................
37

Background
of
the
Project
.........................................................................................................................
38

Project
Components
.....................................................................................................................................
39

Design
&
Implementation.
........................................................................................................................................
40

Learning
Paths.
..............................................................................................................................................................
41

Incentive
Framework.
................................................................................................................................................
42

Badge
Design.
.................................................................................................................................................................
47

Conclusion
.......................................................................................................................................................
48

Chapter
4
Project
Outcome
.............................................................................................................
50

Issues
&
Challenges
......................................................................................................................................
51

Future
Development
....................................................................................................................................
52

Recommendations
for
Continued
Study
...............................................................................................
54

Conclusion
.......................................................................................................................................................
55

Appendix
A
Digital
Badging
Platforms
........................................................................................
56

Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS iii
List of Tables
Table 1. Points Systems in Learning Platforms. Points given to each lesson in popular online
learning platforms. ................................................................................................................46

Table 2. Trailhead Point System. Final proposal for Trailhead point incentive system. ..............46

Table 3. Trailhead Usage Metrics. Use of Trailhead system since launch. ..................................50

Table 4. Trailhead Feedback Metrics. Social media posts since launch.......................................50

DIGITAL BADGES AS MICRO-CREDENTIALS iv
List of Figures
Figure 1. IP&T EdTec Badge Hierarchy: Badge levels and types................................................26

Figure 2. CS2N Badging Hierarchy: Badge types and structure...................................................28

Figure 3. CS2N Badge Examples: Example of the CS2N badge design ......................................29

Figure 4. TRAKLA2 Badges: Example of badge types and design..............................................32

Figure 5. RL Hit List: Physical Pin Badges. .................................................................................34

Figure 6. Trailhead Incentive System: Initial design proposal.....................................................44

Figure 7. Final Trailhead Incentive System: Final design of the badging and award system for the
Trailhead project. ..................................................................................................................45

Figure 8. Trailhead Profile and Badge Design. The Trailhead badges and user profile page where
badges are stored...................................................................................................................48

Figure 9. Proposal for integrating projects into incentive framework. The updated Trailhead
incentive framework..............................................................................................................53

Chapter 1
Introduction
Recent developments in technology-enhanced learning such as the Massive Online Open
Courses (MOOCs), Khan Academy, and the availability of Open Education Resources (OER)
have sparked questions about how these new learning experiences, which occur outside a
traditional educational institution, can be assessed and credentialed (Conrad, 2013; Mazoué,
2012). At the same time gamification, or applying game mechanics to non-game activities, has
become a popular topic in education as a way to motivate and engage learners (Lee & Hammer,
2011). Digital badges have emerged from the concepts of gamification not only as a
motivational tool, but also as a potential form of credentialing and assessment that could provide
value to students and employers as a way to recognize the specific skills and competencies
acquired in both formal and informal learning environments (Watters, 2012).
Certification and alternative forms of credentialing are not a new concept and many
industries such as law, accounting, medicine, nursing, and teaching have used a system of
certification and licensing in addition to traditional academic degrees. In fact, the Adult
Education Training and Education Survey (ATES) Pilot Study conducted in 2012 found that
38% of adults in the US reported having some type of alternative credential such as a
certification, certificate, or license, in addition to, or in lieu of a degree (Bielick et al., 2013).
However, what makes badges unique is the ability to recognize individual skills and
competencies and more discrete components of learning. Also, the official launch of the Open
Badge Infrastructure (OBI) by the Mozilla Foundation in 2013 provides a standard framework
for badges not available for other certification programs. This framework enables any individual
or organization to create and issue these micro-credentials while providing a centralized platform

DIGITAL BADGES AS MICRO-CREDENTIALS 2
to communicate the metadata about each badge, including the assessment and evidence supplied
by the badge earner (Rosewell, 2012). Having a mechanism to provide open access to
information about the requirements that were fulfilled to earn a badge makes digital badges
different from other forms of credentialing which are more difficult to validate or assess.
Statement of the Problem
Why do we even need new forms of credentialing? Why not leave credentialing to the
traditional degree programs and universities? One problem is that traditional credentials, such as
university degrees, may not adequately prepare graduates for the available job roles and the
needs of industry. In fact, a 2011 report by ACT found that a higher level of educational
attainment did not necessarily correlate to having specific work readiness skills (ACT, 2011).
The unemployment rate of recent college graduates further validates this issue. The Economic
Policy Institute’s report on the employment outlook for the class of 2012 reported that the
unemployment for college graduates under age 25 was twice as high as the national average at
9.4% (Shierholz, Wething, & Sabadish, 2012). At the same time, employers struggle to find
candidates for available job openings. A 2012 survey of employers conducted by McKinsey
(Mourshed, Farrell, & Barton) found that only 43 percent of employers could find sufficient
skilled entry-level workers.
Even if job seekers have the skills that employers seek, most university degrees do not
convey granular information about the specific skills and competencies that learners have
acquired. Not all university students possess the same skills and competencies and not all
university programs are equivalent, but employers lack an efficient way to gauge candidate’s
skills in specific areas. Some states are working to fill that gap by creating a national work
readiness credential that is separate from any specific educational credential. The work

readiness credential measures entry-level workplace skills such as reading comprehension,
solving problems with math, situational judgment, and active listening through a web-based
assessment tool (National Work Readiness Council, (2014)). Although not directly linked to
digital badges, the work readiness credential is an example of how new forms of credentialing
are emerging in response to this skills gap.
In addition, some universities are developing competency-based degree programs which
assess student performance on a set of core competencies defined with industry input (Fain,
2014). These competency-based programs can provide more granular information about
graduate’s skill sets than a traditional degree. For example, Northern Arizona University, one
university with a competency-based degree program, has developed a second transcript that their
graduates can share with potential employers. This transcript provides information on the
specific skills that students have mastered within the program such as working in a team,
communicating with a diverse population or formulating and substantiating theses (Fain, 2013).
The emergence of these competency-based degree programs points to a need for greater
transparency about the outcomes of learning experiences as well as the need to better align
educational programs with industry needs. Digital badges may also be useful in these
competency-based programs as a way to signal the specific skills and competencies that students
have developed.
It is not just recent graduates that need to communicate their specific skill sets to
employers, most adults will change jobs or careers multiple times in their lives. The median
tenure for US workers with their current employer was 4.6 years in 2012 according to a report by
the U.S. Bureau of Labor Statistics (Hipple & Sok, 2013). In fact, the education received in
college and conveyed by a degree, may no longer be relevant for job roles attained 5 or even 10

years after graduation. In addition, the rapid evolution of technology has required employees to
participate in ongoing training and development to keep pace with these changes. Lifelong
learning and skill development has become a requirement for most careers, and much of this
lifelong learning is occurring outside a traditional classroom. .
With improvements in technology and Internet connectivity, many new forms of
education and training have emerged. MOOC’s and online training courses offer ways to learn
online, and new types of face-to-face instruction have emerged such as coding boot camps and
intensive short-term classes through organizations like General Assembly. More and more
learning is being done outside a traditional classroom, but individuals lack a clear way to
communicate the learning and skills they have acquired through these channels. The cost of
attending a four-year university and the high unemployment of university graduates has also
raised the question about whether these lower cost educational offerings could replace a
traditional university degree if there was a standard method to credential these learning
experiences.
Background and Need
Badges as a form of recognition have a long history in the military where ribbons and
medals have been used to recognize bravery and achievements (Ostashewski & Reid, 2015). The
Boy Scouts have also used merit badges and embroidered patches to recognize achievements for
over 100 years. More recently, video games have used badges, points, and leaderboards to track
virtual achievements. It is from this intersection of merit badges and video games that digital
badges have emerged as both a virtual representation of achievement and as a motivational
element of gamification.

Erin Knight, one of the creators of the Mozilla Open Badge Infrastructure and a
proponent of the credentialing aspect of digital badges, acknowledged the dual nature of badges
in a 2011 interview:
“There is an element of gamification in all of this in that we’ve all experienced badges or
levels in games, and we know that they can be motivating. That’s important. Badges will
range from smaller motivational badges, to larger certification-type badges, but as people
are designing badge systems, many of the principles of game design do and should
apply.”(Watters, 2011, p. n.p.)
The fact that badges can be both a gamification tactic and a form of credentialing is important to
note because some systems use badges purely for motivation and these badges may not be
appropriate or viable as external credentials.
For example, Foursquare (www.foursquare.com), which launched in 2009, rewarded
users with playful badges and status levels for participation (Kincaid, 2009). Foursquare users
checked in at physical locations such as restaurants or bars via their mobile phones and acquired
badges and “Mayor” status that could be shared on social networks. Foursquare badges have now
been retired while the company focuses their business strategy on search and local
recommendations (Gayomali, 2014). However, Stack Overflow (www.stackoverflow.com),
which launched in 2008, is another popular site that continues to use badges to onboard new
users and encourage activities that benefit the community such as answering forum questions
(Halavais, Kwon, Havener, & Striker, 2013). In these examples digital badges are used
primarily to drive user behavior rather than validate skills or knowledge.
One of the first implementations of digital badges in an online learning site was Khan
Academy which announced the addition of badges in 2010 (Kamens, 2010). Peer 2 Peer

University (P2PU) was another early adopter, offering digital badges in their School of Webcraft
(Einsenberg, 2011). New technical training providers such as Code Academy, Code School, and
Treehouse launched their platforms and badging systems soon after. What is different about the
implementation of digital badges in education is that badges are usually tied to some form of
assessment which gives them more value as a credential and evidence of learning (Hickey,
2012). There are other benefits of digital badges in education such as making learning paths
clear, scaffolding learners towards higher levels of certification, and providing motivation
(Finkelstein & Knight, 2013). However, it is the potential to credential learning and skills that
are not currently credentialed that is particularly interesting and disruptive.
What is different about badges as a form of credential is that they function as a micro-
credential of learning, which is a more granular and descriptive credential than a broader degree
or certification. The Microcredentials Research Group at Arizona State University also point out
that micro-credentials differ from other forms of credentialing because they can be issued by
anyone and also can be more easily verified online than traditional credentials (“Microcredentials
Research Group,” 2014). Although the concept of digital badges as a form of micro-credentialing
is still relatively new, many organizations are now providing these credentials. Badges for Vets
(www.badgesforvets.org), which launched in 2013, is an example of using digital badges to
credential prior learning and experience (Fain, 2012). Veterans enter their military experience
and training on the site and receive skill badges that translate their skills and competencies into
non-military terms. Employers can then search the database by skill set to find qualified
candidates for jobs.
The official launch of the Mozilla OBI in March, 2013 provided a framework and
infrastructure that organizations could use and was a step towards defining a standard format for

digital badges as credentials. As of mid-2013 Mozilla reported that there were already 700
badge issuers and over 75,000 badges issued through the site (Ostashewski & Reid, 2015).
Digital badging platforms such as Cred.ly (www.credly.com) and Achievery
(www.achievery.com) have also launched sites to make the process of creating and issuing
badges easier by providing tools and hosting services for organizations that want to create digital
badging systems.
However, despite this progress, there is still a lack of clear guidance and best practices
for organizations that are interested in developing their own digital badges. Also, because of the
popularity of sites like Foursquare, badges are often thought of purely as a motivational or
gamification tool. There is a lack of awareness about the potential value of digital badges as
credentials, and how these credentials could be used and displayed effectively. Additional
research is needed to understand how digital badges can best be designed and utilized to provide
value to both learners and employers as micro-credentials.
Purpose of the Project
The purpose of this project is to gain practical experience designing and implementing a
public-facing digital badge system and to gain a deeper understanding of how digital badges
should be designed if they are to become viable forms of credentialing. Research on the factors
that influence the value of credentials to both learners and employers as well as evaluation of
existing badging implementations and badging platforms will inform the design of this new
badging system. By documenting this project it may provide a useful artifact for others that wish
to implement digital badges in a learning context. If the project is successful it will also provide
a platform for ongoing research and study.
Project Objectives

One of the objectives of this project is to review recent empirical studies to better
understand:
• What are students’ perceptions of the value of traditional academic credentials versus
alternate forms of credentialing?
• What are employers’ perceptions of traditional and alternate forms of credentials?
• What factors influence the acceptance and value of credentials by employers?
Case studies of digital badge implementations in educational environments will be reviewed
to investigate how others have implemented digital badges in educational environments and if
there are any best practices that could be established in designing a new digital badging system.
Finally, the field project will examine the process of implementing a digital badging system
in a new online learning environment from the initial design and development through
implementation and launch. Initial feedback from users and recommendations for the future will
also be explored.
Definition of Terms
Gamification: The use of game elements such as points, badges, and leaderboards in non-game
activities.
Digital Badge: Educause defines digital badges as “an image or symbol representing the
acquisition of specific knowledge, skills or competencies” (Elkordy, 2012). More broadly, digital
badges are virtual representations of achievements.
Micro-credential: A form of credentialing that recognizes specific skills or competencies.
Digital badges are a type of micro-credential. However, whereas badges must have some sort of
icon or visual symbol, micro-credentials often have an icon, but that is not a required component
(“Microcredentials Research Group,” 2014).

Mozilla Open Badge Infrastructure (OBI): A framework launched in 2013 that enables any
individual or organization to create and issue these micro-credentials while providing a
centralized platform for the metadata about each badge, including the assessment and evidence
supplied by the badge earner (Rosewell, 2012).
Skills Gap: The gap between the skills that employers need and the skills that job seekers and
graduates have.
Competency-based Education (CBE): Learning systems that are based on a clearly defined set
of competencies. In these systems students progress at their own pace as they are able to
demonstrate proficiency. Dr. Robert Mendenhall, the President of Western Governor’s
University, makes the distinction that “the most important characteristic of competency-based
education is that it measures learning rather than time” (Mendenhall, 2012). Technology is often
utilized in CBE to provide personalized learning.
ePortfolio: An electronic portfolio and collection of students’ work which provides evidence of
achievements. These portfolios can be used to showcase accomplishments, share with
prospective employers or document specific learning outcomes in a course (Lorenzo & Ittelson,
2005).
MOOC: A term which stands for Massive Online Open Course. MOOC’s generally have no
limit on enrollment and are open to anyone to participate. MOOC’s are typically structured
similarly to university courses some and are offered in both synchronous and asynchronous
formats (EDUCAUSE Learning Initiative (ELI), 2013).

Summary
Current forms of credentialing do not necessarily meet the needs of employers, job
seekers, and students. There are gaps in the types of learning activities and experiences that are
credentialed and broader credentials, such as degrees, do not convey enough detailed information
about the skills and competencies that students have acquired. In learning systems, digital
badges provide a way to recognize achievements that can be used as a motivational tool and may
also provide value as a credential of learning. These micro-credentials have the potential to fill
some of the gaps in current forms of credentialing. This paper explores the factors that influence
the acceptance and value of credentials by students and employers and may impact the
acceptance of digital badges as a viable form of credentialing. This paper also examines the
development of digital badging systems and how thinking about digital badges, not only as a
form of gamification, but also as a form of credentialing, may impact the design and
implementation of these systems.

Chapter 2
Literature Review
As discussed in the previous chapter, digital badges can play multiple roles in educational
contexts. Attention is often placed on the gamification and motivational aspect of badges, but digital
badges have the potential to recognize and credential learning as well. In fact, for adult learners in
particular, credentials, or the resulting career opportunities that these credentials provide, may
provide more motivation for learning than the gamification aspect of badges (Finkelstein & Knight,
2013).
The focus of this literature review is how digital badges can provide a valuable credential
to both badge earners and employers. What factors determine whether a credential is
meaningful? How do new credentials become accepted and what factors are most important to
both students and employers? How can digital badges move beyond simply a gamification
mechanism to provide value as a form of credentialing as well?
In addition, to prepare for the field project and the design of a digital badge system, the
existing literature was reviewed to identify case studies of digital badge implementations in
educational contexts. The objective of this research was to explore how these systems were
designed including:
• What technology was used to create, issue and manage the badges?
• How was the incentive structure designed in terms of the type and number of
badges offered?
• To what extent do these systems focus on the use of badges for gamification vs.
the use of digital badges as micro-credentials?
• What were the considerations in designing the graphical look and feel of the
badges?

• What feedback or recommendations could be applied to future digital badge
implementations?
Because badges and micro-credentials are a relatively new concept there are few existing
empirical studies on their use and perceptions. Of the handful of studies that do exist, most have
focused on badges for learner motivation or reputation building and engagement within a
community site.
Given the limited number of studies on badges as external credentials, this literature
review also includes studies of other forms of academic credentialing and assessment including
ePortfolios and certification. In particular, studies on certifications within the IT industry are
reviewed because of the relatively recent evolution of certifications in that field. The IT industry
was also chosen because the rapid evolution of technology requires ongoing learning and
training and a potentially higher need for ongoing certification or credentialing. Employer
perceptions of online degrees were also considered given the recent emergence of online
education and the similar perception other new forms of credentialing, such as badges, may
encounter.
Student Perceptions of the Value of Credentials
Before looking at student perceptions of alternate forms of credentialing it is interesting
to look at student perceptions of the value of traditional university degrees. A study by Griffin,
Jones, and Spann (2008) focused specifically on Generation Y students and their motivations for
pursuing an undergraduate degree and found that students could be characterized into two
distinct groups they termed knowledge-seekers and certification-seekers. Students classified as
knowledge-seekers reported more intrinsic motivations for learning while students classified as
certification-seekers reported more extrinsic motivations for learning. A questionnaire was

administered to 167 students at the School of Business at A&M University in Alabama and
included a mix of sophomores, juniors, and seniors born between the years of 1980 and 1988.
Cluster analysis was used to separate the responses to the nine questions in the survey into two
groups. The resulting analysis classified 76 students as knowledge-seekers based on their
responses, and 82 students as certification-seekers. Respondents in the certification-seeker group
agreed with the statements that it was their degrees and not their grades (4.73 on a seven-point
scale) or knowledge (4.54) that would get them a good job after college whereas the respondents
in the knowledge-seeker group agreed less frequently with those statements. Although this study
focused on a narrow demographic of students within a particular region and age range, it does
indicate that some students see the main purpose of education as attaining the credentials desired
by employers.
A study of higher education students in the United Kingdom examined students’
perceptions of their academic credentials and found that students felt a strong need to
differentiate themselves in the job market beyond the credentials bestowed by their academic
degree (Tomlinson, 2008). In this qualitative study Tomlinson conducted semi-structured
interviews with 53 undergraduate students in their final year of study from a range of subject
disciplines. Although students viewed their higher educational credentials as having value in the
labor market, many expressed concerns about the intense competition for a limited supply of job
opportunities and a difficulty in differentiating themselves from the large number of graduates
with similar credentials and degrees. A business major in this study expressed his concern over
the value of the degree this way:
There are so many people with degrees now that it is becoming difficult for
employers to tell them apart…with so many people with degrees it means far less

than it used to, like you are another person who has gone through the conveyor
belt. I’d say you were at advantage having one but it won’t be enough on its own
to get you a job. (Tomlinson, 2008, p. 55)
These students expressed the need to add distinction to their academic credentials (e.g., through
grades, awards and other activities) or to add additional post-graduate credentials in order to
stand out from other candidates in the job market. This study also identified that a main concern
among these students was the ultimate outcome of their studies and how they could market their
skills to employers rather than “learning for its own sake” (Tomlinson, 2008, p. 56).
The results identified by Tomlinson (2008), that students need to find ways to distinguish
themselves beyond their degree, contradict the findings by Griffin et al. (2008) who found that
many students believed the value of their education was in the degree itself, and not in their
grades or other distinctions. However, both studies found that students are concerned about how
to market themselves to employers and are sensitive to the importance employers place on
credentials.
Student Perceptions of Digital Badges and Alternative Credentials. In a recent
quantitative and qualitative study researchers evaluated adult learner attitudes about badges in
two open online courses (Cross, Whitelock, & Galley, 2014a). The Cloudworks platform was
used to manage and award Open Badges in the courses and learners could display these badges
in their Cloudwork’s profile and also add them to their Mozilla backpacks. There were
approximately 220 active participants in each course and, of those, approximately 50% applied
for one or more badges. Researchers evaluated learner attitudes about the badges through end of
course surveys, badge statistics as well as blog, forum and Twitter posts. A total of 88 students
responded to the course surveys between the two courses studied. Overall, the role of the badges
as evidence of learning was reported as an important function by 42% of survey respondents. One

respondent indicated that their institution agreed to recognize the badges as evidence of professional
development activity. However, many reported uncertainty about whether employers would value
these credentials. One respondent expressed concern about the level of achievement the badge
represented saying that if the MOOC was, “going to be one line of my CV then the badges were
perhaps at too granular a level” (Cross et al., 2014a, p. 9). Another respondent indicated that they
were “a little curious to see if there is any reaction to badges at my place of work when I list them as
a part of my continuing education” while another commented, “I don't know if anyone in my
institution will care, but in case they do, I have the badges” (Cross et al., 2014a, p. 10). Despite the
uncertainty around the value of the badges as a credential, 66% of the survey respondents felt that the
badges were a positive addition to the course and helped to guide and motivate study, monitor
progress, and get feedback. Approximately 25% of respondents expressed a negative view of the
badges and did not find them motivating or useful.
In a qualitative study conducted by Rughinis and Matei (2013) students were asked about
their perceptions of the external value of badges earned in their university courses. Semi-
structured interviews were conducted with 26 students and 8 instructors to evaluate their
perceptions of two different badge architecture systems implemented at the University
Politehnica of Bucharest. Researchers observed that some students saw these badges as being
only for fun rather than for any public significance or benefit. Other students did see a wider
significance in the badges they earned, but were unsure how to use these badges on their resume
or in conversations with employers (Rughiniş & Matei, 2013, p. 87). These results reflect the
current early stage of badge implementation and a lack of general information and awareness that
is available about how they can be used. Unfortunately, few details were provided regarding the
substance of the interviews, how the results were analyzed, or about the badge systems

themselves so there is limited ability to analyze or assess other factors that impacted these
results.
Although not specifically tied to badges, Woodley and Sims (2011) studied student
perceptions of ePortfolios and found similar mixed results in terms of students’ perceived value
of this system in the job market. Similar to the Open Badges metadata, ePortfolios offer a way to
display tangible evidence to an employer of the work completed towards a credential. In this
study 60 students in the second year of a 3-year undergraduate program at the University of
Victoria in Australia completed a web-based survey with a mix of quantitative and open-ended
questions. Students were asked about the importance of their ePortfolio in showcasing their
skills to employers and 38% felt that it was “important” or “very important” while 33% rated it
as “not important” or “not at all important” (Woodley & Sims, 2011, p. 169). In the open-ended
comments students were also divided in their views, with 50% of them making positive
comments and 40% making negative comments about these ePortfolios. However, many of the
criticisms that students gave related to the features of the platform used (PebblePad), rather than
the portfolios themselves. Researchers noted that the students who had responded negatively
also tended to have provided fewer assets or evidence in their ePortfolio so may not have used
the platform as extensively as other students. Although not a focus of this study, four
respondents reported that they had already shown their ePortfolio to an employer and three of
those reported receiving positive interest from the employer.
Overall these studies found that students value credentials when they believe that
employers will value these credentials. With new forms of credentialing such as digital badges
or ePortfolios students are unsure about how employers will perceive them and also how best to
display or use these new credentials.

Employer Perceptions of the Value of Credentials
The IT Industry provides an interesting example for studying employer perceptions of the
value of credentials because of the rapid rate of change in this field, the lack of a single, unified
industry standard for IT credentialing, and the diversity of IT worker’s educational backgrounds
(T. Adams & Demaiter, 2008). Furthermore, a 2002 report on IT workers in the United States
found that the majority of IT workers did not obtain a degree in an IT related field (Aspray &
Freeman, 2002). Meanwhile the emergence of new technology such as virtualization, service-
oriented architecture (SOA) and security and compliance of cloud-computing systems requires
new categories of skill certification and training (O’Grady, 2011). Wierschem, Zhang and
Johnston (2010) surveyed IT directors to identify the importance employers place on IT
certifications and found that, although employers perceived these certifications as having value
and providing some information about technical competence, the overall perception of the
importance and accuracy of these certifications was mixed. Of the 141 respondents that
completed this survey, 45% of employers either required or desired that their IT personnel obtain
or possess certifications. The open-ended questions provided additional insight with the most
often cited reason that employers found these certifications valuable was, “…providing a
baseline of technical knowledge”(Wierschem et al., 2010, p. 99). However, when asked to rank
the importance of certification in evaluating an applicant for a full-time position the average
response was fairly neutral at 3.08 on a five-point scale. Respondents also ranked the accuracy
of IT certifications as an overall indicator of competence more neutrally at 2.94 on a five-point
scale. Experience was ranked as more important than certification at a weighting of
approximately 60% to 40%. These findings were limited by the fact that the respondents were

all based in IT departments at universities. A wider sampling that includes respondents from the
private sector would be beneficial to validate these results.
A mixed-methods triangulation study conducted by Heise (2009) also evaluated the
perceived value of technical certifications by employers and then compared those results to
employee perceptions. This study overwhelming showed the perceived importance of these
credentials in the hiring process, despite the perception that these credentials do not necessarily
provide an accurate picture of an applicant’s skills. In this study approximately 150 IT hiring
managers from northeast Indiana were surveyed using an open-ended questionnaire, with a total
of 59 responses. Descriptive and thematic text analysis were used to identify themes from the
employer surveys. From these themes a quantitative survey was designed and administered to
188 students that were near graduation or had graduated from an IT program in the last 5 years
and had sought employment in the IT field. The results showed that both employers (75%) and
employees (82%) believed IT certifications were valuable in the hiring process. The combination
of credentials most preferred by employers (36%) was a degree, job experience, and some
certification. However, despite the importance placed on these certifications, 84% of employers
reported that they had hired IT certified staff that “did not perform technology tasks to their
expectations” (Heise, 2009, p. 42) and 77% of employees said that they did not feel certifications
accurately reflect an applicant’s IT skills. There were numerous discrepancies in this report
between the data cited in the text and the supporting tables, which does raise some concerns
about the validity and accuracy of the results. Also, the geographical location of participants was
limited to Northern Indiana and should be validated with a wider sample.
Mahatanankoon (2007) surveyed IT professionals in the mid west region of the United
Status and found that formal professional development activities such as training and

certification made a significant contribution to the likelihood of promotion. On the other hand,
informal and non-credentialed professional development activities were found to have no
correlation to promotion. A quantitative survey was completed by 91 IT professionals who had
been working in their current companies for an average of 4.8 years. Respondents were asked to
enter the number of hours per month they spent on both formal and informal professional
development activities as well as the number of times they had been promoted by their current
company. Although it is not clear how many of these learning activities included an actual
certification or credential, respondents who reported more hours spent on formal learning
activities reported a higher rate of promotion. The study was limited to IT professionals in the
early stages of their career and did not measure other factors influencing promotion such as work
performance or motivation. It would be difficult to draw any cause and effect conclusions from
this study, but the results do indicate a positive correlation between formal learning activities and
career advancement.
The Impact of Credential Source on Employer Perceptions. Another factor to be
considered regarding the perception of value of a specific certification or credential is the
influence of the issuing organization. A quantitative study by Gleghorn and Gordon (2012)
evaluated the difference in employer perceptions of IT certification based on whether the
certification was vendor-specific or vendor-neutral. Two certifications were compared: Cisco’s
Certified Network Professional (CCNP) certification and the Certified Information Systems
Security Professional (CISSP) certification offered by the Information Systems Security
Consortium (ISC). Information security managers from the North Carolina Chapter of the
Information Systems Audit and Control Association (ISACA) were surveyed and out of the 200
total members, 40 completed the survey. Participants were asked which certifications, if any,

they would like their IT staff to hold. The majority of employers participating (81.3%) favored
the vendor-neutral CISSP certification, while only 18.8% favored the vendor-specific CCNP
certification. Participants were also asked how likely they were to promote staff with vendor-
neutral or vendor-specific certifications and a slight preference was shown for professionals with
vendor-neutral certification. However, this preference for promoting vendor-neutral certified
professionals varied depending on the years of experience of the hiring manager. One limitation
of this study is the small sample size (N=40). Also, this study did not take into account the
differences in the specific skills assessed by these two certifications or the primary technologies
employed by these organizations. These two factors could have a bigger impact on employer
preferences than the branding of the certification itself.
The findings by Gleghorn and Gordon (2012) also contradict the results of the larger
study of employer and employee perceptions conducted by Heise (2009). In this study employers
were asked about the value of “standardized certifications” versus “individual brand
certifications” and the results showed that 55% of employers felt both types were of equal
importance while 24% felt that vendor-specific certifications were more valuable (Heise, 2009,
p. 45).
Although not directly related to certifications, Adams and DeFleur (2006) looked at
employer’s perception of graduates from online degree programs versus candidates with degrees
from traditional universities and found that the institution issuing the degree was an important
factor in employer perceptions. In this study employers were selected randomly across several
major metropolitan cities in the United States with a total of 269 employers participating from
multiple industries. A quantitative survey showed that employers overwhelmingly favored
candidates with a traditional degree (96%) versus one with an online degree (4%). However, an

important finding from the qualitative component of the survey were “numerous comments that
the issue was not whether the degree was earned online or in a traditional sense, but whether the
online courses were taken with an institution with a reputation of quality” (Adams & DeFleur,
2006, p. 41). In other words, for these employers the online degree would be more accepted
depending on where it came from. One limitation of this study is that employers were not asked
to compare degrees from specific institutions, but only the general concept of online or
traditional degrees. Also, this study is now 7 years old and online degrees have continued to
increase in popularity and are now offered by many well-known schools which may have shifted
employer perceptions since that time.
The Impact of the Labor Market on Employer Perceptions. A qualitative study of
hiring managers in Detroit and Seattle compared employer perceptions of IT workers who held
associate degrees to those with bachelor degrees (Van Noy, Jacobs, & Columbia University,
2012). This study found that employer perceptions of these credentials varied between the two
cities and may be impacted by the specific characteristics of each labor market. Hiring managers
in Seattle expressed more negative perceptions of associate degree holders (46%) than hiring
managers in Detroit (26%), indicating that associate degrees signified less academic ability and
initiative compared to bachelor degrees. One reason for the difference in employer perceptions
offered by the researchers is the higher concentration of candidates with bachelor degrees in
Seattle relative to Detroit, although no specific data on the number of degrees was cited (Van
Noy et al., 2012). Hiring managers in Seattle tended to view community colleges as more
vocational and technical, while hiring managers in Detroit viewed them as more transfer-oriented
and a less expensive pathway to higher education. No data was given regarding the number of
employers that were interviewed in this study or how those interviews were analyzed. However,

the results do point to a potential variation in employer perceptions of credentials depending on
the specific factors of each local labor market. Also, while employers in both cities felt that the
completion of degrees was important and indicated positive attributes, they also indicated that
neither degree conveyed all the skills and competencies that they desired in IT staff which could
indicate the potential value of an additional form of credentialing.
The Impact of Employers’ Personal Experience and Beliefs. A more recent study of
employer perceptions of online degrees found that employers’ personal experience with online
learning had a direct impact on their perceptions of the value of online degrees (Fogle & Elliott,
2013). In this study 71 employers from multiple industries and regions of the United States
completed a quantitative survey on their willingness to hire graduates from online universities.
In contrast to the earlier study by Adams and DeFleur (2009), this study showed more
acceptance of online degrees among employers. When asked which candidate they would hire,
50% of employers indicated that they would select the candidate with a traditional, on-campus
degree. The remaining 50% of employers selected candidates with either a hybrid degree (i.e.
50% online and 50% on-campus) or a completely online degree. However, even more
significantly, this study found a strong correlation between the educational background of the
respondent and their liklihood of hiring a candidate with an online degree. Employers with a
tranditional on-campus degree were twice as likely to hire a candidate with an on-campus
degree, while employers with a hybrid or online degree were 14 times as likely to hire candidates
with online degrees. The researchers also examined the correlation between other demographic
factors such as gender, industry, size of firm, age, and census area. Although there was a slight
bias and correlation found against online graduates in smaller firms, none of the other
demographic variables yielded a strong correlation with perceptions of online degrees. However,

one of the limits of this study is the small sample size and the even smaller percentage of
respondents who had an online or hybrid degree within that sample.
A qualitative study of French employers by Bailly (2008) also supports the findings that
an employer’s experience and beliefs play an important role in the perceptions of academic
credentials. In this study, over 20 interviews were conducted with individuals involved in the
recruiting function in the accounting and retail industries. Bailly found that employers’ views
about the origins of employee skills, and whether these skills could be learned or were innate,
varied by industry and influenced employers’ perceptions of the overall value of academic
credentials. The retail employers believed that the most important characteristic of success was
commitment and other innate qualities that could not be learned, while employers in the
accounting firm believed that any necessary skills could be learned and therefore valued
academic credentials more highly. These findings were correlated with statistics from the Centre
d’Études et de Recherches sur les Qualifications (CÉREQ) which showed, at the time of the
interviews, that a higher proportion of employees with formal academic qualifications could be
found in accounting (32%) than in retail (7%). This study is fairly limited in scope in terms of
the sample of employers and the industries that they represent, but it does point to a variation by
industry in the overall perception of the value of academic credentials.
Conclusion
Given the early stages of digital badge implementation, further studies are needed to
assess the potential value of these micro-credentials to students and employers. Studies have
shown that students are concerned with how to market and differentiate themselves to employers
and will pursue credentials they perceive to be valuable to employers. However, digital badges
are still very new and students do not yet know how to display or use them effectively and are

not sure if these credentials will be valued by employers (Cross, Whitelock, & Galley, 2014;
Rughiniş & Matei, 2013). For digital badges to be considered valuable by students more
education is needed about how to use and display these credentials and more information is
needed about the employers who look for or consider these credentials for recruitment.
Employer perceptions of the value certifications and degrees varies and could be
influenced by a variety of factors including their own familiarity with the credentials, the issuing
organization and brand, the specific local labor market characteristics, and even their own
personal beliefs. Studies have shown that employers value alternative credentials such as
certifications, but at the same time are skeptical about the specific skills that these credentials
convey. The Mozilla OBI provides a mechanism to provide transparency about how a digital
badge was earned that may solve this concern about certifications. There is an opportunity for
digital badges to provide value to employers that is not currently provided by degrees and
certification programs if creditability for these micro-credentials can be established. More
awareness and education about digital badges is needed so that employers and learners become
familiar with them, and more time is needed to see how these credentials evolve.
Badge Design & Implementation
Four recent case studies were identified that implement digital badges in a learning
system. Three of these case studies focus on learning systems designed for adult learners and
one case study documents a system designed for both K-12 students and teachers. All four of
these case studies relate to computer science education, . which is a limitation of this research.
One researcher remarked, “engineering education is especially inclined towards using
achievement-type rewards, due to widespread engagement with the gaming culture” (Rughinis,
Foley, Restivo, Uhomoibhi, & Helfert, 2013). It may be that computer education courses are

early adopters of digital badging and which is why more of these examples were found in the
literature.
Case Study One: IP&T EdTec Course. Instructors at Brigham Young University
designed a digital badging system for a 1-credit instructional technology course (course number
286) which teaches secondary education majors technology skills (Randall, Harrison, & West,
2013). Objective. One objective for the digital badge implementation was to motivate students to
do better work on each of their assignments. The second objective was to provide a way for
students to showcase their skills to employers.
Badge Design. Three levels of badging hierarchy were established for this course: the
lowest level for small achievements, a second tier for mastering larger technologies (which also
corresponded to projects), and a final badge level for course-level mastery. The highest-level
badge would be awarded automatically if all project-level badges were completed. In addition,
the authors allowed for three additional levels of badges that could be earned after the course was
completed as part of a teacher’s ongoing professional development. These additional levels
included a strategy level for creating a plan to implement a technology in the classroom, and an
applied-level for showing evidence of successfully implementing this technology. After the
strategy and applied badges are earned the recipient would automatically receive the highest-
level Technology Integration badge.

Figure 1. IP&T EdTec Badge Hierarchy: Badge levels and types.
Recreated from “Giving credit where credit is due: Designing Open Badges for a Technology
Integration course” by Daniel L. Randall, J. Buckley Harrison, and Richard E. West, 2013,
TechTrends, p. 93. Copyright of TechTrends 2013.
Assessment. Projects in this course are evaluated by an instructor or a teaching assistant
and graded against a rubric. Automatic assessment and peer assessment were not used for badge
awards and assessment. Although this assessment method ensures greater quality control, the
authors point out that this assessment method limits scalability of the badging process.

Technology. The technology used for the creation and issuing of badges for this course
was a commercially available solution called the Badge-It-Gadget-Lite plugin for WordPress.
Students can display badges from their course on their WordPress sites through WPBadger
Display or they can use badgewidgethack.org to display their badges on the web. Unlike
traditional credentials, badges can have a set expiration date. Given that the badges in this
course relate to specific technology skills and technology can change rapidly, the authors debated
how long to make the badges valid for. They were concerned that if the badges expired too
rapidly learners would not come back and earn them again so they decided to set the expiration
at 10 years. After that time, significant changes to technology would likely have occurred
making it necessary to update the badges.
Recommendations and Feedback. No data was provided in the case study regarding
feedback collected from learners or employers about the perceptions of the badges or badging
system or any impact on student motivation or learning. The authors point to the need for further
studies regarding the implementation of this system. They also provide a recommendation
around the need for a better badging platform that would allow for the bulk issuing of multiple
types of badges to many users and then would email each student a link to collect all badges for
the course.
Case Study Two: Computer Science Student Network (CS2N). Carnegie Mellon
University’s Computer Science Student Network (CS2N) is a web-based learning system
targeted at K-12 teachers and students for learning about computer science and robotics (Higashi
& Shoop, 2012). Students earn badges by completing activities along multiple curriculum
pathways as well as for continued participation on the site. Teachers earn badges to become
validated instructors of the content and badge issuers.

Objective. The objectives of the CS2N badge implementation were “providing
motivation, articulating curriculum, and serving as lasting indicators of learners’ achievements”
(Higashi & Shoop, 2012, p. 423).
Badge Design. The CS2N badging hierarchy is designed with four levels of badges, and
then aligns to an industry certification at the highest level. At the lowest level, small and
medium badges provide motivation as well as document learner progress. At the next level,
Knowledge Badges align with major content milestones and assess content proficiency. The
fourth level of badges are the Teaching badges and these require human assessment. Once a
teacher has completed a Teaching Badge they can then assess other students in the CS2N system.
A student who completes this human-verified level of achievement receives a “gold” level
badge. The alignment of the badging system to the real-world credentials at the top-most level
was done to provide more incentive for completing the pathways and to support, rather than
replace, credentials that are already defined and valued by stakeholders.
Figure 2. CS2N Badging Hierarchy: Badge types and structure.
Reprinted from “The roles of badges in the Computer Science Student Network” by R. Higashi,

S. Abramovich, R. Shoop, C. Schunn, 2012, GLS 8.0 Conference, Conference Proceedings, p.
424. Copyright of CS2N.org 2012
The graphic design of the badges in the CS2N system provides information on the
relative weight of the qualification depending on the size and shape of the badge. The badge
name, date of issuance and description appear on the badge. More detail, such as the exact
requirements for earning the badge, expiration date and links to the learning pathway, are
available by clicking on the badge.
Figure 3. CS2N Badge Examples: Example of the CS2N badge design
Reprinted from CS2N.org. (n.d.) Badges. Retrieved from: https://www.cs2n.org/teachers/badges
Copyright of CS2N.org
Another key component of the CS2N badging hierarchy and design are the learning
Pathways that are illustrated visually using Badge Maps. The Badge Maps show all of the steps
in the learning path, the outcome of the learning path, as well as the learner’s progress within the
path.
Assessment. The small and medium CS2N badges are awarded based on a computer-
administered exam. Teacher and gold level badges involve a human assessment process.

Technology. No information was provided regarding the underlying technology used to
issue and reward badges. The CS2N website indicates that the badges conform to the Mozilla
OBI framework and users over the age of 13 can share badges to a Mozilla Backpack or sites
such as Facebook and LinkedIn. C22N badges also conform to the Learning Resource Metadata
Initiative (LRMI) metadata that allows them to be discovered by search engines and educational
databases and identifies the educational standards that the badges align with and represent.
Recommendations and Feedback. The authors report that overall learners have enjoyed
the badging component of the learning site, but that this positive feedback has not necessarily
correlated to an increase in learning performance. No information was provided on feedback
from employers or industry stakeholders.
A separate qualitative study was conducted with middle school students using the CS2N
site to study the motivational effects of digital badges on learners with different levels of prior
knowledge (Abramovich, Schunn, & Higashi, 2013). A total of 36 seventh graders and 15 eighth
graders participated in the study and students were split into two groups based on the results of a
pre-test on proportional reasoning skills. The results of this study found that students who had
scored lower on the pre-test, and were less skilled in the content area, were more positively
impacted by the participation badges. No correlation was found between the skill badges and
motivation in the low-performing group. Students who scored higher on the pre-test did show a
correlation between the skill badges and motivation. Students in this group who earned only a
few skill badges experienced a drop in expectations for success, whereas students who earned
many skill badges experienced an increase in expectations for success. No correlation was found
between high-performing students and participation badges.
Based on the research conducted so far the authors recommend considering the abilities

and motivations of learners when choosing the balance between the number of skill and
participation badges to design. They also recommend providing as much detail as possible to
learners about how to earn badges to counteract any negative impact on motivation (Abramovich
et al., 2013).
Case Study Three: TRAKLA2 and Daechschen). Digital Badges were integrated into
a Data Structures and Algorithms course at Aalto University, in Finland. The first
implementation occurred in Spring 2012 and was repeated in 2013 using different technology. A
total of 300 students participated in the course between the two implementations (Haaranen,
Hakulinen, Ihantola, & Korhonen, 2014).
Objective. Digital badges were implemented in this course as a way to encourage and
motivate good study habits.
Badge Design. A total of eight achievement badges could be earned during the course.
Each of the badges related to completing the online course exercises and the badge awards were
based either on time management, attention to detail, or good study habits. For example, the time
management badges related to submitting exercises before the deadline or completing them the
fastest. The attention to detail badges related to the having the least number of submissions to
get the correct answers. (Hakulinen, Auvinen, & Korhonen, 2013).
There was no explicit badge hierarchy, but earning the most difficult badge within a
category would most likely result in earning the other badges in that category. The course
designers chose this nested structure for the awards to make it possible for users to earn all of the
badges available during the course
For the course exercises students were shown a piece of code that they needed to simulate
with input data to show the execution of an algorithm. The input data was chosen randomly so

students could try to solve the exercises multiple times. Each exercise was given a point value
and the total points were reduced by 50% if students submitted them after the deadline. Points
were not reduced for multiple submissions, but the attention to detail badges were tied to
minimizing the number of submissions needed to arrive at the correct answer.
Figure 4. TRAKLA2 Badges: Example of badge types and design.
Reprinted from “Empirical study on the effect of achievement badges in TRAKLA2 online
learning environment” by L. Hakulinen, T. Auvinen, and A. Korhonen, 2013, Learning and
Teaching in Computing and Engineering, p. 49. Copyright of IEEE Computer Society 2013.

Technology. Originally the badging system and course was designed using the
TRAKLA2 open source Learning Management System (LMS). In the second iteration of the
course the badging system was designed using the A+ LMS which is a Software as a Service
(SaaS) platform. This second system was called Daechschen. Daechschen was built using the
Django web-framework and integrated with the A+ LMS and API’s. (Haaranen et al., 2014)
Recommendations and Feedback. The authors report in a separate qualitative study that
feedback collected from students after the course was generally positive, but overall a small
number of students actively collected badges (Hakulinen et al., 2013). The recommendation
from the authors is to provide some type of external motivation or to tie the badges more closely
to the final grade in the course to provide more motivation.
Students were able to see the badges they earned, but there was no mechanism to share
the badges or make them visible to others. The authors also suggest that having a mechanism to
share the badges and make them visible to others would make them more motivating for
students.
Case Study Four: RL Hit List. Instructors at the University Politehnica of Bucharest
used digital badges in a Computer Networks course which is part of the 3rd
year curriculum in
the Computer Science program (Rughinis et al., 2013).
Objective. The goals of implementing the badging system in this course was to publicly
recognize top students in a visible public forum, to create a community of students and faculty
across multiple years and courses, to motivate students to engage with the course and labs, and to
stimulate conversation among students and faculty.

Badge Design. Currently a total of 26 badges can be earned during the course
(https://systems.cs.pub.ro/teaching/courses/rl/hit-list/). Badges are awarded for top scores on the
various exams as well as in-class participation and overall involvement in the course. There is no
explicit badge hierarchy for the awards.
A unique feature of the RL Hit List badges is that they are both digital and physical in
nature. In addition to a virtual badge, students also receive a metallic pin badge that is awarded
ceremoniously during the course.
Figure 5. RL Hit List: Physical Pin Badges
Reprinted from “Badge architectures in Engineering Education: Blueprints and Challenges” by
Răzvan Rughiniș, 2013, CSEDU. 5th International Conference on Computer Supported
Education. Proceedings, p. 5. Copyright of INSTICC Press 2013.
From a design standpoint the authors state that the badges and website were designed to
be minimalist and professional rather than playful to align with the intended audience of a
community of professionals.
The course site contains a leaderboard where badges earned are listed by student ID and
student name as well as the date when each badge was issued.
Assessment. Badges are awarded for top scores on the various exams on the course.
Participation badges are more subjective and are awarded by instructors based on in-class
participation.

Technology. No information was provided regarding the technology used to create and
issue the badges to participants, but the course site was published using WordPress.
Recommendations and Feedback. One challenge cited by the authors has been creating
and maintaining the ritual dimension of awarding the material pins during the courses and
making sure the award experience is consistent.
Another challenge has been the participants’ understanding of the badge architecture and
objectives. Initially it was assumed that instructors and students had a clear understanding of the
objectives, but feedback has shown that most have understood the badges only as a motivational
tool. The recommendation from the author is to provide more detail about the badge architecture
and rationale when designing a badge system so that it is well understood by all stakeholders.
The last recommendation provided by the authors is to think about badges as a
mechanism for driving conversation and to focus on the conversational infrastructure to support
students, instructors and the community having active dialogue about these achievements.
Conclusion
Two of these case studies had an explicit goal of providing learners a way to showcase
their skills and achievements and, not surprisingly, both of these systems provided a way for
learners to share their badges outside of the learning site (Higashi & Shoop, 2012; Randall et al.,
2013). These two case studies also had a more complex badging hierarchy with multiple levels
of awards. The lower-level badges in both of these examples were designed for motivation and
provided more of a gamification element, while the higher-level badges were skill-based and
provided a credential.
The other two case studies had a simple badging structure with no hierarchy and the
objective of these badge systems was more about influencing behavior or providing recognition

than providing a way to showcase individual skills (Haaranen et al., 2014; Rughinis et al., 2013).
Both of these examples placed more emphasis on the gamification component of badges and no
mechanism was provided to share these badges outside of these sites. One of these systems
included a leaderboard as well as physical pins that were given as awards which further
emphasized the gamification element.
One of the challenges of designing digital badges as micro-credentials is the variety of
ways digital badges are being used within learning systems and the perception of badges as
gamification elements rather than credentials. Some digital badges are not designed to be
credentials and recognize much smaller achievement than other badges, which could be
confusing for recipients and employers to understand how these various badges compare or have
value. The total number of badges that could be earned in each of these systems varied and the
number of badges available could also have an impact on their perceived value. More research is
needed to understand how the number of badges that can be earned impacts learning
understanding and value of these badges.
Information on the technology used to create and issue badges was only available for two
of the case studies. In one case a commercially available solution was used and in the other case
the system was developed internally. More information is needed about the infrastructure
requirements to support badging systems. Commercially available badging platforms are making
that process easier, especially for small organizations, but more information is needed to
compare these solutions and understand their limitations and benefits.

Chapter 3
Project Development
Salesforce.com is a technology company established in 2000 that is the leading provider
of cloud-based Customer Relationship Management (CRM) software. In addition to its Software
as a Service (SaaS) products, Salesforce.com also provides a platform for building enterprise
web and mobile applications. One of the challenges facing the company is how to increase the
number of skilled developers that are familiar with the technology to fulfill the demand for
Salesforce-skilled talent within the ecosystem of customers and partners. Salesforce.com has an
existing certification program managed by the training organization and employers see value in
the certifications as a way to qualify candidates. However, the number of people completing
these certifications each year has not kept pace with the demand in the marketplace.
The Developer Marketing team at Salesforce.com is tasked with growing this developer
community and providing learning pathways for developers to increase their skills. The
department maintains an extensive library of free online documentation and workbooks and has
also produced a MOOC in partnership with Udacity. Although these existing resources have
been well received, the department would like to increase the percentage of users who are
completing these self-paced training resources and continuing to learn and deepen their
knowledge of the platform.
The idea for a new learning platform started with the concept of providing assessments or
challenges that users could complete and then to have those challenges automatically scored or
validated using the Salesforce.com platform API’s. The goal was to provide a way for users to
test their knowledge and to better identify what they need to learn as well as to make the learning
experience fun and interactive.

The other goal of this project was to provide better learning paths or guides through the
content so that users knew what the next step in their learning should be. Although there is a
substantial amount of documentation and training content already available, users have lacked
clear direction on how to navigate through that content.
Initially digital badges were proposed primarily as a way to motivate users and make the
experience fun. I was invited to participate in this project and to contribute to the design of the
incentive and award system because of my interest in digital badging. My hope was to
encourage to the team to consider these digital badges as potential micro-credentials as well as
motivational components of the new system.
Background of the Project
This project officially kicked off in July 2014 with the goal of rolling out a beta version
to customers by mid-October 2014. The project was code-named Trailhead and a small core
team was assembled to help prioritize the features and functionality that could be built for this
initial release.
At Salesforce.com all employees and projects start by creating a document called the
V2MOM which stands for vision, values, methods, obstacles and measures (Benioff & Adler,
2009). The first task for the Trailhead core team was to create a V2MOM and come to an
agreement about the overall vision for this project. After a few rounds of meetings and
discussions the vision was defined as:
“Grow the number of skilled Force.com declarative and programmatic developers and
increase feature adoption by radically redefining the learning experience; create
community-driven, engaging, contextual, scalable, assessment-driven content that
motivates learning.” (Salesforce.com Powerpoint presentation, July 2014)

The goals for the Trailhead project included:
• Make learning paths visible, personal, and easily navigable
• Break learning down into bite-size modules
• Make individual learning visible by recognizing skills and competencies
• Motivate learning and make it fun and engaging
• Drive awareness and adoption of new features
• Provide analytics to measure feature-level adoption
• Provide a platform for all learning content that the department produces
The metrics of success to be measured included:
• number of learning path page views
• number of users who completed at least one assessment
• total number of assessments completed
Project Components
A project leader was assigned to lead the core team, Sandeep Bhanot, and the project was
divided into multiple work streams with a lead assigned to each of these streams. The work
streams for the Trailhead project included:
• Design & Implementation (Jacob Lehrbaum): Creating the UI and interface as well as
building the features and functionality of the learning platform.
• Assessments (Samantha Ready): Creating the assessment questions and mechanism that
would test user’s knowledge of each topic.
• Learning Content (Kim Shain): Updating existing documentation and learning content
to match the Trailhead format as well as leading the content creation of new lessons.

• Incentives & Learning Paths (Lisa Tenorio): Identifying the components of each
learning path, determining how the content would be divided, and designing the
gamification and badging system for the platform.
• Marketing (Lauren Grau): Defining the nomenclature and branding for the platform,
creating the graphic design for the badges and incentives, creating introductory videos
and guides, and executing all marketing and promotion of the learning platform.
Design & Implementation. One of the initial decisions for the team was whether this
learning platform should be built in-house or if we would use commercially available technology
to deliver it. Given the short timeframe and the complexity of the integration with the
documentation content, the Developer website, and the assessment mechanism, the decision was
made to build the Trailhead platform in-house.
There are a variety of hosted solutions available for creating, issuing and managing
digital badges (see Appendix 1). However, these solutions are limited in terms of the systems
that they integrate with. Cred.ly is one of the more popular digital badging platforms and does
integrate with the Salesforce CRM, but not with the Force.com developer platform which was a
requirement for the assessment mechanism and for displaying the badges within the developer
community site. It was also a requirement to integrate this platform with our in-house content
management tools so integration with any commercially available solution would have required
additional development work and potentially taken longer.
One advantage of these commercially available badging solutions is that most of them
integrate with the Mozilla Open Badge Infrastructure and enable badges to be shared through the
Mozilla Open Backpack. Since we were building Trailhead in-house the team needed to decide
if we would integrate the Mozilla OBI for the Trailhead badges. Although not difficult, this

would require some additional development work. Since it was unclear whether the Trailhead
badges would be valuable to users and employers as micro-credentials, it was decided not to
implement this for the initial release and to wait and see if this was something we should
implement for a future release.
Learning Paths. As mentioned earlier, one of the key reasons for developing the new
learning platform was to make clear learning paths for users to follow. From a badging design
standpoint it was also important to start with the overall structure of the learning paths in order to
define the content structure, learning objectives, assessments and rewards.
One of the questions about the learning paths was whether those paths should be defined
by specific tasks, by user roles or by levels. Another question was whether these paths should be
independent or build on each other.
As a starting point for the learning paths a mind map drawing of everything a user should
know about the Salesforce platform was created with contributions from the technical members
of the team. From the mind map, groupings were created to help structure the content by feature
area and topic and then that information was put into a spreadsheet and turned into a content
outline that could then be mapped against our existing content to see where we were missing
content.
After a few iterations the content was structured into three levels. One of the biggest
debates during the project was what to call each of these content levels and various names were
proposed such as lessons, topics, units, paths and modules. Eventually it was decided the three
content types would be: trail, module and unit. Each module represented a feature area and
would contain 3-5 individual units. Each unit would have an associated challenge or assessment.

Three trails were identified for the first release and each of these trails each had five to seven
content modules.
Incentive Framework. Gamification is a popular tactic to motivate user behavior and
new learning platforms such as Khan Academy and Treehouse have implemented badges and
points to help motivate learners to complete courses. One of the objectives for Trailhead was to
motivate more users to complete courses and gamification was seen as a specific tactic to employ
to make learning fun and achieve a higher completion rate. Although the initial goal of the
Trailhead badges and incentives was primarily gamification, as the project evolved the potential
value of badges as a way to make learning visible and to provide clear pathways for learning was
also recognized. It was agreed that the gamification aspect of the badges would be a lesser
priority to these other goals, although still an important part of the system.
One of the early tasks of the projects was to define how many levels the reward system
would have and how those levels would work. In addition to the case studies found in the
literature, other learning sites which utilize digital badges to teach technology were reviewed
including Khan Academy, Code Academy, Code School and Treehouse. Of these sites, Khan
Academy was selected as the best model from a badging and incentive standpoint. The elements
of the Khan Academy incentive system that seemed most relative to Trailhead included:
• An award system that combines points and badges
• Participation, community, and skill-based badges
• Clear learning paths and rapid feedback via automated assessment
• Mastery challenges that enable users to skip over content they may already know
One concern for the design of the Trailhead badges was that they would have enough
meaning and value for users. An initial question was whether badges should be awarded for

completing an individual lesson or if they should they be reserved for completing a more
substantial learning goal such as a module of learning content (usually 3-5 lessons) or even a
broader functional area. At the highest level the learning content for Trailhead was organized
into Trails and each Trail included several modules. Another question was if there should be a
separate type of award for completing an entire Trail (equivalent to a course) or if there should
be any award at all for that level.
The initial badge system proposal for Trailhead included three different types of awards:
points, badges & patches. Points would be the lowest-level of award provided for motivation
and documenting progress, rather than credentialing. The goal of using points at the lowest level
rather than badges was to reduce the total number of badges and to reserve badges for more
meaningful accomplishments. Points could be earned for completing individual lessons and
potentially other types of activities in the future such as community participation or attending
events. Badges would be awarded for completing a challenge at the module level, or for
completing all of the individual unit challenges in a module. There would also be the
opportunity to award special badges in specific feature areas, such as badges for different levels
of expertise within a feature area or for community engagement and participation. One concern
was making sure that if we did award badges for community-based achievements that they would
be clearly distinguishable from the skill-based based badges from a design standpoint. Finally,
the initial proposal included a final level of award which was differentiated from the badges by
calling it a patch. That final level would be awarded automatically for completing all modules in
a particular Trail.

Figure 6. Trailhead Incentive System: Initial design proposal.
Due to the time constraints for the initial release it was not possible to create a challenge
or assessment at the module level. So, the decision was made to have badges earned only
through the completion of each lesson-level challenge rather than have a separate module-level
challenge. This simplified the design, but meant that there was no way for users to skip directly
to a master challenge rather than complete all units. There was some resistance from the internal
stakeholders to include both points and badges because of the desire to keep this system as
simple as possible. However, in the end it was decided to keep points and badges and eliminate
the top-level award instead.

Figure 7. Final Trailhead Incentive System: Final design of the badging and award system for
the Trailhead project.
One open question that was not addressed in this initial badge design is how the
Trailhead badging hierarchy would relate to the existing Salesforce certifications. Salesforce
currently offers seven types of certification that are assessed through an online exam proctored
by a third party. The content that is currently available in the Trailhead system does not yet
cover all of the topics covered by Developer or Administrator certification exams. However, as
more content is produced it is possible that users could acquire the skills to prepare for these
exams through the Trailhead system.
The final challenge in designing the incentive system was determining how many points
to award for each activity. Should all lessons be worth a different amount of points depending
on the difficulty of the topic? Or, should each lesson be worth the same amount? One factor that
needed to be taken into account was that not every lesson had an assessment. Some lessons were
planned to be evaluated via a multiple choice assessment rather than through direct product
application. However, the multiple choice test capability would not be available for the initial

release. Therefore, many lessons in the initial release could be marked as complete simply by
clicking a button without any type of assessment. These lessons should most likely be worth
fewer points then those that required users to implement something directly in the product.
Once again other learning platforms were reviewed to evaluate how their point systems
were structured and the number of points these systems awarded for a similar achievement. In
reviewing these systems there appeared to be two main point structures: those based on
denominations of 100 and those based on denominations of 10.
Table 1. Points Systems in Learning Platforms. Points given to each lesson in popular online
learning platforms.
Platform
Points
per
Unit/Lesson

Khan Academy 300
Code School 250
Treehouse 30
Code Academy 7- 8 (1 pt. per exercise)
For the Trailhead point system there needed to be enough differentiation between the
awards for completing the challenges vs. the multiple choice assessments or other participation
awards that may be implemented in the future. It was decided to use a point system based on
denominations of 100 to create the most flexibility between different award levels. There was
also a feeling that enabling users to gain points more quickly would be more motivating.
Table 2. Trailhead Point System. Final proposal for Trailhead point incentive system.
Task Points
Completing a Unit (No Challenge) 100

Completing a Unit (With Challenge) 300
Completing all Units in a Module 200
Completing all Units in a Trail 1000
Community Badges (TBD) Points
Attending a workshop 100
Attending a webinar 100
Answering a forum question 50
Badge Design. One of the advantages of most digital badging platforms is that they have
templates to facilitate the graphic design of badges. However, since the Trailhead badging
system was developed in-house we also needed to design the look and feel of the badges. The
Creative team within Developer Marketing was responsible for creating the badge design. One
requirement for the design was to differentiate the badges that required programming skills from
the point-and-click badges with some type of visual treatment. The final design concept was to
use circular badges with a simple icon. The icons were chosen to correspond as much as possible
to the feature itself. The programmatic badges have a thin border on the perimeter of the circle,
while the point and click badges do not.

Figure 8. Trailhead Profile and Badge Design. The Trailhead badges and user profile page
where badges are stored.
Some learning platforms use playful names for badges and originally the concept for the
Trailhead badges was that they would be more fun and playful with names like “Apex Ninja”.
However, because of the interest in seeing how these badges might be used as micro-credentials
it was a conscious decision to keep the badge names straightforward. Each badge name
represents an area or feature of the platform. The names need to be kept fairly short to fit in the
space allotted in the user interface and every badge name must be unique. A total of 11 badges
were planned for the initial release, but a longer list, which also contained future badges to be
issued, was reviewed and edited to make sure that the names were all consistent, short and
unique.
Conclusion
Although the main objective for Trailhead badges was not the micro-credentialing aspect,
the badge design was influenced by the possibility that these badges could serve as credentials in

the future. The look and feel of the badges was kept simple and professional, rather than playful.
The naming of the badges and iconography was chosen to convey as concisely as possible what
skills the achievements conveyed. Badges as a reward were also reserved for skill-based
achievements of a more significant size and to limit the number of badges overall. Currently
badges can be shared within the Salesforce Developer Community site and via social media. The
badges do not conform to the Mozilla OBI which would enable greater sharing and transfer of
the badges across the web, but it is not clear if that is something that users would value or use.
Many employers and job seekers use the Salesforce Developer Community to find opportunities
or talent so if the badges are seen as credentials it may be enough to have them displayed within
this site.

Chapter 4
Project Outcome
The Trailhead learning platform launched on October 13, 2014 and is accessible via the
Salesforce website at: https://developer.salesforce.com/trailhead. The platform launched with a
total of 3 Trails, 11 Modules and 41 Units.
So far very little marketing of the system has been done, other than some announcements
at Dreamforce and some posts via social media. However, many users have already started using
system over the last month.
Table 3. Trailhead Usage Metrics. Use of Trailhead system as of November 28, 2014.
Usage Metrics Total
Users who have started at least one unit 1,176
Users who have completed at least one unit 1,116
Users who completed at least one module 635
Users who completed all 11 modules 34
Feedback so far has been overwhelmingly positive. Many users have written blog posts
or tweeted about their experience using the platform.
Table 4. Trailhead Feedback Metrics. Social media posts since launch.
Feedback Metrics Total
Tweets 1,832
Facebook posts 80
Blog posts written 17

Most positive feedback has been about the assessments and about the gamification element of
earning points or badges. One of the bloggers who has written about Trailhead commented about
the competitive nature of developers in general:
“Challenges and automated checking are going to be key factors in the success of this
program. The Salesforce developer community are a competitive bunch, and anything
involving badges and points gets a lot of interest.” (Bowden, 2014)
Another blogger commented on the satisfaction of getting a reward when you get the right
answer to a challenge:
“First, I love the challenges. They’re not too hard, but they’re not a step-by-step, so I had
to really read what they were asking me to do. There is something super satisfying about
pushing the button to check your code and getting a reward when you get it right.”
(Jameson, 2014)
So far, no users have commented on the potential value of the badges as a credential or
way to demonstrate skills to potential employers. Users seem motivated to collect the badges
without really knowing if they will have any future value other than for bragging rights. Many
users who have earned badges have posted screen shots of the badges they’ve earned to social
media to share their accomplishments.
Issues & Challenges
One of the challenges of this project was the short time-frame to plan and execute.
Because of the lead time needed to develop the infrastructure and content, there was very little
time to research the overall badge design and system before critical decisions needed to be made.
Ideally it would have been beneficial to spend more time researching gamification system design
specifically to understand best practices for creating gamified learning experiences.

Some questions have come up from users regarding how Trailhead badges relate to
Salesforce certifications. This was a question that was not tackled in the initial release, but will
need to be thought through for the future. Having badges map to higher-level certifications may
also give the badges more value and credibility as micro-credentials.
Future Development
Currently the Trailhead team is working on adding additional features and functionality to
the system as well as developing new content.
One of the main features being worked on is the addition of computer-based multiple choice
assessments. Not all units can be or need to be assessed within a Salesforce developer
environment. Currently these units have no assessment and points are awarded simply for
confirming completion. For each of these units, three to five multiple choice questions will be
added depending on the number of learning objectives for the unit. All questions must be
answered correctly to complete the unit, but users can attempt the assessment multiple times.
Another feature being worked on is to add a leaderboard to the system so information about
the top scoring users is visible to all users. The goal is to provide more motivation to complete
units and modules by providing this element of competition to the site. Users have indicated that
they would like to see this feature with one saying, “a leaderboard would be cool…something
that shows the top scorers for the last month, or biggest increase in points in the last
week…would give everyone a chance to see their name up in lights” (Bowden, 2014). In
addition to the leaderboard, a plan to provide physical rewards such as embroidered patches or t-
shirts is also being discussed.
A new type of content, called projects, is also being added to the site. Projects are more like
tutorials and will take a user through the steps of building something from start to finish, but will

not provide conceptual information or in-depth content on a specific feature area. A proposal for
how projects will fit into the overall content map and incentive framework is being reviewed.
The current proposal is that projects would reside at the module level and users will receive a
badge for completing an entire project.
Figure 9. Proposal for integrating projects into incentive framework. The updated Trailhead
incentive framework.
Because projects give users more of a big picture view of how features work together rather than
in-depth knowledge in a specific area, the proposal is to make the badge design for projects
visually different from the other skill badges. Another idea under consideration is to make the
badge names for the projects more fun and playful since they are not tied to specific features.

FINAL_Tenorio_masters_thesis

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (17)

Semelhante a FINAL_Tenorio_masters_thesis

Semelhante a FINAL_Tenorio_masters_thesis (20)

FINAL_Tenorio_masters_thesis