SHCR Review 2015 - Appendix 1 Literature Review

1 | S H C R i m p a c t r e v i e w – A p p e n d i x 1
The School for Health and Care Radicals:
what impact has it had?
Appendix 1: A review of existing literature
Stella Martorana, CIPD

Contents
Note.........................................................................................................................................................2
Introduction ............................................................................................................................................3
The need for bottom-up and top-down interventions........................................................................3
Social Learning.....................................................................................................................................3
Connectivism.......................................................................................................................................6
Massive Open Online Courses...........................................................................................................10
Traditional evaluation models...........................................................................................................15
References.............................................................................................................................................20
Note
This appendix forms part of the review of the 2015 School for Health and Care Radicals run by NHS
Improving Quality. The review was conducted by the research arm of the CIPD between November
2014 and September 2015. The main report, which presents an overview of the findings, is also
available.

Introduction
In this section, we present a literature review exploring existing thinking and evidence on social
learning and organisational change that underlie the focus, principles and methods of the evaluation,
in particular the impact model.
The need for bottom-up and top-down interventions
Top-down initiatives to change human behaviour and organisational cultures such as imposed
reforms have often failed their goal. As highlighted by Robert Francis, who led the inquiry into the
failings at Mid Staffordshire Foundation Trust, ‘whatever initiatives are started at the top, unless the
clinical soil is fertile, the seeds will inevitably fall to stony ground at trust level’ (Mid Staffordshire NHS
Foundation Trust Inquiry 2011).
One of the major weaknesses of top-down initiatives relies in the underpinning behaviourist
paradigm, which describes human behaviour and learning process in terms of stimulus and response.
On the other hand, programmes designed to solely influence the behaviour focusing on the individual
dimension can struggle with the application of learning outcomes to real-life contexts. In fact,
situational factors do play a role in the display of new behaviours – for instance, poor organisational
systems can sabotage any change programme (Dixon Woods et al. 2013).
Consequently, a balanced yet challenging approach of top-down and community-based interventions
encompassing governance, social and individual levels increases the odds of impacting and changing
the targeted set of behaviours. The complexity of social cognition requires an equally complex and
inclusive change programme with solid theoretical basis. The simple causal relationship between
stimulus (the reform) and response (change) needs replacing with more sophisticated models.
Social Learning
Albert Bandura’s (1971) social learning theory, is one of the most influential in this area, outlining
crucial processes involved in any learning experience such as observation, cognitive processes and
self-regulative influences. Further, it is also a social cognitive theory for organisational management
(Wood & Bandura, 1989) that identifies new levers for change such as social interaction.
Firstly, observation is a powerful type of learning. There is no need to experiment directly all possible
actions and their consequences, because we use other people’s experiences to modulate our
behaviour or emotional response. This concept links to the one of tacit knowledge (Eraut, 2000),
which will be discussed further in the section on connectivism.
Secondly, any external event is filtered through cognitive processes. Attention and retention
processes are essential to ensure memory of reinforcements such as reward and punishment, which
inform action. Models are then transformed into symbolic codes acting as clues in people’s mind.
According to this theory even symbolic codes in our mind can be manipulated and changed. This
happens when people are involved in rich discussion of emotional values where new definition of
concepts is promoted (Bandura, 1977).
For instance, the first webinar of the SHCR explains the difference between a rebel and a trouble
maker to help individuals channel and productively use their energy. Emotion-arousing words are
used to facilitate the conditioning and the emotional connection. Adopting a concept based on a new
meaning and emotional response precedes any change in the status quo.
Lastly, people are able to create self-regulative influences, which are self-reflective and self-reactive
systems mediating the effect of contextual factors. For instance, internal standards of behaviour or

people’s belief in their efficacy are self-regulated mechanisms informing decisions and actions
(Bandura, 1991).
In the following sections we discuss important themes emerging from Bandura’s theory and the
theoretical contributions of other influential authors.
Practice in a social environment
Learning requires memory of reinforcements and new content, therefore retention processes play a
crucial role. They are enhanced by mental or actual practice, which increases the odds of new
behaviour display (Bandura, 1971).
Consequently, behaviour is learned before it is performed. Also Marsick and Watkins’ theory of social
learning (2001) argues that learning occurs unconsciously, as a by-product of social activity.
The importance of practice is stressed by Rogers’ (2013) diffusion of innovation theory. According to
him people are more likely to adopt new behaviours if they can experiment with them. Students of
the SHCR have the opportunity to discuss what they learnt with peers and embed it into practice.
Those who engage on social media such as Facebook share ideas on how to be effective change
agents and become more familiar with the content of the modules, in other words they practice new
learnings in a protected environment. The comments below, posted on the Facebook group of the
school exemplify the concept.
I'd say the networks and social interaction are invaluable. They provide a safe space for us as a group
to reflect on, help one another with, and work through the material that we heard presented on
Friday. I would, therefore, say that, for me, the effectiveness of the two learning modes are
indissociable and complementary.
The level of engagement with the school varies widely, therefore the impact model devised by CIPD
(see Appendix 2) includes this aspect among the assumptions.
Diversity
Bandura (1977) points out that innovative behaviour emerges solely through modelling, therefore
diversity is a pre-requisite. In this context innovative behaviour is expected from ‘radical agents’, who
are supposed to challenge the status quo. Also Engestrom (2001), author of the Expanding Learning
Theory, points out that diversity and boundary crossing is essential in the learning process.
This means that as long as conformity is the organisational rule, individuals won’t be able to pursue
radical change; the environment will lack of new and diverse models from which individuals can learn.
According to Deloitte (2013) diversity needs to be valued and pursued: “leaders and organizations
must let go of the idea that there is one right way and instead focus on creating a learning culture
where people feel accepted, are comfortable contributing ideas, and actively seek to learn from each
other” (ibidem).
I think the conversation over the weekend has helped to embed the ideas, with all these great
additional viewpoints layered on and offering a breadth of perspectives.
The CIPD impact model for the school includes outcomes such as ‘inject new ideas’, ‘knowledge of
change methods’ which imply exposition to different views and ideas.
Contextual barriers and facilitators
According to Bandura (1977) both learning and its translation into practice depend massively on the
presence of contextual barriers or positive reinforcements. He moves even further by saying that fear

and defensive behaviour can be extinguished by observing others engagement in feared activities
without any adverse consequences’ (1971). Here the importance of the contextual response is
emphasized.
Along the same lines, an influential training evaluation model formulated by Kirkpatrick and
Kirkpatrick (2009) developed the concept of organisational drivers: “action and processes that are
carried out by others that are designed and implemented to reinforce the participants to perform
their critical behaviours” (2009:7). In essence, they enable the translation of learning outcomes into
new behaviours.
The weakness of these theories lies on the incapacity to explain why adverse consequences do not
always discourage individuals from performing specific actions. For instance, new social movements
often survive sanctions and oppositions. Social identity theory (Tajfel, 1982) suggests that situational
consequences are counter-balanced by intra-group processes acting like a glue. This theoretical
framework also informs the impact model devised by CIPD, where intermediate outcomes such as
‘support and validation through network of peers’ are identified.
In the case of the NHS, many individuals can find themselves in organisational contexts disseminated
with barriers. The manipulation of their group membership can be seen as an application of social
identity theory. Belonging to a group of ‘change radicals’, which is composed not necessarily by their
colleagues but by people working in different companies and countries sharing the same goal,
provides them with a new social identity besides group values and shared norms that will support and
sustain their behaviour.
Self-regulated influences
To overcome the limits of explanations based on situational factors only, which leads to a weather
vane model of individual, Bandura introduces the concept of self-regulated influences (1971). Self-
regulated behaviour refers to internal standards that enable us to act and judge, for instance. These
are acquired through previous reinforcement processes. An example is self-efficacy, which is defined
as ‘people's beliefs about their capabilities to produce designated levels of performance that exercise
influence over events that affect their lives. Self-efficacy beliefs determine how people feel, think,
motivate themselves and behave’ (Bandura, 1991).
There are four determinants of people’s beliefs about their self-efficacy:
Master experience. Previous successful experiences increase self-efficacy perceptions
Vicarious experience. When people see individuals similar to themselves making efforts to
achieve select objectives and finally succeed, this will positively affect confidence in their
abilities
Social persuasion. This is a verbal empowerment exercise, which consists of instilling high
levels of self-efficacy. This motivates and sustains the performance of individuals, who try
hard to succeed.
Physiological status. Emotional and physiological reactions such as anxiety or fatigue are used
to infer the ability to carry out specific tasks, such as speaking in front of an audience
(Bandura, 1977).
The emphasis on beliefs and perceptions offers the key to interpret the model. The actual difficulty of
a task is not in discussion, yet the perception of difficulty matters. In addition, self-efficacy is a

situational concept, because people’s perceptions of their abilities to accomplish tasks relates to
specific contexts.
Conclusions on social learning
Bandura’s theory sheds light on the complexities of learning and emphasises the cognitive, interactive
and social aspects of it. Undoubtedly, the instruction model where content is transferred from one
person to another is surpassed; the behaviourist theory according to which learning derives merely
from the presence of rewards or punishment in the environment, is de-emphasised. Important
concepts and themes such as diversity, social interaction, and self-efficacy emerge from his
theoretical framework.
Many other authors such as Vygotsky (1962) proceeded along similar lines. He suggested that
interactions between learners and their peers enable learning and that culture also has a role in
knowledge construction. He devised a model of a complex act as the result of subject, object and
mediating artefact, in other words in this model the individual cannot be separated from his cultural
context.
Similar assumptions are the core of Lave and Wenger’s situational learning model (1991), where
learning cannot be disentangled from the context in which it happens but is actually the result of
interactions between individuals belonging to the same community. According to them, learning
implies knowing how to behave in the community and impacts on people’s identity because they are
rooted in the group membership. However “rather than asking what kind of cognitive processes and
conceptual structures are involved, they ask what kinds of social engagements provide the proper
context for learning to take place” (Hanks, 1991:14).
These insights are reflected in various ways in the design of the SHCR and in the CIPD impact model.
In its assumptions, the model takes social networking to be a central aspect of how participants apply
learning to practice. We also assume that participants are motivated to learn and become a change
agent.
The insights are also reflected in our understanding of a good change agent. In particular, in the
intermediate outcomes of the school, we identify knowledge sharing, encouraging and supporting
others and having support and validation through a network of peers.
In the following sections we will discuss the learning theory informing Massive Open Online Courses
such as the SHCR, which is called ‘Connectivism’.
Connectivism
“Learning is now happening through communities of practice, personal networks, and through
completion of work-related tasks...know-how and know-what is being supplemented with know-
where” (George Siemens, 2012).
With these words Siemens briefly outlined the concept of connectivism, which has been envisaged as
the learning theory for the digital age (Siemens, 2014). This theory explicitly informs Massive Open
Online Courses, but encompasses any social and interactive digital learning model.
According to this theory, knowledge is distributed along a network and learning depends on the
weight of the connections between the units. The final learning outcomes don’t reside in specific
nodes but result from the pattern of links.
Each person can be seen as a node and also technologies are embedded in the net. Consequently,
interaction is the source of learning because it enables changes and achievement of new and ever-

changing states. It is worth clarifying that interaction does not equate with conversations between
individuals, but needs to be conceived in broader terms, encompassing interaction with books and
movies for instance.
The flow of information in and out of the system (individual and organisation) is essential and its
diversity is necessary for learning purposes. The importance of diversity is discussed further in the
section on Social Learning (see above).
As Siemens explains, this theory ‘addresses the challenges that many corporations face in knowledge
management activities. Knowledge that resides in a database needs to be connected with the right
people in the right context in order to be classified as learning’ (Siemens, 2014). Moreover, knowledge
needs to be made explicit and this introduces the concept of ‘tacit knowledge’, to which we now turn.
Tacit knowledge
Tacit knowledge derives from unplanned and non-formal learning and has clear linkage with memory
of specific events (Eraut, 2000). The concept is underpinned by Tulving’s theory of memory (in Eraut
2000), which draws a distinction between memory for specific events (episodic) and general
knowledge (semantic). Generally, semantic memory is derived from the episodic one, to the extent
that recalling the actual events sometimes is not possible. The knowledge we have, but are unable to
explain (other than through our recollection of individual episodes of experience) becomes ‘tacit’
(Polanyi, 1966).
For instance, a person maybe socialised into values and norms of an organisation by interacting with
the members of the community and participating in events, without being aware that a learning
process is taking place (Eraut, 2000). This recalls also the peripheral learning process outlined by Lave
& Wenger (1991) which leads to knowledge on how to behave in community of practices.
Kolb’s uses the term experiential learning: “personally experienced events are stored in episodic
memory and, over time, used to construct generalised knowledge structures in semantic memory”
(Kolb, 1984:8 in Eraut, 2000:117).
Explicit learning can lead to implicit learning as well as tacit knowledge can lead to explicit knowledge.
The latter has been conceived as a knowledge management resource (Herschel, Nemati, Steiger,
2001) enabled by storytelling, conversation and dialogue with other individuals. Discursive practices
force individuals to articulate their learning and facilitate reflection upon them. Also Kolb (1984)
recognised that reflection on experiences unlocks explicit knowledge.
In an organisational context “the conversion of tacit knowledge to explicit knowledge is critical
because it is a prerequisite to the knowledge amplification process wherein knowledge becomes part
of an organisations’ knowledge networks” (Herschel, Nemati, Steiger, 2001:107).
In the following section we will introduce the main features and advantages of social technologies in
learning and organisational settings. The School for Health and Change Radicals integrate social media
in the learning experience, therefore the evaluation included qualitative analysis of comments posted
on Facebook and also data on twitter and Facebook users.
Social technologies in organisational settings
Interaction on social media platform is wide spread. In the UK, over eight in ten of adults go online
and over two-thirds of those have a current social networking site profile (Ofcom, 2014). The most
popular platforms are Facebook, Twitter, Youtube and Linkedin, which are used for personal purpose
in 76% of the cases (CIPD, 2013). Few people use them for work purpose (26.6%) but recently
organisations have begun to exploit their potential.

The literature on their uses in organisational settings has flourished in recent years and one of the
main qualitative studies has been carried out by CIPD in 2013. Their in-depth analysis of case study
organisations led to the identification of important benefits. Below we present the ones which link
more closely with the SHCR experience.
Shared purpose. Company values and mission can be embedded in online messages promoting
engagement and identification with the organisation.
Employee voice. This refers to open and dynamic online communication where potentially all
individuals can express their views and engage in open conversation.
Workplace community. Online conversation involving both personal and work themes can be of
inherent value in itself as they can strengthen the identification with the organisation and promote
better work relationships.
Insight and innovation. Employee insight and experience can be shared with the community and lead
to creative and collective re-elaborations.
Learning and development. Social media enable collective learning through multi-directional dialogue
on interactive intranet, platforms, and forum.
Distributed leadership is possible because all employees have an opportunity to express their voice
and build informal networks across the organisation. Traditionally the literature has focused on the
individual leader (Turnbull, 2011) but the contemporary challenges in the world of work require more
responsive, flexible leadership structures able to cope with changes in workforce, work practices and
markets. Leadership needs to be ‘adaptive’ (Heifetz and Laurie 1997; Heifetz 2009 mentioned by
Turnbull, 2011) and underpinned by systemic assumptions. ‘Distributed leadership does not negate
the requirement of leadership at the top, but requires this leadership as part of a wider leadership
capability, where the social capital of the organisation is harnessed (Northern Leadership Academy;
2007:8). This concept will be discussed further in the following section.
The research carried out by CIPD raised also questions on the relevance of the content shared online.
The case study of Santa Fe Group offers an interesting perspective (CIPD, 2013). The organisation
developed an enterprise social learning network, The Academy Online, after previous unsuccessful
attempts to involve employees in an enterprise social network. Narrowing the focus on learning and
development gave this a focus and facilitated online dialogue, with many employees started sharing
views and making questions on select topics. This turned into a powerful tool to build an
organisational culture in a company with geographical dispersion (ibidem).
In the case of the SHCR, Tweetchats and Facebook group are focused on topics discussed during the
weekly webinars facilitating online conversation and involvement. Personal topics still surface in the
forms of personal anecdotes, requests for new ideas to tackle difficult situations. These are valued
positively, as expressions of outcomes such as ‘encourage and support others’, ‘realise the value of
relationship’ or others depending on the specific input.
The SHCR does not impose the use of social media, people who register to the school are free to
decide if they want to participate in Tweetchats, become members of the Facebook group and
interact with others more generally. This freedom is crucial according to Jane Hart (2011), Founder of
the centre for Learning & Performance Technologies. According to Hart, a top-down approach to
implementing social learning needs replacing with a supportive bottom-up approach. In other words,
individuals already collaborative need to be supported and encouraged instead of forcing everybody
to work and learn collaboratively.

In the following section, we discuss distributed leadership, one of the main benefits of using social
technologies in organisations.
Distributed leadership
“Distributed leadership is not something ‘done’ by an individual ‘to’ others, or a set of individual
actions through which people contribute to a group or organisation… Distributed leadership is a
group activity that works through and within relationships, rather than individual action.” (Bennet et
al., 2003: 3)
The literature on distributed leadership and affine concepts such as ‘shared’ or ‘collective’ leadership
has flourished in the last decades. The concept permeates the work of authors such as Gibb (1954)
who explains how leadership is best conceived as a ‘group quality, as a set of functions which must be
carried out by the group’. His theory was formulated over fifty years ago and Bolden (2011) even
suggests that the concept dates back to centuries ago.
The concept, along with its implications in organisational settings, has been embraced fairly recently
though. The reasons and implications are mainly political, as highlighted below by Day et al. (2009, in
Bolden, 2011:260)
“It is often some form of external pressure that prompts efforts to distribute leadership more broadly,
for example, pressure to improve disappointing school performance, introduction of new policies and
programmes requiring new teaching and learning capacities. Greater ‘distribution of leadership’
outside of those in formally established roles usually depends on quite intentional intervention on the
part of those in formal leadership roles”
The contemporary challenges in the world of work require more responsive, flexible leadership
structures able to cope with changes in workforce, work practices and markets whereas the
traditional literature has focused on the individual leader (Turnbull, 2011).
Nowadays leadership needs to be ‘adaptive’ (Heifetz and Laurie 1997; Heifetz 2009 in Turnbull, 2011)
and underpinned by systemic assumptions.
Bennett et al. (2003 in Bolden, 2008) identified three main features of distributed leadership,
encompassed by most literature on this concept.
• Leadership is the property of a network of people instead of an individual
• The boundaries of leadership are open
• Expertise is distributed along the network rather than located in few individuals only
These three characteristic clearly link to the theoretical framework called ‘Connectivism’, which is
discussed further in chapter 3.
Recognising the role played by the network rather than the individual should not lead to ignore the
role played by vertical leadership though. The key is how vertical and shared leadership can be
utilised to leverage the workers’ capabilities (Pearce, 2004 in Bolden, 2011: 264).
Bolden outpoints that formal leaders are “only the tip of the iceberg” (2008:42) and Harris (2003)
stresses that this concept is useful to overcome the leader/follower divide.

In essence, ‘distributed leadership does not negate the requirement of leadership at the top, but
requires this leadership as part of a wider leadership capability, where the social capital of the
organisation is harnessed’ (Northern Leadership Academy; 2007:8).
Social technologies can be powerful resources to harness the aforementioned social capital, as
highlighted by a CIPD research (2013) which identified distributed leadership as one of their benefits
(see chapter 3).
They overcome corporate barriers and offer employees the opportunity to express their voice and
build informal networks across the organisation. The School for Health and Social Care exploits this
potential and use social media to leverage distributed leadership.
In the following section we will discuss Massive Open Online Courses, a label that can be applied to
the SHCR. Students from all over the world can participate in a social learning experience where they
interact with peers and share ideas and opinions, building informal networks and transforming tacit
knowledge in explicit learning.
Massive Open Online Courses
Massive open online courses (MOOCs) are worldwide and web-based courses open to anyone who
logs on and registers. They result from the social evolution of digital learning, which arose in the
nineties’ with the online publication of videos of lecturers and went mainstream with the OpenWare
Project led by the Massachuttes Institute of Technology.
Their fast-moving nature has eluded clear definitions and evaluation models. They attract different
types of learners, are now delivered in professional contexts as well, and their features continue to
evolve.
The first MOOC was pioneered by researchers George Siemens and Stephen Downes, who led a
course on at the University of Manitoba in 2008. The success, confirmed by 25,000 enrolments,
marked a decisive turning point in the history of digital learning.
Stanford University replicated the model four years later for a course on Artificial Intelligence which
attracted 58,000 students. Predictions of 100% increased rate in registrations followed (Deloitte,
2014) and recently the Bill & Melissa Gates foundation funded the MRI (MOOC Research Initiative) to
stimulate further research on the trend.
The outreach of MOOCs is impressive because there are no admission requirements. Any financial
barrier to education is lifted, promoting the mass dissemination of knowledge. 160,000 students
enrolled to the MOOC on Artificial Intelligence run by Stanford University in 2011 and 308,000
students enrolled to the first six batch of UK MOOCs, developed by Edinburgh University.
We could argue that technology is the biggest barrier to the mass potential of MOOCs for two
reasons. Firstly, the possession of digital devices is a pre-requisite; secondly, poor confidence with
them can discourage individuals from enrolling. However, MOOCs always offer instructions and
demonstrations on how to navigate through them. Moreover, the ‘Net Generation’ (Brown, 2005)
also called ‘Digital Natives’ (Prensky, 2012) are familiar with technology and accustomed to interact
on digital platforms such as Facebook, Twitter and Linkedin.
Consequently, the lack of barriers leads to global and massive participation. The 2015 SHCR attracted
registrants from 36 countries: Argentina, Australia, Austria, Belgium, Brazil, Canada, Denmark,
England, France, Georgia, Germany, Greece, India, Ireland (Republic of), Italy, Netherlands, New

Zealand, Nigeria, Norway, Northern Ireland, Pakistan, Qatar, Russian Federation, Saudi Arabia,
Scotland, Singapore, Slovenia, Spain, Sweden, Switzerland, Thailand, Tunisia, Ukraine, USA and Wales.
Classification of MOOCs
The most basic taxonomy of MOOCs has been developed around the social and interactive dimension
of learning, distinguishing between xMOOCs and cMOOCs (Bates, 2014).
xMOOCs derive from the lecture teaching model. The focus is on content delivery although some
interaction with other students or with the lecturer is possible. Their specific platforms store content
related to the course such as video recordings besides important data on users and their learning
experience such as demographics, attendance, marks in computerised tests and final results.
cMOOCs, such as the SHCR, are underpinned by the connectivist theoretical approach. Downes and
Siemens, creators of this model, stress that learning results from the process of sharing knowledge
between participants (Downes, 2012).
This taxonomy has been useful to develop an evaluation framework for the SHCR, in fact the model
focuses on intermediate outcomes such as ‘realise the value of relationships’, ‘encourage and support
others’, ‘align with a shared mission’, and others which are distant from the results of formal
examinations and test.
Schneider (2013) developed a taxonomy on the general structure and a complimentary one on the
interactive learning environment (ILE).
According to the former, MOOCs need to be categorised on the basis of the following aspects: name,
ID, publisher, platforms, domain, level, target audience, use, pace and accreditation.
It is worth mentioning that ‘accreditation’ introduces one of the most popular debates on MOOCs,
regarding how and for which purpose the learning outcomes need to be assessed and certified. This
will be discussed in the following section.
Engagement with MOOCs
“The person taking the MOOCC is like a person reading a book, playing a game, or taking a trip to the
city. It is impossible to talk about ‘the objective’ of such an activity – some people want to learn
something (and others something else), others are doing it for leisure (and others as part of their
jobs), others to make friends (and others to get away from their friends for a while)…” (Michelle
Barclay)
The evaluation of MOOCs is a challenge to the way education providers have assessed and certified
learning outcomes so far. Traditional evaluation methodologies need re-visiting in the light of new
types of learners, knowledge models and technologies.
Technology is so significant within our lives that researchers labelled new generations on the basis of
their familiarity with digital devices. Prensky (2012) coined the term ‘Digital natives’ to describe a
generation of tech-savvy individuals with a positive attitude to e-learning, accustomed to the speed of
the internet and to multi-task. The ‘Net Generation’ is goal and achievement oriented, with a
preference for active learning and social activities (Brown, 2005).
Also the diversity in learners’ needs and intentions is critical. Some are curious on select topics and
participate for the sake of knowledge itself, others use MOOCs as gap fillers useful in their profession
or want to test and certify their interest in a topic before enrolling to University; others are keen to
experiment innovative e-learning systems.

Phil Hill (2013) identified five student patterns in Coursera-style MOOCs. Using his terminology and
descriptions, the five user profiles are the following.
No-Shows. These students appear to be the largest group of those registering for a Coursera-
style MOOC, where people register but never login to the course while it is active.
Observers. These students login and may read content or browse discussions, but do not take
any form of assessment beyond pop-up quizzes embedded in videos.
Drop-Ins. These are students who perform some activity (watch videos, browse or participate
in discussion forum) for a select topic within the course, but do not attempt to complete the
entire course. Some of these students are focused participants who use MOOCs informally to
find content that help them meet course goals elsewhere.
Passive Participants. These are students who view a course as content to consume. They may
watch videos, take quizzes, read discuss forums, but generally do not engage with the
assignments.
Active Participants. These are the students who fully intend to participate in the MOOC and
take part in discussion forums, the majority of assignments and all quizzes & assessments.
Figure 1. Emerging student pattern in Coursera-style MOOC (from Hill, 2013)

The drop-out rate is massive according to further data mining. The University Of Pennsylvania
Graduate School Of Education (Penn GSE) found that MOOCs have a 4% completion rate, with most
user dropping-out after 1-2 weeks (Perna, Ruby, 2013).
Katy Jordan, PhD student at Open University, collated data on 29 different MOOCs; her findings show
completion rates between 13% and 40% with an average of 7.6%. She also investigated the effect of
two independent variables, assessment type (auto grading vs peer) and course length on the
completion rates (Jordan, 2013). Data have been represented on a visualisation tool, which shows
higher completion rate for short and auto-graded MOOCs, although the statistical significance of this
relationship is unclear.
Fig. 2 Completion rates for MOOCs (Jordan, 2013)

Fig. 3 Completion rates for MOOCs (Jordan, 2013)
According to Andrew Ho, professor in Harvard’s Graduate School of Education, “focusing on
completion rates limits our imagination of what might be possible with MOOCs. A better
criterion for success might be for students to complete more of the course than they thought
they would, or to learn more than they might have expected” (Harvard Gazette, 2014).
In fact, the criticism towards MOOCs lies on comparisons with traditional courses and on the ways
completion rates are calculated.
The completion rate is often calculated by # certificate earners divided by the total # of people who
registered for a course, which is massive, but many applicants know that they won’t complete it since
the beginning (Reich, 2014). In a research carried out on EdX MOOCs, 58% of registrants wanted to
earn a certificate, 25% intended to audit, 14% were unsure and 3% intended to browse. Interestingly,
users with the intention of completing the course were 4.5 times more likely to do so (ibidem).
Lastly, the comparison between those who complete the course with the high number of registrants
should be complemented by comparisons with those who would have accessed the select content
and interacted with their peers in an ILE that overcome geographical location, time commitments,
financial situation and other barriers. Likely, that figure will be extremely low.
Certifications
How and for which purpose the learning outcomes need to be assessed and certified are frequent
questions with multiple answers.
Most providers release certificates attesting the attendance and completion of the course upon
payment of a charge. They are promoted as documents which testify interest in a select topic and can
be used to increase the chances to be accepted in important Universities or to showcase learning
achievements to employers (Coursera, EdX, FutureLearn, 2015).
Here the disruptive potential of MOOCs towards tuition-based degree programmes is unveiled,
although learning providers state to offer alternative credentials.
Below a list of certificates released by three popular MOOC providers to date
COURSERA
Verified certificate. According to Coursera (2015), this is an official, verifiable, shareable document
that can help advance participants’ education and career. Users need to record their typing pattern,
take a photo of themselves and one of their ID. When submitting coursework, the identity needs to
be confirmed by providing matching typing sample and photo.
Specialisation Certificate. This is a group of related courses aimed at the attainment of specific skills
and their application in a capstone project. A Verified certificate need to be earned for each course
and the capstone project is available only after users’ have completed all courses in the specialisation.
EDX
EdX offers honor code certificates of achievement (the only ones where the identity of the
user is not verified), verified certificates of achievement and XSeries certificates of

achievement. The latter are similar to the Specialisation certificates released by Coursera,
because participants need to complete a series of courses making up a series. As publicised
on Edx website, they are a ‘true’ achievement you can use to advance your career or simply
impress yourself. Contrary to Coursera, EdX does not verify the typing style.
FUTURE LEARN
FutureLearn releases statements of attainment upon completion of an exam. These
documents show participant name, name of the course, name and logo of the University,
percentage score achieved. It also outlines the subject areas, the number of hours of study
required per week and clearly states that the final exam was carried out under invigilated
conditions. Exams are delivered by Pearson VUE in phisical centres, where the identity is
checked.
Traditional evaluation models
Kirkpatrick (1959) devised a successful evaluation model for training courses in the fifties,
which identified four levels of learning outcomes and was recently refined and developed
(2009).
1. Reaction reflects the user satisfaction to a select training event
2. Learning refers to the acquirement of specific knowledge, skills, attitudes, commitment
based on their participation in a learning event.
3. Behaviour. Knowledge and skills acquired during the training need to be transferred to the
job
4. Results. ‘To what degree targeted outcomes occur as a result of the training event and
subsequent reinforcement’ (Kirkpatrick, 2009:21)
An additional level investigating the wider implications and effects of learning on the society
is included in the alternative learning model of Kaufman (1995).
The evaluation of learning has often been focused on specific training events (level 1 and 2) therefore
based on metrics such as attendance (level 1) or scores (level 2). According to Kirkpatrick, the most
productive approach to evaluation needs to start from the end, in other words from the definition of
results. Careful planning is a pre-requisite to training evaluation along with the involvement of
business leaders.
Firstly, the top management expectations need to be clarified, in order to define their concept of
success and break it down into measurable indicators. The agreement of business leaders and
evaluators is crucial at this stage, where their specific accountabilities are defined.
Once there is agreement on the desired results, trainers need to involve line managers in the
definition of behaviours that are critical to the achievement of successful results (level 3) but also
define which situational factors (processes, systems, rewards, etc.) will support the new behaviour.
At this stage, it is possible to identify and develop specific training events (level 1&2).

Our evaluation conflicts with this view, where a top-down approach prevails, because individuals have
different expectations of a MOOC, which might not align with top management’s view. The evaluation
needs to remain open to what might emerge, integrating a bottom-up approach to evaluation.
Frameworks for the design and evaluation of MOOCs
Grover et al (2013) devised a framework for the design and evaluation of MOOCs which is intertwined
with the distributed intelligence framework devised by Pea (1993).
“The intelligence revealed through these (cognitive) practices are distributed – across minds, persons,
and the symbolic and physical environments, both natural and artificial. Gregory Bateson remarked
that memory is half in the head and half in the world” (Pea: 1993:47).
There is a linkage with the connectivist theory as well because distribution implies connections and
the transmission model of education is replaced with the generation of distributed knowledge in a
network environment.

Fig. 4 Framework for the design and evaluation of MOOCs (Grover at al 2013:2)
The framework identifies four dimensions across which the intelligence is distributed and require the
participation of all professionals involved in the design and implementation of the digital course,
including technologists, data scientists and instructional designers.
The interactive learning environment includes content, instruction, community and assessment. The
above course elements are designed on the basis of different factors such as underpinning learning
theories, learners’ background and intentions, availability of select platforms and more.
For instance, some MOOCs emphasise the transmission of content (cMOOC) and other ones the
interaction between peers (xMOOCs). EdX, Coursera and many others MOOCs providers break down
video content into short recordings (5-10 minutes) to facilitate the learning process whereas others
hold onto the traditional lecturing model and broadcast long videos.
Also the assessment varies on the basis of users and aim of the course; for instance, courses that will
lead to credits for students will require detailed forms of assessment.
Learner background and intentions captures the variety of needs and goals moving participants. The
variety in intentions, from curiosity to fun or skills enhancement leads to the conclusion that course
completion is not a good measure of learning outcome (Grover et al. 2013). Research described in
previous section of this document also show that intention can be a good predictor of completion
rates.
Technology of infrastructure encompasses all technology aspects from the software to the social
media platforms and the learning analytics engine powering the MOOC. It involves decision on how
data on users and their interactions are collected and analysed.
Finally, the evidence-based improvement is conceived as “meta-MOOC process underpinning design
decisions around the ILE and technology infrastructure. Evidence-based improvement is powered by
data mining and analytics designed to measure the desired course learning outcomes, and
incorporates qualitative evidence from sources like forums and surveys” (Grover et al, 2013:3).
According to this model, data needs to be collected from each dimension and translated into
indicators of learning, engagement and distributed intelligence (ibidem).

The dimensions and elements of this model have been included in our impact model, described in
Appendix 2. For instance, learner background and intentions are assumptions (motivated to learn and
be a change agent; varying specialism and seniority). The interactive learning environment is included
in the activities as well as the technology infrastructure. Analytics regarding the social interaction on
platforms such as Twitter and Facebook were collected as part of the research.
The research team also conducted in-depth follow-up interviews focused on participants’ stories of
their involvement with the school and subsequent activity as change agents (see Appendix 5). This
gave us insight into the potential the school has to contribute to organisational, service and patient
outcomes more widely.
An alternative framework was proposed in 2013 as part of the MOOC Research Initiative (Momentum,
2013). It is the result of an online discussion between George Siemens, founder of the first MOOC and
e-learning experts, and it is also informed by the Effective Lifelong Learning Inventory developed at
the University of Bristol. It identifies nine components, as shown in the graph below.

Implications for the current review of the SHCR
A key implication of these insights for the evaluation of the SHCR is that we need to understand the
learner’s objectives from their own point of view and evaluate the outcome based on what they came
to the learning environment expecting, rather than be limited to the intentions of the course
designers.
Thus, in developing the impact model, we relied on interviews with participants from the 2014
programme as well as discussions with the course designers and SHCR documents. These participant
interviews enabled us to build an understanding, in a bottom-up approach, of what they valued in the
school and how they benefitted from it.
We also used these interviews to test the language that the course designers used in describing the
school. Using cognitive interviewing, we asked participants survey-type questions using phrases such
as ‘change radicals’ and ‘rock the boat while staying in it’ and then asking the interviewees what they
understood by those terms. In this way, we helped to ensure not only that the concepts were
reflective of participants’ experiences, but that the language we used to describe these concepts
were also reflective of participants’ realities.
Another implication is that the in-depth qualitative interviews being undertaken a few months after
the 2015 school will be important to contextualise the changes we are measuring in the pre and post
evaluation surveys.
However, the SHCR is designed to help people develop in reasonably specific ways and it is right we
also assess that as well as participants’ experiences. Thus, the impact model also reflects the design of
the school and the thinking of those delivering it.
Furthermore, the qualitative bottom-up interviews are important to frame a more positivist
evaluation, but do not invalidate it. At a certain point it is entirely valid, once one has developed
theory in an inductive way, to test that theory with a broader population. This underpins our basic
approach of the two-wave survey, immediately before the 2015 school and six months later.
This evaluation approach fits better with the SHCR than it may do with many other MOOCs. Although
its resources remain open for participants to use, the school was run in particular through live
sessions delivered on five consecutive Friday mornings. This introduces a degree of standardisation in
the delivery of the school and how people participated, which makes it appropriate to adopt a
method including a survey at two discrete points in time.

References
BANDURA, A. (1971) Social Learning Theory. New York: General Learning Press.
BANDURA, A. (1977) Self-efficacy: toward a unifying theory of behavioral change. Psychological
Review. Vol 84, No 2. pp191–215.
BANDURA, A. (1991). Social cognitive theory of self-regulation. Organizational Behavior and Human
Decision Processes. Vol 50, No 2. pp248–287.
BATES, T. (2014) What is a MOOC? [Online] October 12th
2014. Available at: Online Learning and
Distance Education Resources http://www.tonybates.ca/2014/10/12/what-is-a-mooc/ [Accessed:
19/02/2015]
BENNETT, N., WISE, C., WOODS, P.A. & HARVEY, J.A. (2003). Distributed Leadership. Nottingham:
National College of School Leadership.
BOLDEN, R. (2008) Distributed leadership. In: Marturano, A. and Gosling, J. (eds) Leadership: the key
concepts. London: Routledge
BOLDEN, R. (2011) Distributed leadership in organizations: A review of theory and research.
International Journal of Management Reviews. Vol 13, No. 3. pp251-269.
BROWN, M. (2005) Learning Spaces. In: OBLINGER, D., OBLINGER, J. (eds). Educating the Net
Generation. [Online] Educause. Available at: Educating the Net Generation
https://net.educause.edu/ir/library/pdf/pub7101l.pdf
[Accessed: 19/02/2015]
CIPD (2013) Social technology Social Business?. London: Chartered Institute of Personnel and
Development [Accessed: 19/02/2015]
COURSERA (2015) Terms of Service. [Online] Available at: https://www.coursera.org/about/terms
[Accessed: 19/02/2015]
DELOITTE (2013) Diversity’s new frontier. Diversity of thought and the future workforce. Available at:
http://dupress.com/articles/diversitys-new-frontier/#end-notes [Accessed: 10/03/2015]
DELOITTE (2014) Massive Open Online Courses (MOOCs). Not disruptive yet, but the future looks
bright. Available at: http://www2.deloitte.com/global/en/pages/technology-media-and-
telecommunications/articles/2014prediction-massive-open-online-courses.html
[Accessed: 10/03/2015]
DIXON-WOODS, M., BAKER, R., CHARLES, K., DAWSON, J., JERZEMBEK, G., MARTIN, G., MCCARTHY, I.,
MCKEE, L., MINION, J., OZIERANSKI, P., WILLARS, J., WILKIE, P. & WEST, M., 2013. Culture and
behaviour in the English National Health Service: overview of lessons from a large multimethod study.
BMJ Quality and Safety. Vol 23, No 2. pp106-115.
DOWNES, S. (2012) Connectivism and Connective Knowledge. [Online] Available at:
http://www.downes.ca/files/books/Connective_Knowledge-19May2012.pdf . [Accessed: 19/02/2015]

EDX (2015) Learning at edX. [Online] Available at: https://www.edx.org [Accessed: 19/02/2015]
ENGESTROM, Y. (2001) Expansive Learning at Work: Toward an activity
theoretical reconceptualization. Journal of Education and Work. Vol 14, No 1. pp133-156
ERAUT, M. (2000) Non-formal learning and tacit knowledge in professional work. Journal of
Educational Psychology. Vol 70, No 1. pp113-136
FUTURELEARN (2015) Workplace Learning. [Online] Available at:
https://www.futurelearn.com/workplace-learning [Accessed: 19/02/2015]
GIBB, C. A. (1954). Leadership. In G. Lindzey (Ed.), Handbook of social psychology. Vol 2. pp877-917.
Reading, MA: Addison-Wesley.
GROVER, S., FRANZ, P., SCHNEIDER, E. and PEA, R. (2013) The MOOC as Distributed Intelligence:
Dimensions of a Framework for the Design and Evaluation of MOOCs. In Proceedings of the 10th
International Conference on Computer Supported Collaborative Learning (Madison, WI, June 16-19).
HANKS, W.F. (1991) Forward to J.Lave and E.Wenger, Situated Learning: Legitimate peripheral
participation. Cambridge: Cambridge University Press
HARRIS, A. (2003) Teacher Leadership as Distributed Leadership: Heresy, Fantasy or Possibility? School
Leadership and Management. Vol 23, No 3. pp313-324.
HART, J. (2011). Social Learning Handbook [Online] Available at:
http://c4lpt.co.uk/resources/social-learning-handbook/social-learning-strategies/ [Accessed:
11/03/2015]
HARVARD GAZETTE (2014) MIT and Harvard release working papers on open online courses. [Online]
Available at: http://newsoffice.mit.edu/2014/mit-and-harvard-release-working-papers-on-open-
online-courses-0121 [Accessed: 19/02/2015]
HERSCHEL, R., NEMATI, H., STEIGER, D. (2001). Tacit to explicit knowledge conversion: knowledge
exchange protocols. Journal of Knowledge Management. Vol 5, No 1. pp107-116
HILL, P. (2013) Emerging Student Patterns in MOOCs: A (Revised) Graphical view. [Online] 10th
March
2010. Available at: E-literate http://mfeldstein.com/emerging-student-patterns-in-moocs-a-revised-
graphical-view/ [Accessed: 19/02/2015]
JORDAN, K. (2013) Synthesising MOOC completion rates [Online] 13th
February 2013.Available at:
MoocMoocher https://moocmoocher.wordpress.com/2013/02/13/synthesising-mooc-completion-
rates/ [Accessed: 19/02/2015]
KAUFMAN, R., KELLER, J., & WATKINS, R. (1995). What works and what doesn’t: Evaluation beyond
Kirkpatrick. Performance and Instruction. Vol 35, No 2. pp8-12. Available at:
http://home.gwu.edu/~rwatkins/articles/whatwork.PDF [Accessed: 19/02/2015]
KIRKPATRICK, D. L. (1959). Techniques for evaluating training programs. Journal of ASTD. Vol 11, No 3.
pp3–9.

KIRKPATRICK, J., KIRKPATRICK, W. (2009). The Kirkpatrick Four Levels: A Fresh Look After 50 Years.
Available at:
http://www.kirkpatrickpartners.com/Portals/0/Resources/Kirkpatrick%20Four%20Levels%20white%2
0paper.pdf [Accessed: 19/02/2015]
KOLB, D. (1984). Experiential learning. Englewood Cliffs, NJ: Prentice Hall
LAVE, J., WENGER, E. (1991) Situated Learning. Legitimate peripheral participation, Cambridge:
University of Cambridge Press.
MARSICK, V. J., & WATKINS, K. E. (2001). Informal and incidental learning. New directions for adult and
continuing education. Vol 2001, No 89. pp25-34.
MID STAFFORDSHIRE NHS FOUNDATION TRUST INQUIRY (2011). ‘Closing submissions’. Available at:
www.midstaffspublicinquiry.com/node/505 [Accessed: 09/03/2015]
MOMENTUM (2013) MOOC Framework. [Online] Available at: Momentum Edthemes
http://momentum.edthemes.org/about-mooc-jam [Accessed: 26/02/2015]
NORTHERN LEADERSHIP ACADEMY (2007). How Does Leadership Make Difference to Organisational
Culture and Effectiveness? – An Overview for the Public Sector. Available at:
http://www.cihm.leeds.ac.uk/document_downloads/new_nla_paper_leadership_and_culture__2_.pd
f [Accessed: 19/02/2015]
OFCOM (2014). Adults’ Media Use and Attitudes Report 2014 – UK [online]. Available at:
http://stakeholders.ofcom.org.uk/market-data-research/other/research-publications/adults/adults-
media-lit-14/ [Accessed: 19/02/2015]
PEA, R. D. (1993). Practices of distributed intelligence and designs for education. In: SALOMON, G.
(eds). Distributed cognitions. New York: Cambridge University Press.
PERNA, L.W., & RUBY, A. (2013). Life cycle of a million MOOC users. MOOC Research Initiative,
Arlington, Texas. Available at:
http://www.gse.upenn.edu/pdf/ahead/perna_ruby_boruch_moocs_dec2013.pdf
POLANYI, M. (1966). The Tacit Dimension. Garden City: NY
PRENSKY, M. (2012) Teaching the right stuff. [Online] 2012. Available at:
http://marcprensky.com/writing/Prensky-TheRightStuff-EdTech-May-Jun2012.pdf. [Accessed:
19/02/2015]
REICH, J. (2014) MOOC completion and retention rates in the context of student intent. [Online]
Available at: Educause Review http://www.educause.edu/ero/article/mooc-completion-and-
retention-context-student-intent [Accessed: 19/02/2015]
ROGERS, E.M. (2013). Diffusion of Innovations. New York: Simon & Schuster International
SCHNEIDER, E. (2013) Welcome to the moocspace: a proposed theory and taxonomy for
massive open online courses. Proceedings of the 1st Workshop on Massive Open Online Courses at
the 16th Annual Conference on Artificial Intelligence in Education (2013). Memphis, TN. Available at:
http://people.csail.mit.edu/zp/moocshop2013/paper_21.pdf [Accessed: 19/02/2015]

SIEMENS, G. (2012). What is the theory that underpins our MOOCs. Available at:
http://www.elearnspace.org/blog/2012/06/03/what-is-the-theory-that-underpins-our-moocs/
[Accessed: 19/02/2015]
SIEMENS, G. (2014) Connectivism: A Learning Theory for the Digital Age [Online] December 12th
2014.
Available at: http://www.elearnspace.org/Articles/connectivism.htm [Accessed: 19/02/2015]
TAJFEL, H. (1982). Social psychology of intergroup relations. Annual Review of Psychology. Vol 33.
pp33-74.
TURNBULL, J. K. (2011). Leadership in Context, Lessons from New Leadership Theory and Current
Leadership Development Practice. London: The King’s Fund
VYGOTSKY, L. S. (1962). Thought and language. Cambridge MA: MIT Press.
WOOD, R. E., & BANDURA, A. (1989). Social cognitive theory of organizational management. Academy
of Management Review. Vol 14, No 3. pp361-384

SHCR Review 2015 - Appendix 1 Literature Review

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a SHCR Review 2015 - Appendix 1 Literature Review

Semelhante a SHCR Review 2015 - Appendix 1 Literature Review (20)

Mais de Horizons NHS

Mais de Horizons NHS (20)

Último

Último (20)

SHCR Review 2015 - Appendix 1 Literature Review