Assessment, technology and learning: who is in the driving seat? Prof Josie Taylor, Director, Institute of Educational Technology, The Open University This presentation examines assessment in the context of open educational resources and informal learning. I examine the concept of assessment 2.0, and the emancipatory effect of new forms of e- assessment that put students in the driving seat.

1. Assessment, technology and learning:
who is in the driving seat?
Prof Josie Taylor
Director, Institute of Educational Technology,
The Open University
Middlesex, 2011

2. The Open University, UK
Higher education needs to
embrace a more open
future. This will entail
changes that are likely to
be profound
We haven’t yet fully
understood what these
changes are, or what the
impact will be on
organisations, staff or
students

3. OpenLearn at The Open University
2006 – William and Flora
Hewlett foundation
provided the OU with
funds to investigate
sharing educational
resources and more
open approaches
www.open.ac.uk/openlearn

4. Open Educational Resources
Our definition of OER:
“The open provision of
educational resources, enabled
by information and
communication technologies,
for consultation, use and
adaptation by a community of
users for non-commercial
purposes.”
www.open.ac.uk/openlearn

5. OpenLearn
 Designed on a model
analogous to the open
source software
movement
 >14 million unique
visitors have used
OpenLearn since 2006
 Gradual build of user
base
Olnet.org

6. Studies by OLNet: Patrick McAndrew
Undertaking analysis of
user behaviour.
The results from one of
these studies (n = 2,011)
highlighted two distinct
clusters of learners:
"volunteer" students
"social" learners
McAndrew, Scanlon & Clow (2010)

7. Volunteer students
Volunteer students sought the content they wanted to learn
from, and they expected to work through it. These learners
were most interested in:
 more content
 tools for self-assessment
 ways to reflect on their individual learning.
McAndrew, Scanlon & Clow (2010)

8. Social Learners
Social learners were less motivated to work through the
content. Rather, they seem to see learning as a way to meet
people with shared interests. These learners were most
interested in:
 communication tools
 advanced features on the website.
McAndrew, Scanlon & Clow (2010)

9. What are these informal learners
trying to do?
 How might they frame their tasks?
 How will they know when they have succeeded? i.e. what
‘counts’ as success?
 What will be the quality of the experience?
 How can we best support them?

10. Process of emancipation for new learners
as confidence builds
Lots of other stops along the way...

11. Learner emancipation & institutional
authority
 For learners:
 Not ‘just’ skill/meta-skill acquisition
 Profound developmental stages for the individual
 For the academy:
 what is a university for?
 In an open world, who determines what is (or should
be) of value?
 Who holds the power to say ‘this is worthy’ or ‘this is
valid’?

12. Digital Literacies
Mary Lea & Robin Goodfellow
 Learners bring a wealth of experience to bear – some
appropriate, some not
 Learners are engaged in meaning-making
 Recognition of the central role of texts in construction of
knowledge and practice of learning
 Potential shifts of power between learners, communities
and institutions
 Role of the institution is critically important
 Boundaries of ‘texts’ are more fluid and unstable than in
previous times

13. Improving our understanding of student behaviour?
 ‘Rich accounts in the literature of students’ use of
technology’
 ‘No detailed or in depth examination of what students
actually do in contexts when using different applications, or
how meanings are being made from, and through,
engagement with digital technology’
 ‘Recognition of the central nature of texts both in the
construction of knowledge and the practice of learning’
Lea and Jones (2011)

14. Ecological approach
 Interrelationship among all the different communication
technologies and
 the cultural communities that grow up around them
 the activities they support.
 ‘Interactivity is a property of the technology, while
participation is a property of culture.’
Jenkins (2004)

15. Ecological Space in which
learning happens
Members
Contribute
feel some
when you
connection
want
– they care

16. Affinity Spaces
Gee (2009)

17. Distinctions between formal and
informal learning spaces
Formal education system Informal affinity space
 Conservative  Experimental
 Static  Innovative
 Structures to sustain are  Structures to sustain are
institutional provisional
 Remain little changed over  Can respond to short-term
long periods of time needs and temporary interests
 Communities are bureaucratic  Communities are ad hoc and
and often national localised
 Does not allow for easy  Allows for easy moves in and
movement in and out out of informal learning
communities
Gee (2009)

18. Web 2.0 Technology supports open pedagogy:
peer support, communication and sharing
Issues for Institutions Wider sociological issues
 Blurring boundaries  Need to prepare learners
between formal and for future information
informal learning overload as the Web
 Ways of learning are grows exponentially
coming out of the  Need to sharpen critical
academy awareness, critical skills,
 Emancipation is in the and concepts of authorial
hands (and feet!) of voice
learners

19. The e-Assessment Challenge
…Push of Constructivist Learning
…Pull of institutional reliability and accountability
Slide courtesy of D.Whitelock, 2011

20. Assessment 2.0
Denise Whitelock
Characteristic Descriptor
Authentic Involving real-world knowledge and skills
Personalised Tailored to the knowledge, skills and interests
of each student
Negotiated Agreed between the learner and the teacher
Engaging Involving the personal interests of the students
Recognition of existing skills Willing to accredit the student’s existing work
Deep Assessing deep knowledge – not memorization
Problem orientated Original tasks requiring genuine problem
solving skills
Collaboratively produced Produced in partnership with fellow students
Peer and self assessed Involving self reflection and peer review
Tool supported Encouraging the use of ICT
Characteristics of Assessment 2.0, Elliott (2008) in Whitelock & Watt (2008)

21. Pedagogical Framework for self-publishing with
social software
1. Establishment
Learners actively create personalised learning environments with social
software e.g. weblogs, wikis, social bookmarking and aggregation.
2. Interpretation
Learners develop a structure and adapt it to their perceived needs.
3. Reflective Monologues
Learners publish to their software platform and establish their identity
4. Reflective Dialogues
Learners extend their learning environment by developing social networks.
5. Distributed knowledge artefacts
Learners collaborate with others, distribute their work, and gather
artefacts for review and reflection.
Bartlett-Bragg (2007) in Whitelock & Watt (2008)

22. The 4Ts pyramid to facilitate moving forward with
Assessment Frameworks and Web 2.0 Tools
Tool development
Transfer of learning from assessment tasks
which include Advice for Learning
Transformation of
Assessment tasks
Training of Staff
Tool
development
Adapted from Whitelock (2010)

23. Advice for Action
 The role of socio-emotive content in feedback is critically
important and cannot be ignored (e.g. Draper, 2009b).
 Assessment practices that focus on self assessment and
peer feedback need to develop towards ‘Advice for Action’,
i.e. stimulus advice for transformational change in students
to get them to:
 think differently
 to reconceptualise the way they respond
 to engage actively in the discourse
Whitelock & Watt (2008)

24. Open Comment
 Automated formative assessment tool
 Free text entry for students
 Automated feedback and guidance
 Open questions, divergent assessment
 No marks awarded
 For use by Arts Faculty
Whitelock & Watt (2008)

25. Stages of analysis of students’ free text
entry for Open Comment:
Advice with respect to content (socio-emotional support
stylised example):
STAGE 1a: DETECT ERRORS E.g. Incorrect dates, facts.
(Incorrect inferences and causality is dealt with below)
 Instead of concentrating on X, think about Y in order to
answer this question
Recognise effort (Dweck) and encourage to have another go
 You have done well to start answering this question but
perhaps you misunderstood it. Instead of thinking about
X which did not…….. Consider Y
Whitelock & Watt (2008)

26. Computer analysis continued
 STAGE 2a: REVEAL FIRST OMISSION
 Consider the role of Z in your answer
 Praise what is correct and point out what is missing
 Good but now consider the role X plays in your answer
 STAGE 2b: REVEAL SECOND OMISSION
 Consider the role of P in your answer
 Praise what is correct and point out what is missing
 Yes but also consider P. Would it have produced the
same result if P is neglected? … and so on
 Several other stages of analysis not discussed here
Whitelock & Watt (2008)

27. Role of technology
Adrian Kirkwood and Linda Price
 Student behaviour is not driven by technology per se, but
by the way in which technology is used to support learning
and teaching.
 ‘If academic staff genuinely want their students to be
analytical and critical thinkers, and able to apply their
learning to novel situations and transfer their learning
to solve real problems … then their assessment
methods should firstly, encourage the development of
such abilities; and secondly, provide students with the
opportunity to demonstrate that they have developed
these higher order abilities.’ Scouller (1998)
 This is as true for Web 2.0 as it was in 1998
Kirkwood & Price (2008)

28. Student expectations
 Säljö (1979): what do you understand learning to be?
 Learning as the increase in knowledge.
 Learning as memorisation.
 Learning as the acquisition of facts, procedures, and
so on, that can be retained and/or utilised in practice.
 Learning as the abstraction of meaning.
 Learning as an interpretive process aimed at the
understanding of reality.
 Beaty, Dall’Alba, and Marton (1997) added:
 Learning as personal development.
Kirkwood & Price (2008)

29. Quantitative change (passive):
Learning as the increase in knowledge.
Learning as memorisation.
Learning as the acquisition of facts, procedures, and so
on, that can be retained and/or utilised in practice.
Qualitative change (active):
Learning as the abstraction of meaning.
Learning as an interpretive process aimed at the
understanding of reality.
Learning as personal development.
Kirkwood & Price (2008)

30. Learners are emancipating themselves
 Web 2.0 technology has the potential to
enable many activities to be mediated
on-line, thus making the establishment
of communities much easier, much
cheaper and possibly more rewarding
for participants
 Affinity spaces can be spontaneously
formed around any topic, so can be
rapid, responsive and flexible means to
effective learning

31. Global News
KNOWLEDGE
World Service
openlearn

32. Can assessment practice keep up?
 Appropriate socio-emotive feedback
 Automated, but highly relevant, feedback/forward
 Focus on meta-level skills and their assessment
 Showing students how to value their own work and that of
peers – peer critique is highly valuable both as a giver and
receiver
 The academy no longer holds sway

33. References
 Beaty, E., Dall’Alba, G., & Marton, F. (1997). The personal experience of learning
in higher education: Changing views and enduring perspectives. In P.
Sutherland (Ed.), Adult learning: A reader (pp. 150–165). London: Kogan Page.
 Draper, S. (2009). Catalytic assessment: understanding how MCQs and EVS can
foster deep learning. British Journal of Educational Technology, 40(2), 285-293.
 Elliott, B. (2008). Assessment 2.0: Modernising assessment in the age of Web 2.0.
Scottish Qualifications Authority. Retrieved June 22, 2009, from
http://www.scribd.com/doc/461041/Assessment-20
 James Gee, Situated Language and Learning: A Critique of Traditional Schooling.
New York: Routledge, 2004.
 Henry Jenkins’ blog (accessed 21 December 2009)
http://henryjenkins.org/2006/10/confronting_the_challenges_of.html
 Lea, M., & Jones, S., (2011) Digital Literacies in Higher Education: exploring
textual and technological practice, Studies in Higher Education, 36 (3)

34. References 2
 McAndrew, P., Scanlon, E. and Clow, D., (2010). An Open Future for Higher
Education. Educause Quarterly, 33(1)
 Säljö, R. (1979). Learning about learning. Higher Education, 8(4), 443–451.
 Henry Jenkins’ blog (accessed 21 December 2009)
http://henryjenkins.org/2006/10/confronting_the_challenges_of.html
 Scouller, K. (1998). The influence of assessment method on students’ learning
approaches: Multiple choice question examination versus assignment essay.
Higher Education, 35(4), 453–472.
 Whitelock, D. and Watt, S. (2008). Reframing e-assessment: adopting new
media and adapting old frameworks. Learning, Media and Technology, Vol. 33,
No. 3, September 2008, pp.153–156 Routledge, Taylor & Francis Group. ISSN
1743-9884