2. Context
1. Pressure on staff as HE financing changes
2. Student expectations (esp. under new fee regime)
Key question
What’s ‘the deal’ (going to be)?
3. A ‘sustainability’ consideration for staff
Assessors have to be able to sustain the expectations laid
upon them.
What volume of good assessment practice can they realistically
undertake as the economic screw tightens?
(And does ‘the management’ know?)
4. Agenda (rather loosely organised)
1. Curricula for autonomy
2. Formative assessment, feedback and feedforward
3. Summative assessment (especially grading)
5. Becoming a graduate
Ought to involve young people (and perhaps some older people)
in a significant transformation
Acquiescence Autonomy
Kohlberg 1964
Perry 1970 (reprinted 1998)
King and Kitchener 1994
Kuhn and Weinstock 2002
Baxter Magolda 2009
Guiding learners through the transformation from authority
dependence to self-authorship is a primary challenge for
twenty-first century higher education
Baxter Magolda (2009, p.144)
6. Students often take time to ‘get it’
[School study] habits can continue well into the first year of
university.
(Leckey and Cook, 1999)
[N]ew undergraduates often see the tutor as the 'expert'
who can (and perhaps should) give them 'the information'.
By contrast university history teachers emphasise the need
for student autonomy and independent judgement.
(Booth, 2005)
There may not be a ‘truth’ that can be handed down
7. Students often take time to ‘get it’
Despite help, students did not grasp what was required
of them as regards essay-writing.
(McCune, 2004)
Students need help in order to develop as autonomous
learners.
(Fazey and Fazey, 2001)
At the beginning you have no idea what constitutes a pass
as you have no frame of reference. Need feedback on earlier
work before progressing to next assessment.
(Student, in Johnston and Kochanowska, 2009)
But what if the next assessment task is very different?
8. A pedagogy for autonomy
Students should be encouraged to develop their capacity
to self-assess over a lifetime (Boud, 2000).
Royce Sadler is insistent that higher education should develop
in students an appreciation of standards and how their work
measures up (e.g. Sadler, 2009).
A ‘patient curriculum’? In which ...
Assessment and feedback are of critical importance
11. What surveys say
Feedback has relatively low ratings from ‘graduates’ (NSS; CEQ)
‘Your First College Year’ survey in the US doesn’t ask about it
In first-year experience surveys that did:
Australia Australia UK
FYE 2005 FYE 2010 FYE 2007
33% 33% 57%
found found found
feedback feedback feedback
helpful helpful helpful
12. Good assessment and feedback should…
1. Clarify goals, criteria, standards Sadler:
2. Encourage time and effort on challenging tasks Students
should
3. Give good feedback that helps self-correction
internalise
4. Provide opportunities to act on feedback standards
5. Ensure summative assessment assists learning
6. Encourage dialogue about learning
7. Facilitate self-assessment and reflection
8. Encourage motivation and self-esteem
9. Give students choice regarding assessment
10. Involve students in policy and practice re assessment
11. Support development of learning groupings
12. Inform teachers about their teaching
Adapted from Nicol, 2009
13. So what about students’ expectations
and experience?
14. Perceptions of feedback
Informing students about feedback is important since what
staff consider to be feedback is not always appreciated
as such by students
Students often or very often received it 38.6
Staff often or very often gave it 80.4
0 20 40 60 80 100
AUSSE 2008
15. Match of expectations and experience: assessment methods (A&D)
Have the methods by which
your work has been assessed Per
been as you had expected? N cent
No 55 8.1
Partly 226 33.1
Yes 402 58.9
Total respondents 683 100.0
16. Match of expectations and experience: assessment methods (A&D)
70%
60%
50%
40%
Low prior inf
30%
High prior inf
20%
10%
0%
No Partly Yes
17. Expectations regarding feedback
Issue (abbreviated) 2006 2007 2010
N=979 N=1774 N=3091
% Agree % Agree % Agree
Ready access to staff outside f2f important 87 87 87
I expect teachers to read drafts 52
Feedback on drafts important to learning 92 95
Acceptable time-interval for return of work
1 week 2-3 weeks 4-6 weeks
2006 % 34 57 4
2007 % 38 55 3
2010 % 19 60
Crisp et al (2009); Scutter et al (2011)
18. Expectations re feedback ... and experience
Item Item theme At End year 1 Year 2 Teachers
no. (varies with orientation
group
studied) % Agree % Agree % Agree % Agree
7 ‘Ready’ access to
lecturers, tutors 88 89 61 50 73 57 64 94
8 Feedback on
submitted work 97 97 66 37 74 48 100 59
9 Feedback on
DRAFTS of work 94 91 20 7 21 26 0 22
Colour code: Humanities Science
Note: phrasing adjusted to fit circumstances
Brinkworth et al (2009)
19. Issues relating to the effectiveness of feedback
Inadequate understanding of the task (Glover & Brown 2006)
Student-staff differences in perception (Maclellan 2001)
Feedback not understood (Chanock 2000; Weaver 2006)
Tutors’ intentions re feedback not understood (Higgins et al 2002)
‘Being told’ is not engaging (Crisp 2007)
Disjuncts in the feedback loop (Hounsell et al 2006)
Transferability not perceived (Carless 2006; Duncan 2007) ...
... especially where assessment demands are diverse (Gibbs, ongoing)
Timeliness (Brinkworth et al 2009)
The grade’s the thing (Snyder 1971)
Espoused theory & practice out of sync (Orrell 2006; Orsmond et al 2011)
Where might enhancement efforts most usefully be targeted?
20. Worthwhile feedback?
The feedback on my assignments comes back so slowly that we
are already on the topic after next and I’ve already submitted
the next assignment. It’s water under the bridge, really.
I just look at the mark and bin it.
Collected by Graham Gibbs
Re a diagnostic essay:
It took ages … about two months … by that time I guess you
had forgotten … we had already wrote (sic) another essay.
Murphy and Cornell (2010, p.46)
I found that I did not learn anything from my mistakes as I was
never told what they were.
Misko and Priest (2009, p.15)
21. What some students said
I don’t like these shorthand comments, and we get too many of
them— ‘good analysis’—what does that mean? Where? And
words like ‘weak’, ‘good’ and ‘strong’—I mean, is the whole thing
weak … and blank, pointless comments like ‘wrong’ or ‘irrelevant’
or random question marks, what use is that to anybody?
Brown (2007, p.40)
Terms like ‘check spellings’ and ‘more depth’ are unhelpful,
just as are vague phrases like ‘this is unclear’.
Orsmond et al (2005, p.377)
... I got told that a piece of work was more like an essay than a
literature review. This is not helpful as it does not tell me what
should be contained in a literature review or how it should be
presented.
Weaver (2006, p.388)
22. When writing is not enough
When it (feedback) was written I didn’t understand it, but
obviously you make the appointment to follow through on
that and she clarifies what she means …
I didn’t know where I’d gone wrong, what she meant, and
she explained it, broke it down …
Murphy and Cornell (2010, p.47)
23. Some comments from a recent survey of
first-year students in Art & Design
24. Criteria
The assessment criteria are thoroughly explained at numerous
points throughout the module so you know exactly what they
are looking for.
We get a copy of the assessment criteria with our brief so we
always have it to refer to so we understand what exactly is
expected.
To be honest, I'm not entirely clear how my work is assessed.
It's never been explained in simple terms, instead, paragraphs
of descriptions, I'm sure it must be simpler!
25. Timing
The written feedback was most helpful but was given too late
at the end of the projects meaning there was no opportunity
to improve.
My work is marked at several intervals throughout the year,
rather than everything being marked at the end of the year,
so I have several deadlines instead of one big final deadline.
This reduces stress and helps me to keep on top of my work.
Interim verbal and written feedback half way through an
assignment has been the most useful to me as it has allowed
me time to take on board what has been said and use (or not)
the comments made.
26. Outcomes
You get given a piece of paper at the end where they have
ticked some boxes and given you a percentage. That is not
an assessment, it’s disgusting for the amount of work you do.
Being given areas for improvement or development is the
most useful [aspect of assessment] and that the fact that the
formal aspect of this is written and you can reflect upon what
is written is very useful.
28. Motivation
Pedagogical approaches at tertiary level must motivate
students to learn if deep, effective and engaged learning
is to take place.
(Kift and Field, 2010, p.4, original emphasis)
With striking consistency, studies show that innovative,
active, collaborative, and constructivist instructional
approaches shape learning more powerfully, in some
forms by substantial margins, than do conventional
lecture-discussion and text-based approaches.
(Pascarella and Terenzini, 2005, p.646)
30. Encouraging motivation
Students observed that feedback was given in such a way
that they did not feel it was rejecting or discouraging . . .
[and] that feedback procedures assisted them in forming
accurate perceptions of their abilities and establishing
internal standards with which to evaluate their own work
Mentkowski and Associates (2000, p.82), emphasis added
Boud and Sadler (who argue for students’ internalisation of
standards) would approve
31. Anxiety
I was really, really scared because it was my first report.
I had to hand it in just to see where I’m at. Where am I
standing?
(McGinty, 2011)
Psychological pain
They are writing all over my work and it is like mangled up
and most of the lecturers use red pen and I don’t know it
kind of gets to me if I open it up and it’s covered with red
crosses and marks and it’s horrible. It’s like my work is
bleeding.
(McGinty, 2011)
32. Self-confidence
Week 4 Week 10
Lack of confidence 23% 40%
Expected Actual
Distinction + 7% ~ 7%
High pass 80% 57%
Fail 1% 17%
Cameron (2008)
33. Acknowledging emotion
For [first-year] students feedback goes beyond providing
information on how to improve assessment marks. The
‘effective feedback’ for these students is that which
provides emotional support and facilitates integration into
university.
(Poulos and Mahony, 2008)
To what extent can ‘the personal’ be accommodated in
contemporary HE?
34. Message not quite clear?
The numbers and language used in assessment are opaque
and need to be demystified
35. Standards
Normative (even when expressed in criterion-referenced terms).
Discipline-driven, sometimes with external input.
Difficult to define (like QAA Subject Benchmarks?).
But ‘We know them when we see them’ – or we ought to).
Or assessors may think they know standards, but notions such as
• ‘graduateness’
• ‘employability’
• ‘wicked competences’
• ‘soft skills’
stretch assessors’ confidence to (beyond?) the limit.
Learning outcomes are a kind of proxy, but are inherently fuzzy.
Assessment criteria are likewise fuzzy.
36. The problem with criteria: ‘Conceptualisation’
A/1st: Able to recognise consistency and reconcile inconsistency
between information using cognitive and hypothesising skills.
B+/2.1: Consistent understanding demonstrated in a logical and
lucid manner.
B/2.2: Demonstrate understanding in a style which is mostly logical,
consistent and flowing.
C/3rd: Attempts to demonstrate a logical and coherent understanding
of the subject area but aspects become confused or are underdeveloped.
Refer/Fail: Understanding of the assignment not apparent, or lacks a
logical and coherent framework, or the subject is confused or
underdeveloped.
Price & Rust (1999, p.134)
How do you convey the meaning of all this to students?
37. Dealing with multiple learning outcomes:
a hypothetical example
Learning outcome Fully achieved Partly achieved Not achieved
A
B
C
...
Intended learning outcomes [ILOs] might be categorised as
primary or secondary, thereby influencing the overall assessment.
Developed from Sadler (2005)
Do students appreciate the relative importance of ILOs?
38. The potential for confusion: some examples
Formal statements of expectations are fairly meaningless
without exemplification, even for staff (e.g. Wolf, 1995).
Students’ perceptions of what assessment is seeking may not
align with what staff think they ought to be perceiving
(Maclellan, 2001).
35 of 76 history and politics students did not understand the
meaning of ‘more analysis, less description’ (Chanock, 2000).
Staff themselves did not agree on what was meant by ‘analysis’
and ‘evaluation’ (Webster et al, 2000).
Whilst published assessment criteria are used, not all of them
may be applied – and some extraneous criteria may be invoked
(Webster et al, 2000, regarding u/g dissertations).
39. How well do grades inform?
The [honours degree] classifications are too general,
there’s no way of showing if you were close to the
grade above.
A straight percentage mark would be fairer.
‘Maria’
The Guardian RISE supplement, 18 December 2004
Not so. For a start, there’s variation between subject areas
40. L1 Module means from 13 Owning Organisational Units
80.0
70.0
60.0
50.0
40.0
30.0
CMP 5 LSS 31 ENR 8 PBS 19 NSP 29 ECL 75 BUE 16 LSA 26 SPS 11 HSS 37 HEA 16 LBS 34 LAW 10
41. So why the variation?
Lots of variables may have exerted influence, including:
• Student calibre (entry qualifications; commitment)
• Nature of the subject (hard/soft; pure/applied)
• Curriculum design
• Pedagogic quality
• Resourcing
• Expected standards (intended learning outcomes)
• Mode of assessment
• Nature of the assessment demand
• Marker variability (in some cases, due to differences in School)
42. What is the relationship between mode of
assessment and marks awarded?
43. LJMU L1 module mean scores (N=317), by amount of coursework
Mean CW = <35% CW = 35-75% CW = >75%
Civil engineering surveying 1, CW=30%
65+
60-64.99
55-59.99
50-54.99
<50
BUE HSS LBS LSA LSS NSP
44. LJMU uses ‘percentages’. If you are marking work out of 100,
how do you arrive at the ‘percentage’ mark?
(If marking work out of a smaller total, the same general question applies.)
• You might tot up the marks from components of the
student’s performance.
• You might take a broad view of the overall performance
(e.g. ‘this is of 2.1 standard’), and then decide where in
the relevant band the actual mark should lie.
• You may have some other approach.
What approach do you use?
And what assumptions underlie your marking
methodology?
‘Fudges’? (e.g. Baume et al 2004; Bloxham et al 2011)
45. A question to ponder ...
Should you finely grade work-based achievements,
employability and ‘graduateness’?
If ‘yes’, then how?
Too complex an issue for this session, but a starting point is
Yorke M (2011) Assessing the complexity of professional achievement
http://learningtobeprofessional.pbworks.com/f/CHAPTER+A10+FINAL.pdf
46. What assumptions are being made?
(a)By staff
(b)By students
(c)And by others
40
35
30
25
The psychometric tradition would give 20
15
10
5
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
35
30
25
20
Using learning outcomes would give 15
10
5
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
But it’s rarely as cut and dried as that
49. Grades are not transparent
They can demoralise if they are awarded on a basis that
differs (negatively) from students’ previous experience.
Grading at A-level, or in other countries’ systems, may not
align with UK HE practices.
What does a mark of 64% mean?
And what does the ‘missing’ 36% relate to?
How different is 53% from 54%?
And so on ...
50. Some other issues
Grade only on assignment and/or feedback boxes ticked (or not)
Mark out of sync with comment
Feedback comment on matters outside stated criteria
Students not taught how to use feedback
The difference between being positive and being constructive
The need to provide emotional support (esp for 1st year students)
52. In a nutshell ...
1. Don’t assume that students will understand ‘assessment
language’. Give explanations and, particularly, examples.
2. Provide opportunities for students to develop their
capacity to self-assess, without undue risk.
3. Find ways to make feedback (and feedforward) effective.
4. Clarify what grades signify, in the subject context.
1 and 4 are the responsibility of teachers, and are fairly
straightforward.
2 and 3 are much more challenging, in that they may require
a measure of innovation – creativity even – on the part of
teachers and certainly the active engagement of the students.
53. Some references
Assessment and Evaluation in Higher Education is a particularly useful source of relevant articles, e.g.
• Maclellan, 2001
• Orsmond et al, 2005
• Weaver, 2006
• Webster et al, 2000
Bloxham S and Boyd P (2007) Developing effective assessment in higher education: a practical guide.
Maidenhead: Open University Press.
Boud D & Falchikov N, eds (2007) Rethinking assessment in higher education: learning for the longer term.
London: Routledge.
Nicol D (2009) Transforming assessment and feedback: enhancing integration and empowerment
in the first year. http://www.enhancementthemes.ac.uk/documents/firstyear/First_Year_Transforming_Assess.pdf
Yorke M (2008) Assessing student achievement in higher education: signals and shortcomings.
Abingdon: Routledge (Contains a wide range of references.)
Yorke M (2011) Assessing the complexity of professional achievement
http://learningtobeprofessional.pbworks.com/f/CHAPTER+A10+FINAL.pdf
Yorke M (2011) Assessment and feedback in the first year: the professional and the personal.
At www.fyhe.com.au/past_papers/papers11/FYHE-2011/content/pdf/MantzYorke_abstract.pdf
