Slides from Jisc Assessment and Feedback webinar on analysis of feedback 24 June 2013

1. JISC Assessment and Feedback Programme
Webinar
Analysis of Feedback
24 June 2013 12.00 - 13.00

2. Holly Smith IoE Assessment Careers H.Smith@ioe.ac.uk
Anne Jones QUB e-AFFECT a.m.jones@qub.ac.uk
Maria Fernandez-
Toro
Open University eFEP Maria.Fernandez-Toro@open.ac.uk
Peter Chatterton Daedalus e-
World
Critical Friend peter.chatterton@daedalus-e-world.com
Institute of Education
www.ioe.ac.uk/assessmentcareers
Queen’s University Belfast
http://www.qub.ac.uk/directorates/AcademicStudentAffairs/CentreforEducationalDevelopment/e-AFFECTproject/
Open University http://www.open.ac.uk/blogs/efep/
Speakers

3. To explore the following:
• Why analyse feedback?
• Approaches and tools for analysing feedback
• Institutional experiences of using the tools - resulting
feedback profiles and audits
• Benefits, impact and challenges from using the tools
Webinar objective

4. Why analyse feedback?

5. Why analyse feedback?
• Widely inconsistent practice in feedback (quality & quantity)
• Lack of learner understanding of, engagement with, and
dialogue/action & on feedback
• High teacher effort - low efficiency
• Not utilising self/peer feedback
• Transmitted feedback creates dependency on teacher
• Reduced staff satisfaction as evidence of feed-forward not seen
• NSS scores
Dependent learners
feedback “done to them”
Independent learners
capable of self-review

6. Approaches and tools
for analysing feedback

7. IOE Coding framework
• The score is the number of times a classification
appears in the feedback
• The default unit for analysis was the sentence
• Where a sentence contains clauses that make
distinct points, it was split into separate clauses,
each of which was classified separately.
• Neutral comments that for example describe the
piece of work, but do not make any judgement are
unclassified

8. IOE Feedback tool
P1 Giving praise
P2 Recognising progress or ipsative feedback
Criticisms
C1 Correction of errors
C2 Factual criticisms
C3 Criticism of approach
Giving advice
A1 Specific to content current assignment
A2 General skills in current assignment
A3 For future assignments
Q Clarifications and questions
O Other unclassified statements
Adapted from Orsmond & Merry,
2011 including Hughes, 2011.

9. e-AFFECT - What the students say
Last year their feedback
pointed out spelling
mistakes or referencing
mistakes, but we were
not told how to do things
right
What do they mean
by clear and concise?
Sometimes they say
something to
encourage that is not
really true – am I
excelling at it or are you being nice?
I liked feedback which
helped me improve my
work the next time. Despite
this, I felt that my marks
never really changed much
and they tended to stay at
the same level
What do ‘?’ and
‘What?’ in the
margin mean?

10. e-AFFECT - Analysis of coursework
• Content analysis of depth of feedback using adapted
categories developed by Glover and Brown (2006)
• Indication that there is an strength/error/weakness or
omission (Level 1)
• Provides correction or appropriate response/indication why a
strength (Level 2)
• Provides explanation as to why the student’s response was
incorrect or inappropriate or why suggestion was preferable or
how a strength can be built upon (Level 3)
Essays Lab reports Total
Year 1 40 26 66
Year 2 61 24 85
Total 101 50 151

12. • Analysis criteria based around two dimensions:
o Whether feedback focuses on strengths or weaknesses
o How much information the feedback provides, cf ‘depth’ of
feedback (Brown & Glover 2006) layers of scaffolding
• Analysis tool: Feedback Analysis Chart for Tutors (FACT)
Provides a visual ‘profile’ of a tutor’s feedback
Analysing assignment feedback
Screencast description: http://www.open.ac.uk/blogs/efep/?page_id=114

13. Layers Comments focusing on weaknesses Comments focusing on strengths
1
Error identified only Strength identified only
2
Error categorised, but not corrected Strength categorised or described as per
marking criteria
3
Error corrected Illustrated with specific example from
student’s
performance
4
Explanation given Explanation given
5
Advice given on how to prevent errors in
future performance
Advice given on how to develop
existing strengths in future
Manos blancosManos blancos Good workGood work ????
Layers of scaffolding in assignment feedback
Manos blancos AgreementManos blancos Agreement
Although it ends in O, ‘mano’ is
a feminine noun.
Although it ends in O, ‘mano’ is
a feminine noun.
Manos blancos blancasManos blancos blancas
Revise section 6.1 of
your grammar book
Revise section 6.1 of
your grammar book
You use a wide range of
language structures
You use a wide range of
language structures
No digo que quieran...
Good use of the subjunctive
No digo que quieran...
Good use of the subjunctive
This connector makes it very clear
that a new section is starting here.
This connector makes it very clear
that a new section is starting here.
Good, you could
also look up...
Good, you could
also look up...

14. Feedback Analysis Chart for Tutors (FACT)
Possible uses:
Enables us to compare...
 feedback relating to different criteria
 feedback given by different tutors
 feedback given to more/less proficient students
 feedback related to different types of assignment
(e.g. spoken presentations vs. written essays)
 feedback delivered through different media
(e.g. written vs. audio-recorded feedback)

15. Language
Weaknesses Strengths
Identified only Identified only
Categorised
Categorised /
Described
Corrected /
Modelled
Exemplified
Explained Explained
Future-oriented Future-oriented
Example of FACT analysis grid
(tutor A: beginner assignment)
Weaknesses Strengths
Identified
only
6
Identified
only
Categorised 2
Categorised /
Described
Corrected /
Modelled
8 Exemplified
Explained 3 Explained
Future-
oriented
Future-
oriented
Weaknesses Strengths
Identified only Identified only
Categorised 2
Categorised /
Described
Corrected /
Modelled
1 Exemplified
Explained Explained
Future-oriented Future-oriented
Weaknesses Strengths
Identified
only
Identified
only
Categorised 3
Categorised /
Described
Corrected /
Modelled
Exemplified
Explained Explained
Future-
oriented
1
Future-
oriented
Content Feedback form
Notes on script

16. Language
Weaknesses Strengths
Identified only Identified only
Categorised 3
Categorised /
Described
Corrected /
Modelled
6 Exemplified
Explained 2 Explained
Future-oriented Future-oriented
Example of FACT analysis grid
(tutor B: advanced assignment)
Weaknesses Strengths
Identified
only
4
Identified
only
Categorised 1 5
Categorised /
Described
Corrected /
Modelled
2 6 Exemplified
Explained
5
Explained
Future-
oriented
Future-
oriented
Weaknesses Strengths
Identified only Identified only
Categorised 6
Categorised /
Described
Corrected /
Modelled
4 2 Exemplified
Explained 1 1 Explained
Future-oriented Future-oriented
Weaknesses Strengths
Identified
only
Identified
only
Categorised 1 6
Categorised /
Described
Corrected /
Modelled
2 Exemplified
Explained 2 Explained
Future-
oriented
Future-
oriented
Content Feedback form
Notes on script

17. Questions?

18. Institutional experiences
of using the tools

19. IOE Data
• Analysed formative and summative assessment
feedback for modules on 5 postgraduate programmes
at the IOE (total 228 pieces)
• Recorded the total number of comments in each
category and the average per script
• Ranked the categories to obtain a feedback profile at
programme level as well as an aggregate profile of the
5 programmes.

20. IOE Profile for Summative Assessment
Total comments for 5 programmes
Summative assessment (N= 165)
Category of feedback Average per script Rank
P1 Praise 4.4 1
P2 Ipsative (progress) 0 (negligible) 5
C1-C3 Critique 2.7 2
A1-A3 Advice for current or future
assignments
1.9 (mostly for current
assignment)
3
Q Questions and clarification
requests
0.1 4

21. e-AFFECT - Comments on essays
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
Student response Student skills Achievement
%
Distribution of comments at Year 1
Strengths
Weaknesses
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
50.0
Student response Student skills Achievement
%
Distribution of comments at Year 2
Strengths
Weaknesses

22. e-AFFECT - Further learning comments
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
Source
materials
Dialogue
and
reflection
Future work
%
% Year 1 work
% Year 2 work

23. e-AFFECT - Depth of feedback
Depth of feedback
Indication that there is an
strength/error/weakness or omission (Level 1)
Provides correction or appropriate
response/indication why a strength (Level 2)
Provides explanation as to why the student’s
response was incorrect or inappropriate or why
suggestion was preferable or how a strength can
be built upon (Level 3)
-30 -20 -10 0 10 20
1
2
3
%
Depth of
feedback
Year 2
Weakness Strength
-15 -10 -5 0 5 10 15
1
2
3
%
Depth of
feedback
Year 1
Weakness Strength

24. -40 -20 0 20 40
1
2
3
%
Depth of
feedback
Year 1 Module A
Weakness Strength
-30 -20 -10 0 10 20 30
1
2
3
%
Depth of
feedback
Year 1 Module B
Weakness Strength
-60 -40 -20 0 20 40
1
2
3
%
Depth of
feedback
Year 2 Module C
Weakness Strength
-40 -20 0 20 40
1
2
3
%
Depth of
feedback
Year 2 Module D
Weakness Strength
e-AFFECT – Module depth of feedback

25. e-AFFECT - Feedback workshop

26. Analysing what?
Feedback on language assignments at the OU
Our sample:
 100 writing assignments
 100 speaking assignments
 4 levels (9 tutors per level)
 108 students (3 per tutor)
Feedback consists of:
 200 e-feedback forms
 100 annotated scripts
 100 audio files
Tutor
Student
Well done!

27. Summary of FACT analysis results: Use of the four media
E-feedback forms (writing/speaking)
Contain the lowest proportion of…
 Comments on strengths
 Comments on content
 Categorised strengths/weaknesses
 Advice for future (proportion on script = 0%).
Quite low everywhere else (both in the audio
feedback and on the e-feedback forms)
Script annotations/Audio feedback
Contain the highest proportion of…
Comments on weaknesses (often
adressed at more than one depth)
Comments on language
Corrected errors
Explanations (especially high occurrence
in the audio feedback)
Contain the lowest proportion of…
 Comments on weaknesses
 Comments on language
 Corrected errors/Exemplified strengths
 Explanations
All of these occur a bit more frequently for
the speaking assignment
Contain the highest proportion of…
 Comments on strengths
 Comments on content
 Categorised strengths/weaknesses
All of these occur even more frequently
for the written assignment

28. Questions?

29. Benefits, impact and challenges
from using the tools

30. IOE Benefits, Impact and Challenges
• Enables initiation of discussion of feedback at the
programme team level
• Facilitates reflection on the purpose of feedback
within programme teams
• Feedback practices are very entrenched and resistant
to change

31. e-AFFECT
Benefits, impacts and challenges from using the tools
Benefits
• Staff seeing the real issue rather than an imagined issue
• Engendering dialogue
• Raising awareness of feedback messages
Impacts
• Work in progress
Challenges
• Reaching consensus of the level and quantity of
feedback

32. • Useful research tool: Overall patterns of use of different media for giving
feedback on language assignments at the OU
• Results need to be interpreted with caution (e.g. ‘deeper’ feedback is not
necessarily the most appropriate in all contexts)
• Not suitable for quantitative evaluation by practitioners: Coding requires
complex guidelines in order to be reliable
• Suitable for awareness-raising purposes in staff training events. Materials
include: sample of marked assignments + coding grids + student webcasts
giving their ‘feedback on feedback’
(Online training event: 67% ‘very useful’ – 33% ‘possibly useful’)
• FACT criteria now also presented as a simplified checklist for reflection
• Informs new research strand focusing on feedback alignment
Benefits, impact and challenges
from using the FACT analysis tool

33. Questions?

34. Further info on projects

35. More info on projects
Assessment Careers-Institute of Education
www.ioe.ac.uk/assessmentcareers
http://youtu.be/VSaGbPoXPh0
References:
1. Brown, E. & Glover, C. (2006) Evaluating written feedback. in: B. C. & K. Klegg
(Eds) Innovative assessment in higher education. London, Routledge), 81-91.
2. Hattie, J. & Timperley, H. (2007) The Power of Feedback. Review of Educational
Research, 77, 81-112.
3. Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing Ipsative
Assessment in Higher Education Studies in Higher Education 36 (3): 353 – 367.
4. Orsmond, P. & Merry, S. (2011) Feedback alignment: effective and ineffective
links between tutors’ and students’ understanding of coursework feedback.
Assessment & Evaluation in Higher Education. 36(2): 125-126.
Holly Smith H.Smith@ioe.ac.uk

36. More info on projects
Queen’s University Belfast
e-AFFECT
http://www.qub.ac.uk/directorates/AcademicStudentAffair
s/CentreforEducationalDevelopment/e-AFFECTproject/
http://blogs.qub.ac.uk/e-affect/
Anne Jones a.m.jones@qub.ac.uk

37. More info on projects
Open University
The eFeedback Evaluation Project (eFEP)
http://www.open.ac.uk/blogs/efep/
Screencast description:
http://www.open.ac.uk/blogs/efep/?page_id=114
Maria Fernandez-Toro Maria.Fernandez-Toro@open.ac.uk