Presentation from International Assessment in Higher Education Conference, June 2015 on the evaluation of the use of formative thresholded assessment at the Open University Science Faculty.

1. Formative thresholded
assessment: Reflections on
the evaluation of a faculty-
wide change in assessment
practice
Sally Jordan, Department of Physical Sciences,
The Open University @SallyJordan9

2. Acknowledgements
● Those who had the idea of formative thresholded assessment, in particular Joy
Manners, Stuart Freake, John Bolton, Andrew Norton.
● Those who contributed to the evaluation: John Bolton, Lynda Cook, John Golding,
Janet Haresnape, Richard Jordan, Kerry Murphy, Karen New, Ruth Williams.
● Members of the formative thresholded evaluation group and/or the Science Faculty
Assessment Group: Terri Battrick, Audrey Brown, Simon Clark, Anisha Dave, Basiro
Davey, Ellie Dommett, Anne-Marie Gallen, Ellen Heeley, Katherine Leys, Jane Loughlin,
Jean McCloughry, Donal O’Donnell, Simone Pitman, Claire Rothwell, Peter Taylor,
Vicky Taylor, Joy Wilson, Carlton Wood.
● Ben Palmer for help in extracting data; Doug Clow for advice on data wrangling; Anne
Adams and Ann Jones for advice on qualitative techniques.
● Saroj Datta, Basiro Davey and Ellie Dommett for their related work on SDK125.
● Associate lecturers who reported on their S141 students’ reasons for non-submission:
Amy Clarke, Catherine Halliwell, Jane Kendall-Nicholas, Maria Townsend, Pam
Spence, Sarah Chyriwsky.
● Members of the module teams involved in the project, in particular Claire Rothwell and
Crispian McArthur for S104 and Diane Butler and Jessica Bartlett for S141.
● Tracey Moore, Elaine McPherson, Jennie Bellamy, Gill Knight and other qualification
and curriculum managers for supplying data about TMA submission rates and insight
into reasons for anomalies.
● Financial assistance from eSTEeM, and support from Diane Ford, Steve Swithenby and
Nick Braithwaite.
2

3. The Open University
●Founded in 1969
●Supported distance learning
●200 000 students, mostly studying part-time
●Most undergraduate qualifications are completely open entry,
so students have a wide range of previous qualifications
●Normal age range from 18 to ??
●20 000 of our students have declared a disability of some sort
●13 000 of our students live outside the UK
●Students study on modules (mostly 30- and 60-credit) which
are combined together into qualifications in a number of ways.
3

4. Science Faculty assessment practice
●Tutor-marked assignments (TMAs) and sometimes interactive
computer-marked assignments (iCMAs) combine together into
overall continuous assessment score (OCAS).
●Examination and/or end-of-module assessment (EMA) gives
overall examination score (OES).
●Module result reflects both OCAS and OES.
●Students receive detailed feedback comments on TMAs and
iCMAs, so they have a significant formative function.
●Assignments serve formative (“for learning”)/summative (“of
learning”) double duty.
Historically…
4

5. Our change in emphasis
●Model A Students have to reach threshold (usually 40%) for
OCAS (overall), but their module result is then determined on
the strength of their OES alone.
●Model B OCAS is formative, but students have to demonstrate
engagement by scoring more than 30% in x out of y TMAs and
iCMAs; final score is determined by OES alone.
“Formative thresholded assessment”
5

6. Drivers for change
●“when assessments serve both formative and summative
purposes… formative work will always be threatened due to
the dominance of summative requirements” (Brearley & Cullen
2012).
●To free students from anxiety over the minutiae of grading of
TMAs and iCMAs, placing greater focus on feedback and
dialogue between students and associate lecturers.
●To establish more honest assessment strategies. As Freake
(2008, p.4) points out: “In practice, the summative nature of
the continuous assessment is somewhat illusory for the vast
majority of Physics and Astronomy students since their course
grade is determined by their exam mark, which is generally
significantly lower than their continuous assessment mark.” 6

7. Drivers for change
●“To allow questions to be re-used so “more effort would be put
into optimising the questions, the tutor notes and feedback,
and the hints and answers provided by iCMAs, because they
would be reused each year. Improvements could be made if
necessary in the light of experience” (Freake, 2008, p. 4).
●To remove one of the barriers to more frequent presentation of
the courses and to flexible study rates (since new assignments
would not need to be produced for each presentation) and to
encourage the use of ‘”little and often” assessment, for pacing
(Gibbs & Simpson, 2004-5).
7

8. Evaluation
●Has the move to formative thresholded assessment led to a
change in module completion and pass rates?
●Has the move to formative thresholded assessment led to a
change in TMA and iCMA submission rates?
●Has the move to formative thresholded assessment led to a
change in TMA performance?
●How do the levels of student engagement on iCMAs compare for
summative, formative thresholded, and purely formative use?
●Has the move to formative thresholded assessment led to a
change in the extent or type of plagiarism cases that are
detected?
●What are the relative merits of Model A and Model B formative
thresholded assessment?
●What do students and associate lecturers know about and think
of our assessment strategies?
Our research questions:
8

9. Findings
TMA submission rates and module completion for two
presentations of a level 3 life science module.
9

10. Findings
TMA submission rates and module completion for three
presentations of a level 2 environmental science module.
10

11. Findings
TMA submission rates and module completion for three
presentations of a level 2 Earth science module.
11

12. Impact of TMA submission on
overall success
Not surprisingly, on average, students who submit all
assignments do better on the examination or end-of-module
assessment.
Some students would probably have been better prepared for
the exam if they had done all the TMAs.
However, there is evidence that some students are making a
sensible decision to omit a TMA (usually the last one), or to
worry less about this assignment, and so to concentrate on
revision.
N.B. Correlation does not imply causality!
12

13. Student engagement with iCMAs
“Learning analytics”
13

14. Engagement with iCMAs
Number of questions attempted in purely formative use (L)
and formative thresholded use (R).
14

15. Engagement with iCMAs
Proportion of responses correct at first, second and third try,
and not at all, for an identical iCMA in formative thresholded
(top) and formative use.
15

16. Engagement with iCMAs
The proportion of responses correct at 1st, 2nd and 3rd try,
showing the extent of repeating and blank responses for the
same S104 question in summative, formative thresholded
(Model A) and purely formative use.
16

17. iCMAs as scaffolding
Overall use of formative thresholded iCMAs with hard cut-off
dates.
17

18. Impact on plagiarism
●A review of plagiarism cases in summer 2014 found that, whilst
the re-use of assignments had been accompanied by an
increase in plagiarism cases, it was not possible to attribute
causality, and the increase in cases had not been as large as
feared.
●The introduction of “assessment commentaries” has reduced
the public availability of model answers and, arguably, given
more useful feedback to weak students.
●In formative thresholded assessment, students who copy
assignment answers are only really cheating themselves.
18

19. Student opinion
I really like the fact that I can treat the TMAs with the same
dedication as I have in all the courses, but haven’t got the stress
of aiming for 100% with each one. Looking back on my previous
courses I think I stressed over my TMAs far too much, perhaps
to the detriment of revising throughout the year in preparation
for the exam. I always wanted each TMA to be perfect so that I
could gain the maximum marks for the overall course
assessment. The reality for me has always been that my exam
grade is the one which has determined by final grade. This year
I am still working hard on my TMAs but I am not stressing about
every tiny detail like I used to . I think I can actually say I am
enjoying my TMAs. Using this method of working has also freed
up some of my quite limited time to allow me to go back and
look over the previous books.
A comment from one student:
19

20. Student perception
●The most shocking finding of the whole evaluation has been
the realisation that very large numbers of students and staff do
not understand our previous, well-established assessment
strategies. We need to make our practices clear.
●Students and associate lecturers also called for uniformity of
practice across qualifications (though the findings of a
university-wide consultation on qualification-focused
assessment were less clear on this point).
●How clear and consistent is your assessment practice?
20

21. For the future
●Formative thresholded assessment has been very successful,
but there is a reasonable anxiety that our novice students may
end up being summatively assessed for the first time in an
examination or another high-stakes assessment.
●We are also uneasy about over-reliance on examinations for
the assessment of skills such as problem solving and the
writing up of experiments; and it is difficult (impossible?) to
assess collaborative activity in an examination.
●We have therefore introduced two-stage OES on all new
Science Faculty level 2 modules, with 25% of the available
marks coming from a tutor-marked assignment (with feedback
provided in the usual way) and 75% from an examination.
21

22. References
Brearley, F.Q. & Cullen, W.R. (2012). Providing Students with
Formative Audio Feedback. Bioscience Education, 20, 22-36.
Freake, S. (2008). New assessment models for Physics and
Astronomy…”the times they are a-changing.” OpenCETL
Bulletin, 3, 4-6.
Gibbs, G. & Simpson, C. (2004-5). Conditions under which
assessment supports students’ learning. Learning and Teaching
in Higher Education, 1, 3-31.
The full report “Thresholded assessment: Does it work?” is
available on the eSTEeM website:
http://www.open.ac.uk/about/teaching-and-
learning/esteem/projects/themes/innovative-
assessment/thresholded-assessment-does-it-work 22

23. Sally Jordan
Department of Physical Sciences
The Open University
sally.jordan@open.ac.uk
@SallyJordan9
www.open.ac.uk/blogs/SallyJordan
23