Keele University, October 2017, Chemical Sciences Seminar Series

1. From Tesla to TESTA:
Meanderings in Chemistry
Education Research
Dr Katherine J. Haxton
@kjhaxton
Keele University
November 2017

2. Sam Goodwin, Eilesh Tolly-Brewster
(research students)
Dr Martin Hollamby, Dr Scott Sneddon
(help with administration of tests)
1st and 2nd year Keele University Students over past 3 years
Dr Aleks Radu
(inspiration on confidence scales)
Prof Simon Lancaster, Prof David Read, Prof Paul Taylor,
Dr Suzanne Fergus, Dr Natalie Rowley, Dr Paul
Duckmanton
(feedback on first drafts)
Diagnostic Tests: Acknowledgements

3. Diagnostic Test? What’s that?
Means of assessing understanding, essential
concepts
1st year: Spectroscopy (IR, NMR, UV-Vis,
Equations of Light)
Pre-university knowledge, administered in week 1.
2nd year: NMR, administered in 2nd half SEM 1
prior to mNMR teaching.
1st/early 2nd year knowledge.
Both influence lecture courses

4. Think Like a Chemist
Understanding chemical reactions is a key skill
for a chemist.
But what does that mean?
- Knowledge of types of reactions and
reagents
- Knowledge of reaction conditions and rate
- Knowledge of characterisation techniques
and applications
- Knowledge of the mechanism of the reaction
on the molecular/atomic level

5. Concepts or Threshold Concepts?
Threshold concepts are defined as:
‘transformative, integrative, irreversible, troublesome and
bounded’
e.g.? interpreting 1H NMR in an in/organic context (NOT
simply interpreting NMR)
- bounded – in/organic molecules
- troublesome/integrative – lots of different bits to get
right
- irreversible – hopefully
- transformative – learn it, do it
Threshold concepts typically integrate multiple concepts
and so need broken down for a diagnostic test
Talanquer, V., J. Chem Educ. 2015, 92, 3 – 9 and references within)

6. Diagnostic Tests (concept inventory)
Evaluating the different concepts that may underpin a
threshold concept
Assess ‘Alternative Conceptions’ through MCQs with
appropriate distractors
Evaluate knowledge ‘on entry’ to course/module/lecture
segment
Evaluate ‘learning gain’ if questions of sufficient stretch
included
Many relate to school-uni transition, or focus on ‘general
chemistry’. Few relating to 1st/2nd year transition, or UK HE
context.

7. Alternative Conception? What?
Alternative conceptions make sense to the students holding
them
- constructed as explanations for observed phenomena
- integrated into their personal schema
- believed to be correct until challenged
- may elude detection until the context changes
- may be corrected when new information is presented that
contradicts the alternative conception

8. Example: n+1 rule, I = ½, J coupling
H3CCH2OH H3CCH2SH
H3CCH2COH
In 1H NMR, the CH3 group would be a triplet.
Is this because there are three hydrogen atoms or because
the CH3 group is adjacent to a CH2 group?
H3CCH2COCH3 H3CCH2OCH3
In 1H NMR, one CH3 group would be a triplet, and the
other a singlet. Why?

9. Development Cycle
Analysing
free text
answers
Authoring/editing
MCQ questions
Using test
for
teaching
insights
Refining
test
Concept
map of
topic

10. Interpretation of
NMR in inorganic
context
CouplingIntegration
Electronegativity
Chemical Shift
Abundance
NMR
Activity
Proton
Exchange
Fluxionality/Molec
ular Motion
Molecular
Shape
Solvent
Effects
Functional
Groups

11. Answer Analysis (S. Goodwin)
0 20 40 60 80
VSEPR
Splitting
Integration
Omission (part of question)
Other (question design
issue)
% of errors in answers analysed

12. Typical mNMR Example
Multiple coupling constants,
identifying different
environments,
VSPER critical.

13. Structure of Diagnostic Tests
︎Questions to probe specific alternative conceptions
Confidence scale to rate how strongly held concept is (vs.
guessing)
{Whether topic has been studied before}
[Free text answer space for working or explanation]
Use to alert students and lecturer to potential issues prior to
teaching and direct students to additional learning resources
Analysis: statistical analysis to establish test validity (although
questionable)

14. Have you studied these topics before?
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
UV-VIS IR NMR Eqn of Light
NO YES MAYBE

15. Have you studied these topics before?
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
Chemical Equilibrium Lewis Structures VSEPR
NO YES MAYBE

16. Which of the following travels slowest through space?
0
10
20
30
40
50
60
70
They all
travel at the
same speed
Visible Radio x-rays infrared
Answers Selected (%)
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
LOW MID HIGH
Confidence (%)
Correct
Incorrect
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
YES MAYBE NO
Studied Before? (%)

17. Dunning-Kruger Effect
HowMuchPeopleThinkTheyKnow
How Much People Know
High Confidence,
Wrong Answer
Low Confidence,
Correct Answer
Increasing Confidence,
Correct Answer
Learning?

18. Which of the following molecules has a linear shape?
0
10
20
30
40
50
60
70
80
CO₂ H₂O SO₂ NO₂
Answer Selected (%)
0
20
40
60
80
100
YES MAYBE NO
Studied Before? (%)
0
10
20
30
40
50
60
70
LOW MID HIGH
Confidence (%)
% correct
% incorrect

19. Structural Errors (Eilesh Tolly-Brewster)
O
11
22
O
33
44
55 66
77
88
Omission or addition of
implicit hydrogen (C2
and C6)
Focus groups/interviews conducted with students across
several years, organic NMR context
Random errors and alternative conceptions identified

20. Next Steps
Data analysis
Incorporating Eilesh’s findings into 2nd year
diagnostic test before mNMR teaching
commences
Extending ethical approval to run more focus
groups/interviews, and approval to compare
test results from old/new A-levels

21. Chemistry TESTA: ACKNOWLEDGEMENTS
Dr Jackie Potter, Matt Street and other members of the
Keele TESTA team
“That TESTA thing you presented at ViCE/PhEC a few
years ago still goes down as one of the most mystical
things I’ve seen”
https://twitter.com/EffinBirds, Copyright EffinBirds

22. Curriculum Review
2012/3 – 2016/7, development of new degree programmes
(SH/MChem)
Significant changes to all modules across the years and
creation of 4th year
Major Route Chem:
OLD – 65 summative assessments over 240 credits (in last
year of running)
NEW – 54 summative and 8 formative assessments over
240 credits (in 1st year of running)
A lot of variation since then

23. Transforming the Experience of Students Through
Assessment (TESTA)
Reflection on
Assessment
and Feedback
Student
Questionnaire
Student Focus
Group
Inventory of
Course

24. 0
10
20
30
40
50
60
70
Coursework Written Exam Practical
Exam
Other
%credits Broad Assessment Types
OLD NEW

25. 0
10
20
30
40
50
60
70
80
1st OLD 1st NEW 2nd
OLD
2nd
NEW
3rd OLD 3rd
NEW
%Credits
Coursework Written Exam Practical Exam
Chemistry SH

26. 3rd Year/Level 6 Assessment Types
NEWOLD

27. Shortcomings: Broad Brush
TESTA defines summative assessment as a point
at which marks are allocated.
judgement required where assignments have multiple
assessment points (e.g. Peer vs Tutor)
Many summative assessments have formative
elements
e.g. peer review of a draft laboratory report
Many assessments have several ways of awarding
marks
e.g. presentation with self, peer and tutor assessment
elements

28. Shortcoming
Utterly fails to require the timing of
assessments to be taken into consideration
Reflection on
Assessment
and Feedback
Student
Questionnaire
Student
Focus Group
Inventory of
Course
Timing of
Assessments

29. https://twitter.com/EffinBirds, Copyright EffinBirds

30. The Strange Case of the Spherical
Students

31. https://twitter.com/RissaChem/status/924794407341121537
[Accessed 30/10/17], quoted with permission

32. Approximations towards Ideality
The spherical student is:
- sleeping for 8 hours per 24 hour period
- attending all contact hours (14 – 20 hours/week)
- living off campus with an average commute (door to door)
of 30 minutes each way
- spending 2 hours in the preparation, consumption and
cleaning up of food and drink each day
- working, or engaging in other non-voluntary aspects of
life for around 16 hours per week

33. The average student (RBS Survey)
91.7 hours/month attending lectures, completing
coursework
27.4 hours/month socialising
14.6 hours/month in a part-time job
74% of students rate the stress of studying for their
degree as 6/10 or higher
http://personal.rbs.co.uk/personal/life-moments/student-living-index.html,
accessed 10/10/17

34. “There are many causes of student difficulty
such as stress, poor time management,
employment pressure while studying and life
issues to name a few. However, a major
cause is student workload.”
Bowyer, K., J. Journal of Higher Education and Policy Management, 2012, 34, 239 - 258

35. What is a Workload Model?
A method of evaluating time provided within the
curriculum for study.
May incorporate:
- contact time
- independent study
- quantity of work
- level of difficulty of work
- type and timing of assessments
- institutional resources (access to teachers, study space,
learning resources)
- student characteristics (ability, motivation, effort)
Bowyer, K., J. Journal of Higher Education and Policy Management, 2012, 34, 239 - 258

36. What is workload?
Objective workload
- depends on how much time the lecturer thinks is
appropriate for the ‘average student’
Subjective workload
- depends on students circumstances,
characteristics and ability

37. Impact of Teaching Methods
Type of session Contact Independent Study Pre-session
‘traditional lectures’ 1 1 limited
‘active sessions’ 1 2.5 fixed
workshops 1 2 limited
TBL 1 fixed/high
‘flipped sessions’ 1 fixed/high

38. The Keele Challenge
- Dual/Combined Honours
- Modular System
- Diversity of Teaching Styles
- Expectations of pre-/post- lecture engagement
- Level of challenge in content
- continual drive to BeMore/DoMore/build CV

39. Keele Metrics
4 x 15 credit modules per semester
1 credit = 10 hours of effort
600 hours spread over…
- 12 teaching weeks (effectively 11)
- 1 – 2 self-study weeks
- 2 examination weeks
- 2 – 3 vacation weeks
Expectation that students will work 35 – 40 hours per week
on their studies.

40. Assumptions in preliminary SWM
0. All time is spent in the service of assessment
1. assignments are ‘doable’ 2 weeks before the deadline
- availability of assessment guidelines
- coverage of key content
2. assessments are done in the week before and week of the
deadline
3. assessments take their ‘allocated hours’ (e.g. 10% of module is
15 hours of work)
- contentious, highly variable depending on whether assessment feeds
directly into exam or not
4. takes into account contact time, but falls short on pre-sessional
activities for ‘active learning’ sessions
- more thorough audit of teaching styles would be needed

41. Initial Model
Explains ‘anecdata’ well
- decline in 3rd year project lab attendance correlates to
weeks of high intensity
- minor changes to deadlines can redistribute the intensity
- formative assessment can have significant impact on
workload intensity
- understandable why students may not revise/submit for
purely formative tasks
- focus on 3rd year 16/17 and 17/18 where two
assignments were deliberately moved after viewing 16/17
data

42. Semester 1 16/17 Level 6 SH
4 x taught 15-credit modules + project
EXCLUDES: pre-session activities and revision unless for
formative assessment (CT – 4 hours)
0 20 40 60 80 100 120 140 160 180 200
2
3
4
5
6
7
8
9
10
11
12
% of a 40 hour working week
PROJECT TIME CONTACT SUMMATIVE FORMATIVE

43. Semester 1 17/18 Level 6 SH
4 x taught 15-credit modules + project + dissertation
EXCLUDES: pre-session activities and revision unless for
formative assessment (CT – 4 hours)
0 20 40 60 80 100 120 140
2
3
4
5
6
7
8
9
10
11
12
% of a 40 hour working week
PROJECT TIME CONTACT SUMMATIVE FORMATIVE

44. Next Steps
Expand TESTA to ‘snapshot’ 17/18 course.
Refine the underlying assumptions in the
workload model. Gain student input.