SlideShare uma empresa Scribd logo
1 de 31
Designing and Testing Visual
Representations of Draft Essays for
Higher Education Students
Denise Whitelock 
, Debora Field 
, Stephen Pulman
, John Richardson  ,
 Institute of Educational Technology – The
Open University
 Department of Computer Science –
University of Oxford
Learning Analytics and Student Feedback
• Assessment drives
learning (Rowntree,
1987)
• Communicating
assessment in a
meaningful way
• What, when and
How?
DMW LAK Workshop March 2014
SAFeSEA
Professor Denise Whitelock
Professor John Richardson
Professor Stephen Pulman
An automated
tool
supporting
online writing
and
assessment
of essays
providing
accurate
targeted
feedback
SAFeSEA: Supportive Automated Feedback for Short Essay
Answers http://www.open.ac.uk/researchprojects/safesea/
DMW LAK Workshop March 2014
DMW LAK Workshop March 2014
Grand Challenge representing analysis that
can be readily understood
Feedback to prompt Self Reflection
• Analysis must prompt
“Advice for Action”
• Self reflective discourse
with computer feedback
• Visual representation of
feedback can open a
discourse between tutor
and student
• Prompt peer to peer
discourse
DMW LAK Workshop March 2014
Don’t forget the Human Reaction
DMW LAK Workshop March 2014
DMW LAK Workshop March 2014
www.storiesabout.com
www.storiesabout.com/cr
eativepdp
c.mckillop@rgu.ac.uk
DMW LAK Workshop March 2014
Computer Feedback
• Non-
judgemental/objective
• Is not the tutor who
marks the final essay
• Feedback can be
accessed many times
• Can lead to contact with
tutor with more
questions
DMW LAK Workshop March 2014
Talk Back
• Checking
understanding by
‘talk back’
• Summaries in
OpenEssayist
• Key words = key
ideas
• http://www.open.ac.u
k/researchprojects/s
afesea/
DMW LAK Workshop March 2014
OpenEssayist: What it tells you
• The system’s focus is to present summaries of
students’ own work in different ways, to encourage
them to reflect constructively on what they have written.
• In other words Open Essayist tells them from its
analysis what are the most important or key points in
their essay. They can then think about whether that
was what they intended to emphasise in their essay.
If not then they can make the appropriate changes.
• A very important aspect of the OpenEssayist system is
that it will not tell students what to write, or how to
rewrite sections of their essay, or even what is correct
or incorrect in their essay.
DMW LAK Workshop March 2014
OpenEssayist: A tool for reflection
• Different views with distinct roles presented
• The purpose of this feedback is to encourage you
• to reflect on the draft text you submitted,
• to help you consider how your essay is
organised,
• how the key terms are being used across the
essay
• how they combine to form a cohesive discussion.
DMW LAK Workshop March 2014
OpenEssayist: How it gives feedback
• Three aspects of the students’ essays are analysed by
the system:
• the structure of the essay (which paragraphs
constitute the introduction, the conclusion, the
discussion sections, etc.),
• the key words and key phrases of their essay
(which are the most important words and phrases,
the ones that are most representative of the
essay's overall meaning)
• the key sentences of their essay (which are
whole sentences that are most representative of
the essay's overall meaning).
DMW LAK Workshop March 2014
ESSAY
GRAPHICS
ANALYSIS
Key word spring
DMW LAK Workshop March 2014
Chord diagram
DMW LAK Workshop March 2014
Key sentence adjacency matrix
DMW LAK Workshop March 2014
Middle Space
• Develop new
analytic methods
• Computational
• Representational
• Statistical
• Visualisation
DMW LAK Workshop March 2014
Visualisations as a Thinking Tool
• Visualising text (Bertin,
1981; Johnson et al,
1993)
• Free text visualisation
still problematic
• Do users need training
with visualisations?
• Convention vs.
Instruction?
DMW LAK Workshop March 2014
Short text for illustration of Rainbow
Diagrams
DMW LAK Workshop March 2014
Sentence graph of short text
DMW LAK Workshop March 2014
OU essay awarded high grade
DMW LAK Workshop March 2014
OU essay awarded low grade
DMW LAK Workshop March 2014
Stanford University Boothe Prize essay
DMW LAK Workshop March 2014
Pretend essay: 50 identical sentences
DMW LAK Workshop March 2014
Pretend essay: 10 identical paragraphs
DMW LAK Workshop March 2014
Participants for Empirical Study
Occupation University
Professor of Teaching & Learning OUC
Senior Lecturer Computer Science OUC
Administrator x 2 OUC
Professor OUC
Ph.D student x 6 OUC
Professor of Computer Science OUC
Vice Rector OUC
Senior Lecturer x 2 OU UK
Learning & Teaching Officer x 3 OU UK
Project Manager OU UK
OU student x 2 OU UK
Professor OU UK
Senior Administrator OU UK
Course Administrator OU UK
DMW LAK Workshop March 2014
Findings
DMW LAK Workshop March 2014
“One of the clues was talking about the way the
color nodes, that the red ones are at the end. If
they are a good connection with a good
explanation for each paragraph. They should be
connected together but the darker ones should
be in the middle. But you see the color groups
together so for me it automatically pulls my eye
to this page because all the colors are closer
together and more in the middle. So that would
be the student course assignment essay with
the highest mark.”
“You don’t show me
anybody’s text. You are
not revealing anyone
else’s essay. So
students cannot
plagiarize. But you are
saying ‘Look hang on,
this is the way this
essay connects
together.’ That’s what
telling a good story is
about this linking.”
Creating teaching and learning dialogues:
towards guided learning supported by
technology
• Learning to judge
• Providing
reassurance
• Providing a variety
of signposted routes
to achieve learning
goals
DMW LAK Workshop March 2014
Challenges
• Visualisation
• Social network analysis
• Communication and
collaboration,
• Discourse analytics
• Interpretation training
• What are the necessary
and sufficient conditions
for successful
implementation of
learning analytics?
DMW LAK Workshop March 2014
References
• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J. (2013) ‘OpenEssayist: Extractive
Summarisation & Formative Assessment of Free-Text Essays’. Workshop on Discourse-Centric Learning
Analytics, 3rd Conference on Learning Analytics and Knowledge (LAK 2013), Leuven, Belgium
• Whitelock, D. (2011) Activating Assessment for Learning: are we on the way with Web 2.0? In M.J.W. Lee & C.
McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching. IGI Global.
pp. 319–342.
• Field, D., Pulman, S., Van Labeke, N., Whitelock, D. and Richardson, J.T.E. 2013. Did I really mean that?
Applying automatic summarisation techniques to formative feedback. Proceedings of the 9th International
Conference Recent Advances in Natural Language Processing (Hissar, Bulgaria, Sep. 2013).
• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. and Richardson, J.T.E. 2013. What is my essay really
saying? Using extractive summarization to motivate reflection and redrafting. Proceedings of the AIED
Workshop on Formative Feedback in Interactive Learning Environments (Memphis, TN, Jul. 2013).
• Alden, B., Van Labeke, N., Field, D., Pulman, S., Richardson, J.T.E. and Whitelock, D. 2013. Using student
experience to inform the design of an automated feedback system for essay answers. Proceedings of the 2013
International Computer Assisted Assessment Conference (Southampton, UK, Jul. 2013).
• Field, D., Richardson, J.T.E., Pulman, S., Van Labeke, N. and Whitelock, D. 2013. Reflections on
characteristics of university students’ essays through experimentation with domain-independent natural
language processing techniques. Proceedings of the 2013 International Computer Assisted Assessment
Conference (Southampton, UK, Jul. 2013).
• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. and Richardson, J.T.E. 2013. OpenEssayist: extractive
summarisation and formative assessment of free-text essays. Proceedings of the 1st International Workshop
on Discourse-Centric Learning Analytics (Leuven, Belgium, Apr. 2013).
DMW LAK Workshop March 2014

Mais conteúdo relacionado

Mais procurados

A Review of Distributional models of word meaning (Lenci, 2018)
A Review of Distributional models of word meaning (Lenci, 2018)A Review of Distributional models of word meaning (Lenci, 2018)
A Review of Distributional models of word meaning (Lenci, 2018)Shuntaro Yada
 
Brief assgn 1 essay question
Brief   assgn 1 essay questionBrief   assgn 1 essay question
Brief assgn 1 essay questionsuehwa533
 
Assignment 1 - THE BRIEF
Assignment 1 - THE BRIEFAssignment 1 - THE BRIEF
Assignment 1 - THE BRIEFDarshiini Vig
 
English 1 january 2015
English 1 january 2015English 1 january 2015
English 1 january 2015Jian Leo
 
April 2015 assignment 1
April 2015 assignment 1April 2015 assignment 1
April 2015 assignment 1Xying Lee
 
English 1 january 2015
English 1 january 2015English 1 january 2015
English 1 january 2015Lin Hui
 
Writing a thesis_proposal
Writing a thesis_proposalWriting a thesis_proposal
Writing a thesis_proposalsiega
 
Semi structured interviews for education research
Semi structured interviews for education researchSemi structured interviews for education research
Semi structured interviews for education researchIan Willis
 
6.3.stiffler
6.3.stiffler6.3.stiffler
6.3.stifflerafacct
 
Scientific Presentations
Scientific PresentationsScientific Presentations
Scientific PresentationsKellyHester
 
8bb42b 3917665c9d764a16bfe41914dd206073
8bb42b 3917665c9d764a16bfe41914dd2060738bb42b 3917665c9d764a16bfe41914dd206073
8bb42b 3917665c9d764a16bfe41914dd206073isyroc
 
Essay question august2015
Essay question august2015Essay question august2015
Essay question august2015Ong Ke Qi
 

Mais procurados (18)

A Review of Distributional models of word meaning (Lenci, 2018)
A Review of Distributional models of word meaning (Lenci, 2018)A Review of Distributional models of word meaning (Lenci, 2018)
A Review of Distributional models of word meaning (Lenci, 2018)
 
Brief assgn 1 essay question
Brief   assgn 1 essay questionBrief   assgn 1 essay question
Brief assgn 1 essay question
 
Brief
BriefBrief
Brief
 
Assignment 1 - THE BRIEF
Assignment 1 - THE BRIEFAssignment 1 - THE BRIEF
Assignment 1 - THE BRIEF
 
English 1 january 2015
English 1 january 2015English 1 january 2015
English 1 january 2015
 
April 2015 assignment 1
April 2015 assignment 1April 2015 assignment 1
April 2015 assignment 1
 
ELG 30505 Essay 2 Brief
ELG 30505 Essay 2 BriefELG 30505 Essay 2 Brief
ELG 30505 Essay 2 Brief
 
ESSAY WRITING BRIEF
ESSAY WRITING BRIEFESSAY WRITING BRIEF
ESSAY WRITING BRIEF
 
Final paper guideline gradesheet
Final paper guideline gradesheetFinal paper guideline gradesheet
Final paper guideline gradesheet
 
English 1 january 2015
English 1 january 2015English 1 january 2015
English 1 january 2015
 
ENG PROJECT 2
ENG PROJECT 2ENG PROJECT 2
ENG PROJECT 2
 
Writing a thesis_proposal
Writing a thesis_proposalWriting a thesis_proposal
Writing a thesis_proposal
 
Brief
BriefBrief
Brief
 
Semi structured interviews for education research
Semi structured interviews for education researchSemi structured interviews for education research
Semi structured interviews for education research
 
6.3.stiffler
6.3.stiffler6.3.stiffler
6.3.stiffler
 
Scientific Presentations
Scientific PresentationsScientific Presentations
Scientific Presentations
 
8bb42b 3917665c9d764a16bfe41914dd206073
8bb42b 3917665c9d764a16bfe41914dd2060738bb42b 3917665c9d764a16bfe41914dd206073
8bb42b 3917665c9d764a16bfe41914dd206073
 
Essay question august2015
Essay question august2015Essay question august2015
Essay question august2015
 

Semelhante a Designing and testing visual representations of draft essays for Higher Education Students

Developing a web-based research app to support practice-based enquiry in teac...
Developing a web-based research app to support practice-based enquiry in teac...Developing a web-based research app to support practice-based enquiry in teac...
Developing a web-based research app to support practice-based enquiry in teac...The Higher Education Academy
 
Scaffolding Course Research Assignments to Incorporate Information Literacy S...
Scaffolding Course Research Assignments to Incorporate Information Literacy S...Scaffolding Course Research Assignments to Incorporate Information Literacy S...
Scaffolding Course Research Assignments to Incorporate Information Literacy S...nadinefanderson
 
Scaffolding Course Research Assignments to Incorporate Information Literacy S...
Scaffolding Course Research Assignments to Incorporate Information Literacy S...Scaffolding Course Research Assignments to Incorporate Information Literacy S...
Scaffolding Course Research Assignments to Incorporate Information Literacy S...nadinefanderson
 
Roosevelt questioning day 1
Roosevelt questioning day 1Roosevelt questioning day 1
Roosevelt questioning day 1Jennifer Evans
 
SITE presentation webinar semingson owens
SITE presentation webinar semingson owensSITE presentation webinar semingson owens
SITE presentation webinar semingson owensPeggy Semingson
 
Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?Denise Whitelock
 
Nudging Revision NWP 2
Nudging Revision NWP 2Nudging Revision NWP 2
Nudging Revision NWP 2shelbiewitte
 
Library Live: Collaborating Towards Heightened Information Literacy & Retenti...
Library Live: Collaborating Towards Heightened Information Literacy & Retenti...Library Live: Collaborating Towards Heightened Information Literacy & Retenti...
Library Live: Collaborating Towards Heightened Information Literacy & Retenti...Denise Woetzel
 
Gtn action research presentation slides1
Gtn action research presentation slides1Gtn action research presentation slides1
Gtn action research presentation slides1Laura Anaya-Graham
 
Reinvigorating a Library Workshop Series: Moving Workshops into the Online En...
Reinvigorating a Library Workshop Series: Moving Workshops into the Online En...Reinvigorating a Library Workshop Series: Moving Workshops into the Online En...
Reinvigorating a Library Workshop Series: Moving Workshops into the Online En...Mandi Goodsett
 
Achterman csla 2011reading_online
Achterman csla 2011reading_onlineAchterman csla 2011reading_online
Achterman csla 2011reading_onlinedachterman
 
Csla presentation reading online 2011
Csla presentation reading online 2011Csla presentation reading online 2011
Csla presentation reading online 2011dachterman
 
Curricular Unit Development - Understanding by Design
Curricular Unit Development - Understanding by DesignCurricular Unit Development - Understanding by Design
Curricular Unit Development - Understanding by Designdeliadec
 
2012 Spring Conference Program
2012 Spring Conference Program2012 Spring Conference Program
2012 Spring Conference ProgramLesley Roessing
 
Blended Reading Using Collaborative Online Annotation
Blended Reading Using Collaborative Online AnnotationBlended Reading Using Collaborative Online Annotation
Blended Reading Using Collaborative Online AnnotationDeclara, INC
 
Mapping the journey barcelona pp
Mapping the journey barcelona ppMapping the journey barcelona pp
Mapping the journey barcelona ppAndy Cramp
 

Semelhante a Designing and testing visual representations of draft essays for Higher Education Students (20)

DCLA14_Denise_Whitelock_slides
DCLA14_Denise_Whitelock_slidesDCLA14_Denise_Whitelock_slides
DCLA14_Denise_Whitelock_slides
 
Developing a web-based research app to support practice-based enquiry in teac...
Developing a web-based research app to support practice-based enquiry in teac...Developing a web-based research app to support practice-based enquiry in teac...
Developing a web-based research app to support practice-based enquiry in teac...
 
Scaffolding Course Research Assignments to Incorporate Information Literacy S...
Scaffolding Course Research Assignments to Incorporate Information Literacy S...Scaffolding Course Research Assignments to Incorporate Information Literacy S...
Scaffolding Course Research Assignments to Incorporate Information Literacy S...
 
Scaffolding Course Research Assignments to Incorporate Information Literacy S...
Scaffolding Course Research Assignments to Incorporate Information Literacy S...Scaffolding Course Research Assignments to Incorporate Information Literacy S...
Scaffolding Course Research Assignments to Incorporate Information Literacy S...
 
Content Area Centers
Content Area CentersContent Area Centers
Content Area Centers
 
Roosevelt questioning day 1
Roosevelt questioning day 1Roosevelt questioning day 1
Roosevelt questioning day 1
 
Demystifying Teaching and Learning Research process
Demystifying Teaching and Learning Research processDemystifying Teaching and Learning Research process
Demystifying Teaching and Learning Research process
 
SITE presentation webinar semingson owens
SITE presentation webinar semingson owensSITE presentation webinar semingson owens
SITE presentation webinar semingson owens
 
Belfast IFLS June 2016 Ppt Chris Banister
Belfast IFLS June 2016 Ppt Chris BanisterBelfast IFLS June 2016 Ppt Chris Banister
Belfast IFLS June 2016 Ppt Chris Banister
 
Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?
 
Nudging Revision NWP 2
Nudging Revision NWP 2Nudging Revision NWP 2
Nudging Revision NWP 2
 
Library Live: Collaborating Towards Heightened Information Literacy & Retenti...
Library Live: Collaborating Towards Heightened Information Literacy & Retenti...Library Live: Collaborating Towards Heightened Information Literacy & Retenti...
Library Live: Collaborating Towards Heightened Information Literacy & Retenti...
 
Gtn action research presentation slides1
Gtn action research presentation slides1Gtn action research presentation slides1
Gtn action research presentation slides1
 
Reinvigorating a Library Workshop Series: Moving Workshops into the Online En...
Reinvigorating a Library Workshop Series: Moving Workshops into the Online En...Reinvigorating a Library Workshop Series: Moving Workshops into the Online En...
Reinvigorating a Library Workshop Series: Moving Workshops into the Online En...
 
Achterman csla 2011reading_online
Achterman csla 2011reading_onlineAchterman csla 2011reading_online
Achterman csla 2011reading_online
 
Csla presentation reading online 2011
Csla presentation reading online 2011Csla presentation reading online 2011
Csla presentation reading online 2011
 
Curricular Unit Development - Understanding by Design
Curricular Unit Development - Understanding by DesignCurricular Unit Development - Understanding by Design
Curricular Unit Development - Understanding by Design
 
2012 Spring Conference Program
2012 Spring Conference Program2012 Spring Conference Program
2012 Spring Conference Program
 
Blended Reading Using Collaborative Online Annotation
Blended Reading Using Collaborative Online AnnotationBlended Reading Using Collaborative Online Annotation
Blended Reading Using Collaborative Online Annotation
 
Mapping the journey barcelona pp
Mapping the journey barcelona ppMapping the journey barcelona pp
Mapping the journey barcelona pp
 

Mais de Denise Whitelock

Technology Enhanced Assessment: Do we have a wolf in sheep's clothing?
Technology Enhanced Assessment: Do we have a wolf in sheep's clothing?Technology Enhanced Assessment: Do we have a wolf in sheep's clothing?
Technology Enhanced Assessment: Do we have a wolf in sheep's clothing?Denise Whitelock
 
Who has the crystal ball for moving forward with Digital Assessment?
Who has the crystal ball for moving forward with Digital Assessment?Who has the crystal ball for moving forward with Digital Assessment?
Who has the crystal ball for moving forward with Digital Assessment?Denise Whitelock
 
Learning Analytics and student feedback
Learning Analytics and student feedbackLearning Analytics and student feedback
Learning Analytics and student feedbackDenise Whitelock
 
Good pedagogical practice driving learning analytics: OpenMentor, Open Commen...
Good pedagogical practice driving learning analytics: OpenMentor, Open Commen...Good pedagogical practice driving learning analytics: OpenMentor, Open Commen...
Good pedagogical practice driving learning analytics: OpenMentor, Open Commen...Denise Whitelock
 
Understanding current practice around the Assessment of Multimedia Artefacts
Understanding current practice around the Assessment of Multimedia ArtefactsUnderstanding current practice around the Assessment of Multimedia Artefacts
Understanding current practice around the Assessment of Multimedia ArtefactsDenise Whitelock
 
Looking backwards to move forwards: Seminal research that has influenced key ...
Looking backwards to move forwards: Seminal research that has influenced key ...Looking backwards to move forwards: Seminal research that has influenced key ...
Looking backwards to move forwards: Seminal research that has influenced key ...Denise Whitelock
 
Technology-Enhanced Assessment and Feedback: How is evidence-based literature...
Technology-Enhanced Assessment and Feedback: How is evidence-based literature...Technology-Enhanced Assessment and Feedback: How is evidence-based literature...
Technology-Enhanced Assessment and Feedback: How is evidence-based literature...Denise Whitelock
 
Academics' Understanding of Authentic Assessment
Academics' Understanding of Authentic AssessmentAcademics' Understanding of Authentic Assessment
Academics' Understanding of Authentic AssessmentDenise Whitelock
 
Technology Enhanced Activities for Learning Science for Children in Hospital ...
Technology Enhanced Activities for Learning Science for Children in Hospital ...Technology Enhanced Activities for Learning Science for Children in Hospital ...
Technology Enhanced Activities for Learning Science for Children in Hospital ...Denise Whitelock
 
The challenges of Assessment and Feedback: findings from an HEA project
The challenges of Assessment and Feedback: findings from an HEA projectThe challenges of Assessment and Feedback: findings from an HEA project
The challenges of Assessment and Feedback: findings from an HEA projectDenise Whitelock
 
Supporting Science Studies for children with long term health problems using ...
Supporting Science Studies for children with long term health problems using ...Supporting Science Studies for children with long term health problems using ...
Supporting Science Studies for children with long term health problems using ...Denise Whitelock
 
Synthesis Report onAssessment and Feedback with Technology Enhancement (SRAFTE)
Synthesis Report onAssessment and Feedback with Technology Enhancement (SRAFTE)Synthesis Report onAssessment and Feedback with Technology Enhancement (SRAFTE)
Synthesis Report onAssessment and Feedback with Technology Enhancement (SRAFTE)Denise Whitelock
 
Dmw. E Assessmentlive July2009 Videos
Dmw. E Assessmentlive July2009 VideosDmw. E Assessmentlive July2009 Videos
Dmw. E Assessmentlive July2009 VideosDenise Whitelock
 
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...Denise Whitelock
 
Framing Feedback for Formative Assessment, Denise Whitelock
Framing Feedback for Formative Assessment, Denise WhitelockFraming Feedback for Formative Assessment, Denise Whitelock
Framing Feedback for Formative Assessment, Denise WhitelockDenise Whitelock
 

Mais de Denise Whitelock (16)

Technology Enhanced Assessment: Do we have a wolf in sheep's clothing?
Technology Enhanced Assessment: Do we have a wolf in sheep's clothing?Technology Enhanced Assessment: Do we have a wolf in sheep's clothing?
Technology Enhanced Assessment: Do we have a wolf in sheep's clothing?
 
Who has the crystal ball for moving forward with Digital Assessment?
Who has the crystal ball for moving forward with Digital Assessment?Who has the crystal ball for moving forward with Digital Assessment?
Who has the crystal ball for moving forward with Digital Assessment?
 
Learning Analytics and student feedback
Learning Analytics and student feedbackLearning Analytics and student feedback
Learning Analytics and student feedback
 
Good pedagogical practice driving learning analytics: OpenMentor, Open Commen...
Good pedagogical practice driving learning analytics: OpenMentor, Open Commen...Good pedagogical practice driving learning analytics: OpenMentor, Open Commen...
Good pedagogical practice driving learning analytics: OpenMentor, Open Commen...
 
Understanding current practice around the Assessment of Multimedia Artefacts
Understanding current practice around the Assessment of Multimedia ArtefactsUnderstanding current practice around the Assessment of Multimedia Artefacts
Understanding current practice around the Assessment of Multimedia Artefacts
 
Looking backwards to move forwards: Seminal research that has influenced key ...
Looking backwards to move forwards: Seminal research that has influenced key ...Looking backwards to move forwards: Seminal research that has influenced key ...
Looking backwards to move forwards: Seminal research that has influenced key ...
 
Technology-Enhanced Assessment and Feedback: How is evidence-based literature...
Technology-Enhanced Assessment and Feedback: How is evidence-based literature...Technology-Enhanced Assessment and Feedback: How is evidence-based literature...
Technology-Enhanced Assessment and Feedback: How is evidence-based literature...
 
Academics' Understanding of Authentic Assessment
Academics' Understanding of Authentic AssessmentAcademics' Understanding of Authentic Assessment
Academics' Understanding of Authentic Assessment
 
Technology Enhanced Activities for Learning Science for Children in Hospital ...
Technology Enhanced Activities for Learning Science for Children in Hospital ...Technology Enhanced Activities for Learning Science for Children in Hospital ...
Technology Enhanced Activities for Learning Science for Children in Hospital ...
 
The challenges of Assessment and Feedback: findings from an HEA project
The challenges of Assessment and Feedback: findings from an HEA projectThe challenges of Assessment and Feedback: findings from an HEA project
The challenges of Assessment and Feedback: findings from an HEA project
 
Supporting Science Studies for children with long term health problems using ...
Supporting Science Studies for children with long term health problems using ...Supporting Science Studies for children with long term health problems using ...
Supporting Science Studies for children with long term health problems using ...
 
Synthesis Report onAssessment and Feedback with Technology Enhancement (SRAFTE)
Synthesis Report onAssessment and Feedback with Technology Enhancement (SRAFTE)Synthesis Report onAssessment and Feedback with Technology Enhancement (SRAFTE)
Synthesis Report onAssessment and Feedback with Technology Enhancement (SRAFTE)
 
Calrg2010
Calrg2010Calrg2010
Calrg2010
 
Dmw. E Assessmentlive July2009 Videos
Dmw. E Assessmentlive July2009 VideosDmw. E Assessmentlive July2009 Videos
Dmw. E Assessmentlive July2009 Videos
 
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
 
Framing Feedback for Formative Assessment, Denise Whitelock
Framing Feedback for Formative Assessment, Denise WhitelockFraming Feedback for Formative Assessment, Denise Whitelock
Framing Feedback for Formative Assessment, Denise Whitelock
 

Último

BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...
BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...
BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...Sapna Thakur
 
Unit-IV- Pharma. Marketing Channels.pptx
Unit-IV- Pharma. Marketing Channels.pptxUnit-IV- Pharma. Marketing Channels.pptx
Unit-IV- Pharma. Marketing Channels.pptxVishalSingh1417
 
Introduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsIntroduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsTechSoup
 
Sanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfSanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfsanyamsingh5019
 
The basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxThe basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxheathfieldcps1
 
microwave assisted reaction. General introduction
microwave assisted reaction. General introductionmicrowave assisted reaction. General introduction
microwave assisted reaction. General introductionMaksud Ahmed
 
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK  LOOKBOOK(1) (1).pdfBASLIQ CURRENT LOOKBOOK  LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdfSoniaTolstoy
 
social pharmacy d-pharm 1st year by Pragati K. Mahajan
social pharmacy d-pharm 1st year by Pragati K. Mahajansocial pharmacy d-pharm 1st year by Pragati K. Mahajan
social pharmacy d-pharm 1st year by Pragati K. Mahajanpragatimahajan3
 
Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in DelhiRussian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhikauryashika82
 
Student login on Anyboli platform.helpin
Student login on Anyboli platform.helpinStudent login on Anyboli platform.helpin
Student login on Anyboli platform.helpinRaunakKeshri1
 
Software Engineering Methodologies (overview)
Software Engineering Methodologies (overview)Software Engineering Methodologies (overview)
Software Engineering Methodologies (overview)eniolaolutunde
 
Key note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdfKey note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdfAdmir Softic
 
Grant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingGrant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingTechSoup
 
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Sapana Sha
 
APM Welcome, APM North West Network Conference, Synergies Across Sectors
APM Welcome, APM North West Network Conference, Synergies Across SectorsAPM Welcome, APM North West Network Conference, Synergies Across Sectors
APM Welcome, APM North West Network Conference, Synergies Across SectorsAssociation for Project Management
 
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...fonyou31
 
Q4-W6-Restating Informational Text Grade 3
Q4-W6-Restating Informational Text Grade 3Q4-W6-Restating Informational Text Grade 3
Q4-W6-Restating Informational Text Grade 3JemimahLaneBuaron
 
The Most Excellent Way | 1 Corinthians 13
The Most Excellent Way | 1 Corinthians 13The Most Excellent Way | 1 Corinthians 13
The Most Excellent Way | 1 Corinthians 13Steve Thomason
 
Web & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdfWeb & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdfJayanti Pande
 
Interactive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communicationInteractive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communicationnomboosow
 

Último (20)

BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...
BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...
BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...
 
Unit-IV- Pharma. Marketing Channels.pptx
Unit-IV- Pharma. Marketing Channels.pptxUnit-IV- Pharma. Marketing Channels.pptx
Unit-IV- Pharma. Marketing Channels.pptx
 
Introduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsIntroduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The Basics
 
Sanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfSanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdf
 
The basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxThe basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptx
 
microwave assisted reaction. General introduction
microwave assisted reaction. General introductionmicrowave assisted reaction. General introduction
microwave assisted reaction. General introduction
 
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK  LOOKBOOK(1) (1).pdfBASLIQ CURRENT LOOKBOOK  LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
 
social pharmacy d-pharm 1st year by Pragati K. Mahajan
social pharmacy d-pharm 1st year by Pragati K. Mahajansocial pharmacy d-pharm 1st year by Pragati K. Mahajan
social pharmacy d-pharm 1st year by Pragati K. Mahajan
 
Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in DelhiRussian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
 
Student login on Anyboli platform.helpin
Student login on Anyboli platform.helpinStudent login on Anyboli platform.helpin
Student login on Anyboli platform.helpin
 
Software Engineering Methodologies (overview)
Software Engineering Methodologies (overview)Software Engineering Methodologies (overview)
Software Engineering Methodologies (overview)
 
Key note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdfKey note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdf
 
Grant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingGrant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy Consulting
 
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
 
APM Welcome, APM North West Network Conference, Synergies Across Sectors
APM Welcome, APM North West Network Conference, Synergies Across SectorsAPM Welcome, APM North West Network Conference, Synergies Across Sectors
APM Welcome, APM North West Network Conference, Synergies Across Sectors
 
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
 
Q4-W6-Restating Informational Text Grade 3
Q4-W6-Restating Informational Text Grade 3Q4-W6-Restating Informational Text Grade 3
Q4-W6-Restating Informational Text Grade 3
 
The Most Excellent Way | 1 Corinthians 13
The Most Excellent Way | 1 Corinthians 13The Most Excellent Way | 1 Corinthians 13
The Most Excellent Way | 1 Corinthians 13
 
Web & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdfWeb & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdf
 
Interactive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communicationInteractive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communication
 

Designing and testing visual representations of draft essays for Higher Education Students

  • 1. Designing and Testing Visual Representations of Draft Essays for Higher Education Students Denise Whitelock  , Debora Field  , Stephen Pulman , John Richardson  ,  Institute of Educational Technology – The Open University  Department of Computer Science – University of Oxford
  • 2. Learning Analytics and Student Feedback • Assessment drives learning (Rowntree, 1987) • Communicating assessment in a meaningful way • What, when and How? DMW LAK Workshop March 2014
  • 3. SAFeSEA Professor Denise Whitelock Professor John Richardson Professor Stephen Pulman An automated tool supporting online writing and assessment of essays providing accurate targeted feedback SAFeSEA: Supportive Automated Feedback for Short Essay Answers http://www.open.ac.uk/researchprojects/safesea/ DMW LAK Workshop March 2014
  • 4. DMW LAK Workshop March 2014 Grand Challenge representing analysis that can be readily understood
  • 5. Feedback to prompt Self Reflection • Analysis must prompt “Advice for Action” • Self reflective discourse with computer feedback • Visual representation of feedback can open a discourse between tutor and student • Prompt peer to peer discourse DMW LAK Workshop March 2014
  • 6. Don’t forget the Human Reaction DMW LAK Workshop March 2014
  • 7. DMW LAK Workshop March 2014 www.storiesabout.com www.storiesabout.com/cr eativepdp c.mckillop@rgu.ac.uk
  • 8. DMW LAK Workshop March 2014
  • 9. Computer Feedback • Non- judgemental/objective • Is not the tutor who marks the final essay • Feedback can be accessed many times • Can lead to contact with tutor with more questions DMW LAK Workshop March 2014
  • 10. Talk Back • Checking understanding by ‘talk back’ • Summaries in OpenEssayist • Key words = key ideas • http://www.open.ac.u k/researchprojects/s afesea/ DMW LAK Workshop March 2014
  • 11. OpenEssayist: What it tells you • The system’s focus is to present summaries of students’ own work in different ways, to encourage them to reflect constructively on what they have written. • In other words Open Essayist tells them from its analysis what are the most important or key points in their essay. They can then think about whether that was what they intended to emphasise in their essay. If not then they can make the appropriate changes. • A very important aspect of the OpenEssayist system is that it will not tell students what to write, or how to rewrite sections of their essay, or even what is correct or incorrect in their essay. DMW LAK Workshop March 2014
  • 12. OpenEssayist: A tool for reflection • Different views with distinct roles presented • The purpose of this feedback is to encourage you • to reflect on the draft text you submitted, • to help you consider how your essay is organised, • how the key terms are being used across the essay • how they combine to form a cohesive discussion. DMW LAK Workshop March 2014
  • 13. OpenEssayist: How it gives feedback • Three aspects of the students’ essays are analysed by the system: • the structure of the essay (which paragraphs constitute the introduction, the conclusion, the discussion sections, etc.), • the key words and key phrases of their essay (which are the most important words and phrases, the ones that are most representative of the essay's overall meaning) • the key sentences of their essay (which are whole sentences that are most representative of the essay's overall meaning). DMW LAK Workshop March 2014
  • 15. Key word spring DMW LAK Workshop March 2014
  • 16. Chord diagram DMW LAK Workshop March 2014
  • 17. Key sentence adjacency matrix DMW LAK Workshop March 2014
  • 18. Middle Space • Develop new analytic methods • Computational • Representational • Statistical • Visualisation DMW LAK Workshop March 2014
  • 19. Visualisations as a Thinking Tool • Visualising text (Bertin, 1981; Johnson et al, 1993) • Free text visualisation still problematic • Do users need training with visualisations? • Convention vs. Instruction? DMW LAK Workshop March 2014
  • 20. Short text for illustration of Rainbow Diagrams DMW LAK Workshop March 2014
  • 21. Sentence graph of short text DMW LAK Workshop March 2014
  • 22. OU essay awarded high grade DMW LAK Workshop March 2014
  • 23. OU essay awarded low grade DMW LAK Workshop March 2014
  • 24. Stanford University Boothe Prize essay DMW LAK Workshop March 2014
  • 25. Pretend essay: 50 identical sentences DMW LAK Workshop March 2014
  • 26. Pretend essay: 10 identical paragraphs DMW LAK Workshop March 2014
  • 27. Participants for Empirical Study Occupation University Professor of Teaching & Learning OUC Senior Lecturer Computer Science OUC Administrator x 2 OUC Professor OUC Ph.D student x 6 OUC Professor of Computer Science OUC Vice Rector OUC Senior Lecturer x 2 OU UK Learning & Teaching Officer x 3 OU UK Project Manager OU UK OU student x 2 OU UK Professor OU UK Senior Administrator OU UK Course Administrator OU UK DMW LAK Workshop March 2014
  • 28. Findings DMW LAK Workshop March 2014 “One of the clues was talking about the way the color nodes, that the red ones are at the end. If they are a good connection with a good explanation for each paragraph. They should be connected together but the darker ones should be in the middle. But you see the color groups together so for me it automatically pulls my eye to this page because all the colors are closer together and more in the middle. So that would be the student course assignment essay with the highest mark.” “You don’t show me anybody’s text. You are not revealing anyone else’s essay. So students cannot plagiarize. But you are saying ‘Look hang on, this is the way this essay connects together.’ That’s what telling a good story is about this linking.”
  • 29. Creating teaching and learning dialogues: towards guided learning supported by technology • Learning to judge • Providing reassurance • Providing a variety of signposted routes to achieve learning goals DMW LAK Workshop March 2014
  • 30. Challenges • Visualisation • Social network analysis • Communication and collaboration, • Discourse analytics • Interpretation training • What are the necessary and sufficient conditions for successful implementation of learning analytics? DMW LAK Workshop March 2014
  • 31. References • Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J. (2013) ‘OpenEssayist: Extractive Summarisation & Formative Assessment of Free-Text Essays’. Workshop on Discourse-Centric Learning Analytics, 3rd Conference on Learning Analytics and Knowledge (LAK 2013), Leuven, Belgium • Whitelock, D. (2011) Activating Assessment for Learning: are we on the way with Web 2.0? In M.J.W. Lee & C. McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching. IGI Global. pp. 319–342. • Field, D., Pulman, S., Van Labeke, N., Whitelock, D. and Richardson, J.T.E. 2013. Did I really mean that? Applying automatic summarisation techniques to formative feedback. Proceedings of the 9th International Conference Recent Advances in Natural Language Processing (Hissar, Bulgaria, Sep. 2013). • Van Labeke, N., Whitelock, D., Field, D., Pulman, S. and Richardson, J.T.E. 2013. What is my essay really saying? Using extractive summarization to motivate reflection and redrafting. Proceedings of the AIED Workshop on Formative Feedback in Interactive Learning Environments (Memphis, TN, Jul. 2013). • Alden, B., Van Labeke, N., Field, D., Pulman, S., Richardson, J.T.E. and Whitelock, D. 2013. Using student experience to inform the design of an automated feedback system for essay answers. Proceedings of the 2013 International Computer Assisted Assessment Conference (Southampton, UK, Jul. 2013). • Field, D., Richardson, J.T.E., Pulman, S., Van Labeke, N. and Whitelock, D. 2013. Reflections on characteristics of university students’ essays through experimentation with domain-independent natural language processing techniques. Proceedings of the 2013 International Computer Assisted Assessment Conference (Southampton, UK, Jul. 2013). • Van Labeke, N., Whitelock, D., Field, D., Pulman, S. and Richardson, J.T.E. 2013. OpenEssayist: extractive summarisation and formative assessment of free-text essays. Proceedings of the 1st International Workshop on Discourse-Centric Learning Analytics (Leuven, Belgium, Apr. 2013). DMW LAK Workshop March 2014

Notas do Editor

  1. TODO: finding the right LOGO for Primma