Kendra M. L. Cooper and Hassan Khosravi Proceeding of the 8th International Learning Analytics and Knowledge (LAK) Conference

1. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 1
Graph-based Visual Topic Dependency Models
Supporting Assessment Design and Delivery at Scale
LAK 2018
Hassan Khosravi
University of Queensland
h.khosravi@uq.edu.au
@haskhosravi
Kendra M. L. Cooper
Independent Scholar
kendra.m.cooper@gmail.com
@kendramlcooper

2. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 2
Introduction and Related Work
Two-Weighted Graphs
The TDM Collection
Prototype and Evaluation
Conclusions and Future Work

3. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 3
Introduction
• Educational environments continue to rapidly evolve to
address the needs of diverse, growing student populations.
http://trinitynews.ie/teaching-methods-in-science-from-the-dark-ages/ https://campusdevelopments.leeds.ac.uk/portfolio-item/lecture-theatre-redesign-project/
http://blogs.bmj.com/bjsm/2016/01/27/february-10th-re-launch-of-hugely-popular-bodymatters-massive-open-online-course/http://www.staloysiusla.org/academics/lmu-ideal-institute-2/

4. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 4
Introduction
http://trinitynews.ie/teaching-methods-in-science-from-the-dark-ages/ https://campusdevelopments.leeds.ac.uk/portfolio-item/lecture-theatre-redesign-project/
http://blogs.bmj.com/bjsm/2016/01/27/february-10th-re-launch-of-hugely-popular-bodymatters-massive-open-online-course/http://www.staloysiusla.org/academics/lmu-ideal-institute-2/
Although environments change, challenges surrounding design of
high quality assessments and providing meaningful feedback persist.

5. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 5
Practical Motivation
Administrators
How can we compare the coverage and difficulty of formal assessments across different
offerings of a course?
How can we compare students’ outcomes across different offerings of a course?
Teachers
How can we provide personalized feedback at scale?
How can we create assessments that are on par with assessments from previous offerings?
Administrators
Students
How can we estimate the importance of different topics in formal assessments.
How can we Infer our strengths and weaknesses within a course or module?
Educational Researchers
How can we measure learning gain to determine the effectiveness of different methods?

6. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 6
Research Motivation
• Predicting student academic performance has been an
important research topic in intelligent tutoring systems.
Most of the current algorithms (e.g., Bayesian Knowledge Tracing (BKT),
Additive Factors Models (AFM), Performance Factors Analysis (PFA)) present
student knowledge in the form of independent topics/ knowledge
components.

7. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 7
Visual Analytics and Learning Dashboards
• Present a large amount of data using visualizations.
• The main focus has been on LMS logs and engagement level.
• A variety of visual analytics have been used (Open learner models)
• Lack of availability of visual analytics designed for assessment.

8. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 8
Introduction and Related Work
Two-Weighted Graphs
The TDM Collection
Prototype and Evaluation
Future Work

9. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 9
Formal Definition of Two-Weighted Graphs
• A two-weighted, undirected graph G = (V, E)
– V is the set of vertices representing the topics/ knowledge
components.
– An edge e ∈ E is represented as e = (v, w, c1, c2)
• v and w are vertices (topics) being connected
• c1 represents coverage
• c2 represents achievements.
GRef = (V, E)
Topic C
Topic D
Topic BTopic A
E = (e1, e2, e3, … , e15}
Edge width indicates coverage
• wider is higher
Edge colour indicates achievement
• red to green (greener is higher)

10. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 10
Introduction and Related Work
Two-Weighted Graphs
The TDM Collection
Prototype and Evaluation
Future Work

11. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 11
The Topic Dependency Model Collection
Stakeholders
•select visualization to use • select assessment data to use
Topic Dependency Model Applications
Assessment Data Input
•current, historical
•formative, summative
Homework Assignments
Question Repositories Within a class
Examinations
Quizzes
…
Laboratories
Static Classroom Level Visualisation Models
Cross-class comparison
vs.
e.g., Explore achievements on a class assignment e.g., Compare coverage across two examinations
Define the scope for course assessments
• used by all classes offered for the course
• topics and dependencies covered
• historical levels of achievements
Course Assessment Reference Model
Dynamic Classroom Level Visualisation Models
e.g., Identify assessment achievement trends
for a question repository over time
Program
Administrator
Course
Designer
Course
Co-ordinator
Instructor Student
Educational
Researcher
Within a class

12. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 12
Course Reference Model
• Define required topics and dependencies.
• Used by all classes that run for this course.
• e.g., CS100
Fundamentals
Conditionals
Loops
ArraysFunctions
File I/O
Introduction
Course: Computer Science Introduction to Programming (CS100)
Visual Assessment Reference Model (stakeholders’ view):
Edge width indicates coverage
- wider is higher
Edge colour indicates achievement
-red to green (greener is higher)

13. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 13
Static Classroom Visualisation Models (1)
within a class
Fundamentals
Conditionals
Loops
ArraysFunctions
File I/O
Introduction
Assignment #3 (T. Taylor)
Fundamentals
Conditionals
Loops
ArraysFunctions
File I/O
Introduction
Assignment #3 (cohort)
Visual Achievement Model (stakeholders’ view):
Course: Computer Science Introduction to Programming (CS100)
Achievements Assignment #3 2016 Term 1 Section 001
Student T. Taylor id. 12345
Cohort
Edge colour indicates achievement
- red to green (greener is higher)
Edge width indicates coverage
- wider is higher

14. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 14
Static Classroom Visualisation Models (2)
(1 or more classes)
Assignment #3 (Class 1)
Fundamentals
Conditionals
Loops
ArraysFunctions
File I/O
Introduction
Fundamentals
Conditionals
Loops
ArraysFunctions
File I/O
Introduction
Assignment #3 (Class 2)
Visual Compare Achievement Model (stakeholders’ view):
Course: Computer Science Introduction to Programming (CS100)
Achievements:
Assignment #3 2016 Term 1 Section 001
Assignment #3 2015 Term 1 Section 002
Edge colour indicates achievement
- red to green (greener is higher)
Edge width indicates coverage
- wider is higher

15. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 15
Dynamic Classroom Visualisation Models
(1 class)
Fundamentals
Conditionals
Loops
ArraysFunction
s
File I/O
Introduction
T1
… T2
… Tn
Fundamentals
Conditionals
Loops
ArraysFunction
s
File I/O
Introduction
Fundamentals
Introduction
Conditionals
Loops
Function
s
File I/O
Arrays
Course: Computer Science Introduction to Programming (CS100)
Assessment material: 2016 Term 1 Section 001 Assignments #1- #5 Cohort Author B. Baker, C. Cooper
Visual Achievement Progression Model (stakeholders’ view):

16. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 16
Introduction
Graph Foundation of the TDMs
The TDM Collection
Prototype and Evaluation
Future Work

17. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 17
Topic Dependency Model Generator
https://hkhosrav.github.io/Topic-Dependency-Models/

18. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 18
Historical Data Study (PeerWise)
• Introductory course in C programming for
engineering students, APSC 160, at UBC:
• Data
• 377 students
• 1111 questions
• 21432 responses
• 10 topics

19. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 19
TDM Visualisation Example (PeerWise Data)

20. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 20
Introduction
Graph Foundation of the TDMs
The TDM Collection
Prototype and Evaluation
Future Work

21. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 21
Future Work
• Explore the usability trade-offs in adopting alternative
graph definitions and visualisations including:
– hyper-graphs to support 1 to n topic dependencies
(overcome the current limitation of maximum 2 topics).
– hierarchical graphs to support multiple levels of abstraction
(i.e., the decomposition of topics from program level down to
course level learning objectives).
• Integrate the two-weighted graphs within a comprehensive
educational learning dashboard called RiPPLE.
RiPPLE is an open source, adaptive, student-facing learning research
platform that provides:
1. Knowledge tracing
2. Content recommendation
3. Peer recommendation

22. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 22
TDMs in RiPPLE
https://hkhosrav.github.io/RiPPLE-Core/?demoStudent=true#/question/answer

23. Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale Page 23
Thank you!
Hassan Khosravi
University of Queensland
h.khosravi@uq.edu.au
@haskhosravi
Kendra M. L. Cooper
Independent Scholar
kendra.m.cooper@gmail.com
@kendramlcooper