Leveraging data to improve your online course

1. Leveraging data to improve your online course
Presentation by
Michael Wilder

2. Agenda
•
•
•
•
•
•
What is learning analytics?
Research: past and present
What are the challenges and issues?
Case Study: Journalism 444
Conclusions
Questions

3. Definitions
What is learning analytics?
“Learning analytics refers to the interpretation of a
wide range of data produced by and gathered on behalf
of students in order to assess academic progress,
predict future performance, and spot potential issues.”
(Johnson, Adams, and Cummins, 2012)

4. Definitions
What is learning analytics?
“Learning analytics is the use of intelligent data,
learner-produced data, and analysis models to discover
information and social connections, and to predict and
advise on learning.”
(Siemens, 2010)

5. Definitions
What is learning analytics?
“Learning analytics is the measurement, collection,
analysis and reporting of data about learners and their
contexts, for purposes of understanding and optimizing
learning and the environments in which it occurs.”
(Society for Learning Analytics Research, 2012)

6. Definitions
What is learning analytics?
“LA collects and analyzes the ‘digital breadcrumbs’ that
students leave as they interact with various computer
systems to look for correlations between those
activities and learning outcomes.”
(Educause, 2011)

7. Process
Characterized as “an engine with five stages”
•
•
•
•
•
Capture
Report
Predict
Act
Refine
(Campbell, DeBlois, and Oblinger, 2007)

8. Adoption
Time-to-adoption horizon
The 2012 higher education NMC Horizon Report places
learning analytics in a two- to three-year adoption
window.
(Johnson, Adams, and Cummins, 2012)

9. Opportunities
Application
•
•
•
•
•
•
Predicting outcome achievement
Course and program dashboards
Curricular evaluation
Prioritizing learning outcomes
Set course and instructional policies
Defining academic quality
(Bach, 2011)

10. Research
Pre-existing areas of research
• Social network analysis
• Latent semantic analysis
• Dispositions analysis
(Ferguson, 2012)

11. Research
Newer areas of interest
•
•
•
•
•
Social learning analytics
Analytics for reflective learning
Visual analytics
Textual analytics
Analytics infrastructure
(Ferguson, 2012)

12. Research
Major research groups
• Educational data mining
“How can we extract value from these big sets of learning-related data?”
• Learning analytics
“How can we optimize opportunities for online learning?”
• Academic analytics
“How can we substantially improve learning opportunities and educational
results at national or international levels?”
(Ferguson, 2012)

13. Research
Predictors and indicators
• Activity indicators
(Number of logins, time spent in the learning environment)
• Performance indicators
(Grades, quiz scores)
• Dispositional indicators
(Age, GPA, prior learning experience, financial status)
• Student artifacts
(Essays, blog and discussion forum posts, media productions
(Brown, 2012)

14. Research
Example projects/research tools
•
•
•
•
•
The Signals project
Contextualized Attention Metadata (CAM)
LOCO-Analyst
SMILI Open Learner Modelling Framework
Social Networks Adapting Pedagogical Practice
(SNAPP)
• Gephi
(Ferguson, 2012)

15. Challenges
Learning analytics researchers will need to address:
•
•
•
•
Development of new tools, techniques, and people
Data: openness, ethics, and scope
Target of analytics activity
Connections to related fields
(Siemens, 2012)

16. Challenges
• What is an efficient way to visualize the data?
All at once or separate “dashboards?”
Wordle or Google Fusion Tables?
• How should interventions and responses be
implemented?
Fully automated by an LMS? Mediated by educator and
advisor?
(Brown, 2012)

17. Issues
LMS-centric
The initial model of gathering educational data focused on the
use of learning managements systems.
“Although the latest LMSs offer an increasing set of features,
students are beginning to reach the educational institution with
solid experience on how to interact with their peers in ways not
covered by the LMS.”
(Pardo and Kloos, 2011)

18. Issues
Proprietary software
“Early indications from vendors suggest that findings will be
treated as proprietary and will not be made available to other
researchers.”
• How are various factors weighted in an algorithm?
• Are the concepts being analyzed the rights ones?
• Can researchers adjust the algorithms of vendor tools to
conduct experiments of other factors that might impact
learning?
(Siemens, 2012)

19. Issues
Concerns about ethics and privacy
• Should students be told that their activity is being tracked?
• How much information should be provided to students,
faculty, parents, issuers of scholarships and others?
• How should faculty members react?
• Do students have an obligation to seek assistance?
(Ferguson, 2012)

20. Issues
Concerns about ethics and privacy
• What data is appropriate to collect about student? What
data is inappropriate?
• Who should be able to access the data and view results?
Which data should be reported anonymously?
• What is the impact of showing faculty modeling results? Do
any of the data bias faculty instruction and evaluation of
students?
(Bach, 2010)

21. Issues
Misuse/misinterpretation of data
• Who gets to interpret the data?
• Who owns the data?
• Who benefits from the data? The institution, the instructor,
or the student?
• Can data be use against instructors/departments/institutions
to determine employment, tenure or funding?
• Is time in an LMS representative of engagement? Simply
because educational content pages are open in a browser,
does it mean that a student is processing the information?

22. Case Study
Journalism 444
Advanced Interactive Media Design

23. Case Study
Goals of this study
The primary goal of this study is to explore ways that individual
instructors can use contemporary network data as well as
traditional feedback methods to iteratively improve the quality
of online course curriculum.
This study will use the data generated by the Summer 2012
offering of Journalism 444 as a case study.
No confidential data is shared in this study.

24. Case Study
Journalism 444
• Introduces digital media strategies to young journalists
transitioning from print media to the Internet
• Incorporates a variety of traditional Web development and
Web 2.0 technologies into the curriculum including:
• HTML
• Blogging
• Social media
• Web-based multimedia
• Provided for senior-level undergraduate journalism students

25. Case Study
Journalism 444
• Five-week course delivered between July 9, 2012 – August
12, 2012.
• Nine students participated in the course.
• Course used both institutional learning management system
(WebCT) as well as external Web server with traditional
HTML pages and WordPress blogging system.
• Modules/lessons were delivered through external HTML
pages.
• Announcements, assignments, discussions, and grade book
were delivered through LMS

26. Case Study
Assumptions
•
•
•
•
Educational data is highly contextual.
Data requires comparison over time in order to be valid.
The best person to interpret educational data is the educator.
Educators can improve the educational experience by:
• Improving the quality, scope and sequence of the
curriculum;
• Improving the interface of the learning environment;
• Understanding the behavior of students.

27. Case Study
Data-gathering opportunities
• Student tracking reports from the LMS
• Additional server log analysis for blogs and HTML pages via
Google Analytics
• Qualitative data from:
•
•
•
•
•
•
End-of-module surveys
End-of-course institutional evaluations
Student reflective journals
Student e-mails
Student discussion board posts
End-of-course surveys

28. Case Study
LMS Tracking Reports

29. Case Study
LMS tracking reports
• Summary of Activity
Provides an overview of general Student activity.
• Tool Usage
Provides an overview of how often tools, such as Assessments,
Assignments, or Discussions, are used.
• Course Item Usage
Provides an overview of how often items, such as a quiz, an assignment,
or a discussion topic, are used.

30. Case Study
LMS tracking reports
• Entry Page or Tool
Provides an overview of the pages or tools most frequently used as
course entry points.
• Exit Page or Tool
Provides an overview of the pages or tools most frequently used as
course exit points.
• File Usage
Provides an overview of the files viewed most frequently.
• Student Tracking
Provides a detailed summary of activity information for individual
Students.

31. LMS Reports
Summary of Activity Report: 1001 - 2012 Sumr
July 9, 2012 to August 12, 2012
Total user sessions: 510
Average user session length: 00:25:56
Average user sessions per day: 15
Average user sessions per day on weekdays: 14
Average user sessions per day on weekends:16
Most active day: July 16, 2012
Least active day: July 13, 2012
Most active hour of the day: 8:00 PM - 9:00 PM
Least active hour of the day: 5:00 AM - 6:00 AM

32. LMS Reports
Summary of Activity Report
Interpretation
• Most students log in to the LMS at least once a day with
some students logging in several times.
• Students log in about the same number of times during the
weekend as they do during the week.
• Students log in to the LMS the least and the most during the
first week of class.
• Students prefer to engage with the curriculum generally after
dinner, perhaps after work and perhaps after children have
been put to bed.

33. LMS Reports

34. LMS Reports
Tool usage
Interpretation
• The Weblinks tool is by far the most used LMS tool, which
makes sense since all curricular modules are linked outside
the system.
• The next most popular tools are the discussion, assignment,
folder, and mail tools.
• The Who’s Online tool is used the least and should probably
be removed from this course.

35. Case Study
Google Analytics

36. Google Analytics
Google Analytics provides:
•
•
•
•
•
•
•
•
•
Activity tracking outside of an LMS;
Traffic statistics over a defined period of time;
Visualization of traffic through a Web site (such as online curriculum)
Browser type
Operating system
Internet service provider
Access location (such as country and city)
Mobile access
More

37. Google Analytics

38. Google Analytics
Visit overview interpretation
• Course activity is slow during the first week of class, but picks up speed
rapidly during the second week.
• Activity at the end of the third week and during the fourth week is
volatile with spikes of heavy activity followed by valleys of low activity,
possibly indicating workload juggling between different courses.
• A slowing of activity during the fifth week mirrors a decreased workload
during the last week.

39. Google Analytics

40. Google Analytics
Visit flow interpretation
The bulk of the traffic flow is primarily within module three (“Introduction to
the Dreamweaver Interface”). This module introduces many new ideas to
students and has five lessons.

41. Google Analytics
Browser

42. Google Analytics
Browser interpretation
• Most of the students are using Safari, an Apple product, followed by
Firefox.
• Safari is also the default browser for IOS mobile devices.
• Android browsers are also present.
• Curriculum development should take into consideration limitations of
certain browser, as well as the potential for mobile learning.

43. Google Analytics
Browser

44. Google Analytics
City interpretation
Most of the traffic is coming from Las Vegas and Henderson (local to the
University), but a proportion of the traffic is coming from outside the local
area. This accurately reflects statements made by students indicating he or
she would be out of town during part of the semester.

45. Google Analytics
Browser

46. Google Analytics
Service provider interpretation
Only about an eighth of the traffic comes from on campus,
indicating that most students access this online course
remotely.

47. Google Analytics
Mobile devices

48. Google Analytics
Mobile devices

49. Google Analytics
Mobile access interpretation
• Around 10% of the traffic to the online curriculum is via
mobile devices.
• More than 50% of this traffic is via Apple iPhone or iPad.
• These figures justify further mobile access development.

50. Case Study
End-of-module surveys

51. End-of-module surveys
In order to gather qualitative student feedback that
can inform the improvement of online curriculum,
end-of-module surveys were provided.
During the summer 2012 semester, these surveys were
not mandatory.

52. End-of-module surveys
Following are the questions provided in each end-ofmodule survey:
•
•
•
•
•
•
The content of this unit was useful and relevant to the
objectives of this course.
The content of this unit was clear and organized.
The amount of time provided to accomplish the tasks
What elements of this unit did you find the most enjoyable?
What elements of this unit did you find the least enjoyable?
What suggestions could you give the instructor to improve
this unit?

53. End-of-module surveys
Example response:
What elements of this unit did you find the least enjoyable?
“Tracking down an interview along with doing with the rest of
the module was difficult to do in time.”
What suggestions could you give the instructor to improve this
unit?
“Make an announcement or better reminder for the interview.”

54. End-of-module surveys
Example response:
What elements of this unit did you find the least enjoyable?
“CSS was very difficult.”
What suggestions could you give the instructor to improve this
unit?
“The directions provided in the CSS didn't always match my
Dreamweaver. The directions shown weren't options in my
Dreamweaver. However, I don't know how to fix this but it
made it difficult and time consuming. “

55. End-of-module surveys
Example response:
What elements of this unit did you find the least enjoyable?
“This stuff is so confusing to me! I'm not the most tech savvy
person, and this is an online course so I'm sure that doesn't
help. Haha”
What suggestions could you give the instructor to improve this
unit?
“Nothing, I'm sure I'll get the hang of it.”

56. End-of-module surveys
End-of-module survey interpretation
• The “Professional Web Developer Interview” assignment may
need more scaffolding and time to complete.
• The module on Cascading Style Sheets will need more
development, with particular eye on compatibility between
versions of Dreamweaver.
• Students feel challenged in some modules, but still feel
confident they can continue.

57. Case Study
Reflective Journals

58. Reflective Journals
At the end of every semester, students are asked to reflect on
the course as a whole.
Following is the assignment prompt:
“Take a moment to think about some of the technologies and
strategies you've explored during this class. Write a discussion
post reflecting on your work this last semester. Identify any skills
as an instructional media designer that you've learned that
might be useful to you in your future career endeavors. Also
identify any online technologies that would be useful to learn in
the future.”

59. Reflective Journals
Example response:
“Throughout this class we have spent numerous hours on the computer
learning about technologies and strategies on how to create websites. I can
honestly say that in the beginning I dreaded doing homework for the class.
After seeing the final result and knowing that my hard work paid off I am
truly grateful that I had the opportunity to partake in this class. There are
those classes that you are just happy to get through and there are those
classes that you actually learn and take wisdom away from, this was one of
those classes.”

60. Reflective Journals
Example response:
“This class pushed me beyond my comfort zone, made me frustrated and
was incredibly time consuming - something my non-health science related
classes have not done here at UNLV. I’d say that it’s a good thing because
that’s what classes are supposed to do. Maybe it’s weird to say, but I’m used
to skating through the majority of my classes and winging about 99% of the
work, but that’s not actually possible in this one - and I’m glad because I feel
like I finished with a usable skill (even though I’m clearly a novice.)”

61. Reflective Journals
Example response:
“This class definitely reminded me of when I used to mess around with
codes on MySpace. I didn't know what they meant and I would just change
the sizes and make titles such as "About Me” on MySpace. Back then, it took
a long time to figure out what codes was changing what and what codes
made the font different. After taking this class, I understand what they
meant. I have learned so much in this class about HTML codes and CSS. I
really enjoyed editing my layout to how I wanted it to be. My layout was
what I wanted and it wasn’t premade. It was my own idea.”

62. Reflective Journals
Reflective journal interpretation
Although the course provided a rigorous academic framework,
student satisfaction was high. Students felt a sense of
accomplishment and that their work was relevant and useful for
their future endeavors.

63. Conclusions
• Learning analytics can be useful to the individual instructor in the
pursuit of online curricular improvement.
• Instructors can improve online curriculum by understanding the
context in which student engage with the environment (such as
time of day, heavy traffic patterns, browser type, mobile devices,
etc.).
• Instructors should not discount qualitative methods for learning
how to improve the scope, sequence, and efficiency of online
curriculum.
• Instructors should be very aware of the ethical and privacy issues of
gathering data.
• Teaching and learning is contextual, and the best person to
interpret data is the instructor.

64. Conclusions
A 2011 Educause Learning Initiative brief describes
learning analytics as “the coming third wave, a new
technology with great potential to increase student
academic success.”
(Brown, 2012)

65. Conclusions
Potential
“Theoretically, LA has the potential to dramatically impact the
existing models of education and to generate new insights into
what works and what does not work in teaching and learning.
The results are potentially transformative to all levels of today’s
education system.”
(Siemens, 2012)

66. References
Bach, C., 2010. Learning Analytics: Targeting Instruction, Curricula and Student Support. Office of the
Provost, Drexel University.
Campbell, J., DeBlois, P., and Oblinger, D., 2007. Academic analytics: A new tool for a new era.
Educause Review, 42(4): 40-57.
Educause (2011). 7 things you should know about first-generation learning analytics
Ferguson, R. (2012). The state of learning analytics in 2012: A review and future challenges.
Technical Report KMI-12-01, Knowledge Media Institute, The Open University, UK.
Pardo, A., and Kloos, C., 2011. Stepping out of the box: Towards analytics outside the learning
management system. LAK’11.
Siemens, G., 2012. Learning analytics: Envisioning a research discipline and a domain of practice.
LAK’12, 29.
Society for Learning Analytics Research, 2012. About. http://www.solaresearch.org/about/

67. Credits/Contacts
This presentation made by:
Michael Wilder
Instructional Design Coordinator
E-mail | Web Site