1. Evaluation of the
Current Learning
Management
System at JMU
The Students’
Perspective on
Blackboard
Presented By
Eric Jacobs
Michael May
Jason Poole
Jessica Robinson
2. Introduction
This evaluation plan was prepared by Eric Jacobs, Michael
May, Jason Poole, and Jessica Robinson who are graduate
Introduction students in the Educational Technology department at
Review of Evaluation James Madison University. Coordinating efforts by
Process
Andrea Adams of the Center for Instructional Technology
(CIT), and by Dr. Michele Estes of the Educational
Technology department at James Madison University
have helped advise this evaluation plan.
Background
Blackboard is a learning management system that has
the ability to act as the central hub for the curriculum.
This has become increasingly apparent as distance
learning programs depend on the features of this
evaluand to manage course content and interact with
students in a web-based environment. Blackboard’s
diffusion wide, claiming more than 1,600 institutions
using their LMS (Blackboard, 2011).
3. Purpose
Purpose of • To determine the needs that students have for
evaluation learning management system.
Review of Evaluation
Process • To identify the technologies that faculty and students
are currently using in teaching and learning
• To identify the technologies that faculty and students
are no long using in teaching and learning.
Stakeholders
• Primary Stakeholders
• Andrea Adams
• Dr. Michele Estes
• Secondary Stakeholders
• Evaluation team
• Students of JMU
4. Decisions to be addressed
Purpose of • To what degree does Blackboard meet the
evaluation expectations and learning needs of undergraduate
Review of Evaluation
and graduate students at JMU?
Process
• Do graduate and undergraduate students feel that
Blackboard is a viable solution as a learning
management system for JMU over the next 5 years?
• To what degree do the features in Blackboard meet
the technological needs of the students at JMU?
5. Needs related
Evaluation What are students needs in regards to using a learning management
system in their coursework?
questions
What expectations do students have about learning management
Review of Evaluation
Process systems?
Technology related
What technologies are students using both in Blackboard and in their
day to day lives?
Do students feel that the technology in Blackboard is beneficial to their
learning?
Features related
What features of Blackboard do students find obsolete or in need of
updating?
What features in Blackboard do students engage with most often?
What features in Blackboard do students feel are the most beneficial?
7. Sample for census survey
• Students in high Blackboard use
courses
The Sample • 2,000 students or 10% of JMU
Review of Evaluation enrollment
Process • Expected return rate 50%
Sample for census survey
• ~20,000 students
• Expected return rate ~2%
Sample for interviews
• 2 undergraduate
• 2 graduate
• Purposive sample of Ed Tech Students
8. Instruments Interviews
Review of Evaluation
Process • Recorded using Elluminate Live!
Surveys
• Created and administered using
Qualtrics.
9. Limitations
• Time Constraints
• Return rate of survey data
Evaluation plan
• Proximity of group members from JMU and each other
Review of Evaluation
Process
Logistics & Timeline
• Interviews and survey data completed in November
Budget
10. Interview
Summary #1
Positives
-
• Access to grades
• The more information on Blackboard
the better
• E-mails of announcements
Interviewee Data • Organization of content
• Discussion boards
Gender: Female
Program of Study:
Communications Major, Negatives
Educational Media Minor
• Announcements when viewed on
Class: Junior
Blackboard
General Info: Student has • Instructors that don’t update
taken hybrid courses and is
currently enrolled in an online Blackboard beyond syllabus
course. Experience using
Blackboard in high school.
11. Interview
Summary #2
Positives
-
• Internal Blackboard email tools
• Access to everything in central location
Interviewee Data
Gender: Female
Program of Study:
Educational Technology Negatives
Class: Graduate • Dislike wiki feature
General Info: All classes
were taken on Blackboard.
12. Interview
Summary #3
Positives
-
• All important documents in one place
• Online grade book
• Roster for contacting classmates
Interviewee Data
Gender: Female
Program of Study: Middle
School Education Negatives
Class: Junior • Lots of features she doesn’t know how
General Info: Overall positive to use
experience with Blackboard • Inconsistency among JMU faculty
13. Interview
Summary #4 Positives
• Blackboard and Elluminate are perfect
coupling
-
• Greater connection with course and
instructor
• E-mail features, discussion board,
grade feature
Interviewee Data • Links and Resources
Gender: Female
Program of Study:
Educational Technology Negatives
Class: Graduate • Instructor implementation
General Info: Overall positive • Instructor Usage
experience using • Collaboration -- Real-time lacking
Blackboard. Almost all
courses using Blackboard. • Group sharing is “clunky”
• Synchronous info sharing (IM, chat)
14. 1. Please select one: Male or Female
Gender
Census Survey
Data
# Answer Response %
1 Male 57 27%
2 Female 151 73%
Total 208 100%
16. 3. Please select one: Program of Study
Census Survey
Data
Census Survey
Data
# Answer Response %
1 College of Buisness 25 12%
2 College of Communications 14 7%
3 College of Education 29 14%
4 College of Health and Human Services 31 15%
5 College of Humanities and Social Sciences 44 21%
6 College of Science, Technology, Engineering 47 23%
and Mathematics
7 College of Visual and Performing Arts 8 4%
8 Undecided 10 5%
Total 208 100%
17. 4. Based on your course work at JMU, please describe
Course your course interactions with Blackboard.
interactions with
Blackboard
Census Survey
Data
# Answer Response %
1 I have taken a fully online 57 27%
course that used Blackboard
2 I have taken a hybrid/blended 54 26%
course that meets online and
in the classroom that used
Blackboard
3 I have taken face-to-face 191 92%
classroom courses that
utilized Blackboard
4 I have taken face-to-face 125 60%
classroom courses that do not
utilize Blackboard
18. 5. Which technologies play a role in your daily communications?
Communication
Tools
Census Survey
Data
# Answer Response %
1 social networks 177 85%
2 text messaging 196 94%
3 instant messaging 53 25%
4 email 200 96%
5 other 19 9%
19. 6. What technologies do you regularly use?
Regularly used
technologies
Census Survey
Data
# Answer Response %
1 smartphone 109 55%
2 tablet 14 7%
3 laptop/desktop 194 98%
4 Other 29 15%
20. 7. Would you like to see Blackboard have more
Would like social social features for learning?
features in
Blackboard
Census Survey
Data
# Answer Response %
1 yes 64 32%
2 no 57 29%
3 unsure 76 39%
Total 197 100%
21. 8. Blackboard fits my expectations of how a
learning management system should work.
Blackboard fits
expectations
Census Survey
Data
# Answer Response %
1 Strongly Disagree 4 2%
2 Disagree 11 6%
3 Neither Agree nor 46 23%
Disagree
4 Agree 114 58%
5 Strongly Agree 22 11%
Total 197 100%
22. 9. Overall, how would you describe your
experience using Blackboard?
Overall experience
with Blackboard
Census Survey
Data
# Answer Response %
1 negative 3 2%
2 somewhat negative 15 8%
3 neutral 30 15%
4 somewhat positive 89 45%
5 positive 60 30%
Total 197 100%
23. 10. Overall, how difficult is Blackboard to use?
How difficult is
Blackboard
Census Survey
Data
# Answer Response %
1 Very Difficult 2 1%
2 Somewhat Difficult 20 10%
3 Somewhat Easy 113 57%
4 Very Easy 62 31%
Total 197 100%
24. 11. Please describe your satisfaction level of
Satisfaction with Blackboard as a reliable service.
Blackboard
reliability
Census Survey
Data
# Answer Response %
1 Very Dissatisfied 2 1%
2 Dissatisfied 13 7%
3 Neutral 33 17%
4 Satisfied 122 62%
5 Very Satisfied 26 13%
Total 196 100%
25. 12. Please describe the difficulty in locating
information within Blackboard.
Difficulty locating
info in Blackboard
Census Survey
Data
# Answer Response %
1 Very Difficult 2 1%
2 Difficult 24 12%
3 Neutral 62 32%
4 Easy 90 46%
5 Very Easy 18 9%
Total 196 100%
26. 13. How often do you access Blackboard?
How often do you
access Blackboard
Census Survey
Data
# Answer Response %
1 Never 0 0%
2 Less than Once a Month 0 0%
3 Once a Month 3 2%
4 2-3 Times a Month 3 2%
5 Once a Week 11 6%
6 2-3 Times a Week 72 37%
7 Daily 107 55%
Total 196 100%
27. 14. Please indicate the importance of the listed
features for a learning management system
Importance of
features
Census Survey
Data
# Question Not Very Not Important Very Responses Mean
Important Important Important
1 Email 16 13 80 78 187 3.18
2 Course 2 5 64 116 187 3.57
Annoucem
ents
3 Social 51 86 42 8 187 2.04
Media
28. 15. Please indicate the importance of the listed
features for a learning management system
Importance of
features
Census Survey
Data
# Question Not Very Not Important Very Responses Mean
Important Important Important
1 Discussion 16 33 94 44 187 2.89
Boards
2 Blogs 34 101 47 5 187 2.12
3 Wiki 41 96 42 8 187 2.09
29. 16. Please indicate the importance of the listed
features for a learning management system
Importance of
features
Census Survey
Data
# Question Not Very Not Important Very Responses Mean
Important Important Important
1 My Grades 2 1 22 162 187 3.84
2 Test 5 18 58 106 187 3.42
3 Rubrics and 1 10 59 117 187 3.56
Learning
Outcomes
30. 17. Please indicate the importance of the listed
features for a learning management system
Importance of
features
Census Survey
Data
# Question Not Very Not Important Very Responses Mean
Important Important Important
1 Assignment 2 1 41 143 187 3.74
s
2 Calendar 6 33 72 76 187 3.17
3 Groups 13 52 90 32 187 2.75
4 Journals 26 79 57 25 187 2.43
5 Learning 16 61 76 34 187 2.68
Module
6 Survey 22 72 66 27 187 2.52
7 Podcasts 36 83 52 15 186 2.25
31. 18. Which features do you access MOST often in Blackboard?
Most often
accessed features
Census Survey
Data
32. 18. Which features do you access MOST often in Blackboard?
Most often
accessed features
Census Survey
Data
33. 19. Which features do you access LEAST often in Blackboard?
Least often
accessed features
Census Survey
Data
34. 19. Which features do you access LEAST often in Blackboard?
Least often
accessed features
Census Survey
Data
35. 20. Blackboard meets my needs in my current
courses.
Importance of
features
Census Survey
Data
# Answer Response %
1 Strongly Disagree 6 3%
2 Disagree 13 7%
3 Agree 108 58%
4 Strongly Agree 59 32%
Total 186 100%
36. Open comments section
Comments
Census Survey
• Received 35 comments
Data
• 16 comments directly related to
instructor knowledge and
implementation of Blackboard
• 8 comments related to amount of tools
available vs. the few tools used.
• Other less frequent comments
• Smartphone support
• Interface or navigation confusion
• Dislike changes from previous
versions
37. Example comments
Comments
Census Survey “I think that blackboard does not need to be overloaded with
Data
features. From my experience, most of the features in
blackboard are not used because they are not very important
for a classroom setting. Blackboard is useful because it allows
an instructor to communicate with his or her students and
students to communicate with their teacher. There is no need
to improve blackboard or add more features to it because it
already does what it does well. In fact, I find that having too
many features makes programs stressful and difficult to use
because they demand too much of users. I don't want a
comprehensive life planner and entertainment center with
blackboard. It already does what I ask of it: It provides a way
for my teachers to communicate with me about upcoming
assignments and grades. It does this very well.”
38. Example comments
Comments
Census Survey “In my experience, use of blackboard as a learning
Data management tool largely depends upon the instructor. Only a
few of my classes have used discussion board and only one
has used Elluminate Live! For most classes, blackboard serves
as a way to distribute class documents, announcements, and
contact information. Therefore, most of my use of blackboard
has been with simple tools, which is really all I need
blackboard for.”
“Some features are redundant and confusing, such
as Journals and Online Journal. There is a lack of
uniformity between how different professors use
different tools. It's very clunky and clumsy. I wish it
had a more seamless user interface, like facebook.”
39. Conclusions & Conclusions
Recommendations
• Students are overall pleased with Blackboard as a
tool.
• Many of Blackboards features are unused in courses.
• Students feel instructors us of Blackboard is primary
to its success in courses.
Recommendations
Option 1
Keep Blackboard as JMU’s learning management system
and focus attention on course implementation and faculty
training.
Option 2
Investigate alternative tools that better implement the
functions most used by instructors in their courses.
40. Reflections of
evaluation team
Positives
-
• Recording interviews using Elluminate
• Data collection using Qualtrics
Negatives
• Mass student email process
• Timing of survey
• Evaluators distance from JMU campus