1. LMS Focus Group Report
Background
UT Dallas’ current eLearning system, Blackboard Vista 8.2, is an end-of-life system and
will no longer be supported by Blackboard after 2013. With this end in mind, the
university is conducting a search for a new learning management system. UTD will
publish a Request for Proposals in early Spring 2011.
In October and November 2010, as directed by the LMS Committee, the Office of
Educational Enhancement, assisted by SOM eLearning, conducted a series of focus
groups to solicit feedback from faculty, staff and students about the use of the current
eLearning system and about their needs to determine what features would be
necessary and desired in any learning management system.
Focus Groups
A total of eight (8) focus groups were held - 3 for faculty users, 3 for student users, 1 on
accessibility and 1 for back-end users. See the below tables for attendance and
demographic information of attendees.
At the beginning of each focus group (except for the back-end users group), a brief
survey was administered individually to solicit quick feedback about specific tools used.
A comments area was provided.
See Appendix A for the survey and survey results.
Guiding Questions
For the focus group sessions, a set of “guiding questions” was used to direct the
conversations. This semi-structured approach allowed the facilitator to guide the
discussion and to prevent the focus groups from becoming “gripe” sessions. See
Appendix B for the Guiding Questions.
Observations
Interestingly, many of the features that faculty expressed a desire for in a new system
already exist in the current eLearning. Faculty were either unaware of them or had not
received training. Faculty emphasized that migration from eLearning to new system
should be easy.
Students generally wished that all faculty used or were required to use eLearning. “It
would be convenient.” Not all of the students were taking full online course; thus their
faculty tended to use eLearning for posting syllabi and lecture notes and homework
assignments. Some students took online quizzes. Student disappointment with
eLearning was more from lack of use by faculty and from stability. Students related that
other students don’t like eLearning because it often didn’t work and they need to feel
confident that the system will work.

2.
Focus Group Results
Current usage (from surveys)
See Appendix A
Usability
• “eLearning is powerful but for a power user” (F)
• “the learning curve is more work than it’s worth” (F)
• navigation is easy (S)
• could look more modern, more updated (S)
Tools
Communication
Mail • Tool should be a default in all courses (F)
• Need search by name and keyword (F)
• If mail forwarded out, need to be able to reply back into (F)
• Window doesn’t expand (F)
• Connect UTD email and eLearning mail (F) (S)
Discussions • Not intuitive to create a new post (F)
• In graded discussions, move from student to student (not back to
list) (F)
• Need to reply privately to a post (F)
• grading is clunky (F)
• doesn’t that it opens new into separate, smaller windows (F)
• needs to be easy to use (like Facebook messaging) (S)
• in “show all” should still be threaded (S)
• “read” should apply only to ones actually read (not all new) (S)
• searchable and tagable threads (S)
Calendar • link to Outlook calendar / Google calendar (F)
• color code entries by course (user assigned colors) (F)
Announcements • need ability to edit, delete announcements by TAs (F)
• need ability to hide announcements (F)
Chat • would be essential to online course (S)
• chat not sufficient (moved to Skype) (S)

3. Collaboration
Groups • useful (F)
• manual creation of groups is somewhat hard (F)
• nice but not crucial (S)
• group contact could be expanded (S)
Assessment
Assignments • Easy formatting w/o launching html editor (F)
• Sequential grading of assignments - student to student (no return
to list) (F)
• Easy upload of multiple or large files (eg. video) (F)
• Somewhat difficult to submit (S)
• Some files (MS Access) hard to download (S)
Quizzes • Improve graphic ability (embedding, formats) (F)
• Change settings while quiz is open (F)
• Equation editor isn’t good; integrate “MathType” (F)
• System not “dynamic”, though works well for MC tests (S)
Grading forms • Ability to add comments with each item (F)
Learning Modules
• More visual way (F)
Course Management
• Desired - central location from which to push email, announcements,
exams to all sections (F)
File manager • Should work more like a drop box (F)
• Faster download/uploading of files (F)
Syllabus • Integrate with Coursebook (F)
• Hypertext to calendar (F)
Icons • More choices (F)
Edit pages • Make default “See all” (F)
MyFiles • Nice, files always there (S)
Gradebook
Columns • Rearranging is difficult (F)
• Need per column settings (F)
Formulas • Formula editing is difficult (F)
General • Add categories (F)
• Have default grading schemes (F)
• Make it more like Excel (F)

4.
Integrated Tools
• Library, library reserves (F) (S)
• Video server, YouTube, iTunesU, etc (F)
• Google Docs (F)
• Turnitin - use Grademark (online grading) (F)
• MuchLearning (calculus) (S)
• Skype (S)
Additional tools or functionality
• Online editing of documents, regardless of format (F)
• Better text editor (F)
• Notification system (send reminders after missed exam, send new announcement,
new discussion, etc, RSS feeds) (F)
• Push notification for updates (S)
• Ability to embed videos easily (F)
• Wikis (F)
• LaTex (S)
• Smart Tutor (S)
• Web conferencing (S)
• Resume function on quizzes (S)
• Consistent look/structure between courses (S)
• Picture roster (S)
Accessiblity
• Faculty and students confused this with navigation and stability
Overall Improvements
Login • too many clicks (F) (S)
• should be more like shared drives (accessible when log into
computer on campus) (F)
Mobile • need mobile app (F)
• push notifications, RSS feeds (F) (S)
Public • need ability to make parts public (resources, etc) (F) (S)
Migration • easy porting to new system (F)
Listings • make default - latest entry on top (for anything) (F)
MyCourses • elearning could be more integrated with other UTD systems
• ability to open multiple windows
• media library that allows students to post/upload (S)
• site map (S)

5. Results from Back-end Users Focus Group
Delivered integration w/ PS • functionality delivered - 50% (MH)
• API hooks (SH)
• Tight integration (NTH)
Structure / hierarchy • Detail level - product / service - how much control
do we have (MH)
• Conditional course existence (MH)
• Roles - admin, support - different levels (MH)
Authentication • LDAP (MH) - Shibboleth (SH)
• Point of entry control - single sign on (NTH)
Hardware • No specific hardware as long as below met
• Enterprise level (MH)
Database • No flat architecture - Oracle/ MS SQL (MH)
• Must be able to query (MH)
• Published schemes (SH)
External integration control • We know where our data resides (MH)
• We know who has access to our data (MH)
Archiving • Comprehensive backup strategy (MH)
• Capability for restoring deleted content (MH)
• Daily backup (SH)
Application • Web-based (MH)
• Support multiple browsers/OS (MH)
• High availability (MH)
Migration • Migration support from Vista (SH)
Security • SSL - for mobile (grades) (MH)
Hosting • Meet UTD/state requirements (MH)
Integration Control of content • Course object (MH)
• Individual object (MH)
• Gradebook (MH)
• Member enrollment (MH)
• Term-based (MH)
Support • What kind (MH)
• Multiple levels (MH)
• Process (MH)
• SLAs (MH)
Hosting • Data storage meets FERPA, UTD, state policies
(MH)
Concern-timeline for vendors to get through the vetting process in Information Security.

6. Appendix A
Student Focus Group Survey
What is your status?
Freshman Sophomore Junior Senior Graduate Student
What is your major / school?
How long have you been using eLearning?
How comfortable are you using eLearning?
Very comfortable Comfortable Neutral Uncomfortable Very
uncomfortable
Which course tools in eLearning do your instructors use?
• Announcements • Goals
• Assessments • Gradebook
o Quizzes • Grading forms (rubrics)
o Surveys • Groups
o Self-tests • Learning Modules
• Assignments • Mail
o Individual assignments • Posting syllabus
o Group assignments • Web links
• Calendar • Who’s Online
• Chat / Whiteboard
• Discussions
o Graded discussions
o Journals
o Blogs
o Threaded
Which student tools do you use?
• My Files • My Grades
• My Progress • Notes
Please use the back of this survey for any other comments you’d like to add.

7. Student Focus Group Survey
What is your status?
[2]Freshman [1] Sophomore [2] Junior [1] Senior [8] Graduate Student
What department are you in?
• [2] Arts and Humanities
• [1] Natural Sciences and Mathematics
• [4] School of Management
• [1] Economic Public Policy Sciences
• [0] Brain and Behavioral Sciences
• [4] Engineering and Computer Sciences
• [1] Undeclared
How long have you been using eLearning?
• [9] 1 year
• [1] 3 years
• [1] 5 years
How comfortable are you with eLearning (Please mark your response)
[3] Very Comfortable
[6] Comfortable
[5]Neutral
[] Uncomfortable
[] Very uncomfortable
What tools are you currently using in eLearning (Mark all that apply)
[10] Announcements
[2] Assessments [1] Goals
[1]Quizzes [9] Grade Book
[] Surveys [3] Grading Forms (Rubrics)
[1] Self‐Tests [5] Groups
[4]Assignments [6] Learning Modules
[7]Individual Assignments [11] Mail
[7]Group Assignments
[1]Calendar [12] Syllabus
[4]Chat/Whiteboard
[4]Discussions
[3]Graded discussions [3] Web Links
[2]Journals [3] Who’s Online
[4]Blogs
[8]Threaded
Which student tools do you use?
[6] My Files [11] My Grades
[] My Progress [2] Notes

8.
Please use the back of this survey for any other comments you’d like to add.
• I have never had a semester where every professor used eLearning. As of now, I
have 2 using eLearning, 2 using their own personal websites, and out one using the
library’s eReserves. For this type of software to be effective, use needs to be
mandatory by professors. As of right now, it is simply an inconvenience to use the
software. Currently, the math and science departments have started using
computerized testing. This software, much learning, while primarily a
testing/quizzing software, should have the ability to post
notes/announcements/grades it would be much better for students if there was one
type of software used throughout the university. I imagine it would also save quite a
bit of money on licensing fees (the math department has already purchased 6000
copies of this software. It’s already very difficult to get a professor to learn one new
piece of software – having one central software would greatly help its adoption.
• eLearning is fairly simple to use and I like it but in all honesty eLearning doesn’t
look very cool.
• Library integration. I am also working at UT Southwestern Medical center
(Research) Academic Information Systems. We use UPIC (PeopleSoft now Oracle) to
write eLearning and we have ELM (Enterprise Learning Management). I write
technical training. We are looking for solutions as well. Suggestions: Saiki, Moodle
Rooms
• I use eLearning as an instructor (TA). Very little use of eLearning as a student
• I am currently doing an independent study, so I am not in a “class” that uses
eLearning. However, I am a TA for a professor who uses it only a little (to post
assignments and syllabus). I am assigned to teach next semester though, and I will
be using it a lot – the class will be a “hybrid” (half on‐line, half in person) class.

9.
Faculty Focus Group Survey
What is your role in the university?
[28] Faculty [2] Staff
What department are you in?
[3] Arts and Humanities [3] Brain and Behavioral Sciences
[3] Natural Sciences and Mathematics [3] Engineering and Computer Sciences
[10] School of Management [1] Interdisciplinary Studies
[5] Economic Public Policy Sciences
How long have you been using eLearning?
[2] Never [5] 5 years
[7] 1 year [1] 8 years
[8] 2 years [1] 9 years
[2] 4 years
How comfortable are you with eLearning (Please mark your response)
[12] Very Comfortable [1]Neutral [] Uncomfortable
[14] Comfortable [2]Very uncomfortable
What tools are you currently using in eLearning (Mark all that apply)
[22] Announcements [13] Calendar [13] File manager [22] Mail
[8] Assessments [7] [3] Goals [22] Syllabus
[13] Quizzes Chat/Whiteboard [26] Grade Book [1] SCORM
[5] Surveys [17] Discussions [4] Grading Forms [9] Templates
[8] Self‐Tests [9] Graded (Rubrics) [17] Web Links
[11] Assignments discussions [11] Groups [11] Who’s Online
[15] Individual [4] Journals [14] Learning
Assignments [5] Blogs Modules
[11] Group [21] Threaded [13] Selective
Assignments Release
Please add any other comments below.
• I’m interested in open source products like Sakai
• It is a great resource. My use depends on the course I am teaching and the level of the
course (under graduate or graduate)
• I have helped in past webCT / eLearning initiatives and appreciate this chance for us to
share our experiences. Please don’t hesitate to ask if I can help in anyway. I feel like the
current tool we are using for courses is difficult for students and instructors to use easily
and I am excited to explore other opportunities for us to improve our distance learning and
enhance learning in all UTD courses.
• 1. When grading assignments submitted in eLearning, it should be possible to go straight
from one paper to the next without having to go back to the main screen for that

10. assignment. 2. When Iooking at a list of submitted assignments and sort it by student
name, I find two problems a. The list is sorted by first name, not last. B. When I look at a
paper and then go back to the list, the sorting reverts back to default. Resorting takes
forever. I have 120 students per class submitting at least 10 papers each, in 3 courses. 3.
Students simply don’t know how to use eLearning features. They don’t know how to edit a
submitted assignment, for example. They don’t understand the difference between a public
discussion and email. A more accessible system might cause less confusion in students with
more limited computer skills. 4. Links to TurnItIn and CPR would be helpful. Integration, I
mean. 5. If a TA posts an announcement, I have no access to edit or delete it. I should be
able to change all announcements. Similarly, students can change the calendar for
everyone, and I can’t intervene. I had a student post some of her assignments there, and I
couldn’t delete them. 6. In assignments, HTML Enabler does not work for most students,
which causes widespread panic. 7. The eLearning help number is not helpful. On campus
support couldn’t be better, but I hear many bitter students complain about the remote help
desk.
• I have anywhere from 12 to 17 sections of one course. The most glaring omissions in
eLearning is the lack of centralized location where I can store files, quizzes, assessments,
etc. that pertain to all the sections. I would also like to send mail and post announcements
from one place where I can send it to all the sections simultaneously. Likewise, I would like
to post assessments in one place where they can be accessed by all sections. With all of the
above, I still would like to retain the individual sections because when it comes to posting
grades in Orion, I can just draw them from the individual sections in eLearning. Bunching
all the sections into a mega section would not work for me because for grades I still need to
retain individual sections. Also, different sections are taught by different TAs, who need to
have access only to their sections.
• Keep it simple! A greater number of instructors and students will use it. Basic Blackboard
worked very well at the University of Arizona – Eller College (Walter is the contact if you
would like to call him)
• I have not found the current eLearning system to be useful so I do not use it. I would hope
we could find a more effective program for the university.
• I would love to see a mobile application for the system – particularly discussions and
quizzes.
• Keep hoping for portfolio developments

11.
Appendix B
Focus Group Guiding Questions
Let’s talk about your Current usage
• What is your level of expertise with eLearning?
• What do you use eLearning for?
• What do you like about eLearning?
Let’s talk about Usability
• What do you think about eLearning’s interface? Explain.
• What improvements do you need?
Let’s talk about Tools in a Learning Management System
• What do you like about the tools you are currently using in eLearning?
• What do you not like about these tools?
• Which tools are essential? (need to be part of whatever system we use?)
Let’s talk about Communication
• How do you communicate electronically with your students?
• How do you use eLearning to communicate with your students?
• What is missing from eLearning that would enable you to more effectively communicate
with your students?
Let’s talk about Collaboration
• How do your students collaborate electronically?
• How do you use eLearning for collaboration?
• What is missing from eLearning that would enable you and your students to more
effectively collaborate?
Let’s talk about Assessment
• How do you assess your students electronically?
• How do you use eLearning to assess your students?
• Are you able to assess your students effectively using eLearning?
• How could you better assess your students in an online format?
Let’s talk about Learning Modules
• How do you organize your course (in eLearning)?
• What content tools in eLearning do you use?
• How would you rate each of these content tools and why?
Let’s talk about Course Management
• What course management tools in eLearning do you use?
• How would you rate each of these course management tools and why?
• What course management tools would you like to see made available?

12. Let’s talk about Gradebook
• What do you need in an LMS gradebook?
• What do you like about the eLearning gradebook?
• What do you not like about the eLearning gradebook?
• What features would you like to see added (or changes made to) the gradebook?
Let’s talk about Integrated Tools
• What integrated tools do you use (i.e. Respondus, Turnitin, eInstruction-clickers)?
• What external tools would you like to see integrated with an LMS?
What additional tools or what functionality would you like to see in an LMS? (eg
portfolios, wikis, etc)
Let’s talk about Accessiblity
• Do you feel eLearning is accessible?
• Why/why not?
Let’s talk about Overall Improvements
• How can eLearning (an LMS) be better?