"Rubric for Improving the Quality of Online Courses" presentation given at TLt 2009 conference April 27th in Regina, SK, CA.

1. RUBRIC FOR IMPROVING THE QUALITY OF
ONLINE COURSES
Denise Nelson, Virtual Campus

2. OUTCOMES
Share strategies used to measure the quality

of online courses.
 Discuss the impact of a quality assurance
system on institutional operations and
stakeholders.
2

3. PLAN
SIAST Context & background

 Rubrics...rubrics...rubrics...
Adaptation

 Use
Discuss quality assurance processes &

results
 Future plans?
3

4. VIRTUAL CAMPUS REVIEW
Development and Delivery on Online Courses
4

5. VIRTUAL CAMPUS REVIEW
External review of online courses

 Stakeholder focus groups
 Email interviews
 Web survey
 mySIAST – Intranet call for feedback
5

6. RUBRICS...
Used existing rubrics as guides to develop

online courses
CNIE Criteria for Award of Excellence in

Instructional Design
 BlackBoard Exemplary Course Rubric
 Blood-Siegfried et al. A Rubric for Improving the
Quality of Online Courses
 Blood-Siegfried et al. adapted to SIAST Context
6

7. ANALYSIS OF EXISTING RUBRICS RELEVANT FOR
ONLINE COURSE
California State University, Chico - What does a high quality online course look like?
1.
http://www.csuchico.edu/tlp/resources/rubric/rubric.pdf (Committee for Online
Instruction, 2003)
Towson University, Maryland - Towson Faculty Peer Review of Online Courses
2.
http://www.towson.edu/~mcmahon/peerreview/On-linerubric.pdf
(Ashcraft, McMahon, Lesh, & Tabrizi, 2003)
University of West Georgia - Assessing Interactive Qualities of Online Courses
3.
http://www.westga.edu/~distance/roblyer32.html (M D Roblyer & Ekhaml, 2000; M. D.
Roblyer & Wiencke, 2003)
Grant McEwan College - Criteria for Evaluating the Quality of Online Courses
4.
(Wright, 2004)
http://elearning.typepad.com/thelearnedman/ID/evaluatingcourses.pdf777
University of Wisconsin
5.
http://www.uwex.edu/disted/conference/Resource_library/handouts/Forum1.pdf
(Swan, Bishop, Wisher, & Trollip, 2003)
Robert Wood Johnson Partnerships for Training
6.
http://pftweb.org/reach/reach_home_frameset.htm (Partnership for training, 2003)
Measuring Success: Evaluation Strategies for Distance Education
7.
http://www.educause.edu/ir/library/pdf/eqm0213.pdf (Lockee, Moore, & Burton, 2002)
7

8. BLOOD-SIEGFRIED ET AL. RUBRIC CRITERIA
Organization and Design
1.
Course Content
2.
Instruction
3.
Interaction
4.
Evaluation and Assessment
5.
Student Support Services - added
6.
8

9. USE OF RUBRIC FOR EXTERNAL COURSE
REVIEW
Adapted rubric

Obtained course context information

nature of target audience

design specifications

student/instructor interactions

pace

development and delivery schedule

Faculty team available to respond to inquiries

Provided assessment items and related data not

housed with course (e.g., examinations)
9

10. DISTANCE COLLABORATION
10

11. PROCESS AND DELIVERABLES OF THE REVIEW
Reviewer received documentation & access

Context and rubric documentation

 Access to Master version of online course
 Access to archived online course (demonstrates
interaction, grades)
Reviewer conducted review within ~2 weeks

 Reviewer emailed completed review
documentation
 Reviewer and stakeholders held debriefing
mtg 11

12. EXTERNAL REVIEWER COMMENTS ABOUT USE
OF RUBRIC
Lori Atkinson
Instructional Designer
12

13. CHOICES...
13

14. RATE THE USEFULNESS OF THE RUBRIC FOR
COURSE EVALUATION?
Highly useful

The criterion creates a checklist that will

demonstrate whether the online course provides
the best shot at a quality student experience.
Overall the 7-13 objectives within each criteria

captured the major areas of best practice for
online learning.
I would recommend that the rubric is used for

the basis of a course evaluation or course
development tool. (Lori Atkinson, 2009)
14

15. RATE THE USEFULNESS OF THE RUBRIC TO
INFORM ONLINE COURSE DEVELOPMENT?
Highly useful for instructor mediated courses

 This rubric should be adapted for self-paced
and blended courses to reflect different
student to content (and media), student to
student, and student to instructor interaction
strategies. (Lori Atkinson, 2009)
15

16. WHAT ARE THE STRENGTHS OF THE RUBRIC?
Well grounded in research, reflected in

structure and content (Lori Atkinson, 2009)
16

17. WHAT ARE THE LIMITATIONS OF THE RUBRIC?
Requires an additional document to

synthesize major themes for improvement
(Lori Atkinson, 2009)
17

18. ARE THERE OTHER CRITERIA/RUBRICS THAT
SHOULD BE INCORPORATED?
A section that provides evaluation of the tie

between the outcomes, instruction and
assessment.
This would help to demonstrate areas of the course

that are content heavy and not really related to the
course outcomes.
Learning Learning Assessments
Outcome Activities
Write resume Practice writing Write resume
resume Critique resume
As a development tool this would be helpful to

assess what technology intervention is really a
match for the learning outcome. (Lori Atkinson, 2009)
18

19. LORI ATKINSON, EXTERNAL REVIEWER
19

20. RESULTS OF USING RUBRIC
Identify professional development

needs
Instructional design process

Alignment of outcomes, learning

activities, and assessment
Evidence for resource allocation

Identify important questions

(e.g., Should this course be
instructor led, self-paced, blended?)
Improve learning experience

Review debrief facilitates connection

with other SIAST teams and
examples of high quality SIAST
courses
20

21. DISCUSSION AND FUTURE PLANS?
Professional development to increase

awareness of benefits of using rubric to
guide development and evaluation
 Dialogue re improvements needed in course
development processes (e.g., dealing with
heavy course content; alignment)
 ...
21

22. CONTACT
Denise Nelson
Instructional Designer
SIAST Virtual Campus
Phone: 306-775-7683
Email: denise.nelson@siast.sk.ca
PPT available at
http://programs.siast.sk.ca/nelson/#presentations
22