Date: March 17, 2014
Time: 1:00–4:00 p.m. ET (UTC-4) convert to your time zone; Runs three hours.
Malcolm Brown and Veronica Diaz will moderate this online seminar with Tanya Joosten, Dylan Barth, and Nicole Weber from the University of Wisconsin–Milwaukee.
As the demand for blended and online learning opportunities increases, so does the need to ensure the quality of online education through faculty development programming. And with the increase in the diffusion of blended and online programming across higher education institutions, stakeholders are looking for ways to ensure the quality of the student experience and better understand the impact on student outcomes. Recently, many of us have been asked to provide evidence of the effectiveness of our faculty development programming: administrators are looking for a return on investment in faculty development to ensure quality in blended and online programming, as we are seeing decreases in state funding and enrollments, which leads to cut budgets. In order to for us to determine the effectiveness of our programming using a backwards design approach, we need to first understand what is a good online or blended course as well as what competencies are required of faculty to teach blended and online courses and how those can be best facilitated in a faculty development program. Then we can consider how to evaluate the impact on student outcomes.
This workshop will offer a collaborative and interactive opportunity to connect with colleagues to consider and construct how the effectiveness of faculty development programming can be determined and disseminated. A model of evaluation for a faculty development program will be shared.
Learning Objectives
By actively participating in this seminar, attendees will be able to:
Identify the characteristics of a good blended and online course, including the pedagogical model
Determine what elements and formats should be considered in designing a faculty development program
Share strategies for evaluating the effectiveness of your faculty development program at the course, program, and institutional levels from multiple perspectives, including students, colleagues, researchers, and administration
Understand how these steps fit into a model of evaluation for learning technologies and pedagogical innovation
Privatization and Disinvestment - Meaning, Objectives, Advantages and Disadva...
Determining the Effectiveness of Your Faculty Development Program
1. Determining the Effectiveness of
Your Faculty Development Program
March 17th, 2014
Tanya Joosten, Dylan Barth, and Nicole Weber
Learning Technology Center, LTC@uwm.edu
University of Wisconsin - Milwaukee
2. Overview
Introduction to UWM
Breakout activity, questions to consider in planning
to determine faculty development effectiveness
Sharing of UWM’s model, faculty development,
evaluation strategies, and evaluation tools as well
as other institutions
Conclusions
3. University of Wisconsin –Milwaukee
Learning Technology Center (LTC)
Tanya Joosten, tjoosten@uwm.edu, @tjoosten
Dylan Barth, djbarth@uwm.edu , @dylanbarth
Nicole Weber, nicolea5@uwm.edu, @nwebs
LTC.uwm.edu | LTC@uwm.edu | @UWMLTC
Introductions
4. Outcomes
Identify the characteristics of a good blended
and online course, including the pedagogical
model
Determine what elements and formats should
be considered in designing a faculty
development program
5. Outcomes
Share strategies for evaluating the effectiveness
of your faculty development program at the
course, program, and institutional levels from
multiple perspectives, including students,
colleagues, researchers, and administration
Understand how these steps fit into a model of
evaluation for learning technologies and
pedagogical innovation
10. Evaluation planning
Input
• Medium (F2F
or Online)
• Instructor
demographics
• Discipline
• Course level
Process
• Defining good
blended and
online course
• Defining
pedagogical
model
• Designing
faculty
development
program
Output
• Course level
• Program and
Institutional
level
11. F2F is not the gold
standard
Focus on the
process…the
pedagogy
Inputs
12. Define a good
blended and
online course
Define
pedagogical
model
Design a faculty
development
program
Process
14. Discuss: Steps to evaluating effectiveness
1.) Break into discussion groups using Adobe Connect.
You will be able to use audio or text chat.
2.) Access the Google Docs to document your activities at:
ELIOnlineSeminar.wikispaces.com
3.) Discuss, share, and brainstorm responses to the
following series of questions on the 3 steps to planning
evaluation.
Don’t forget to identify the member of your group and
document your group’s response in the Google Doc.
18. 2. What elements and format should be
considered in designing and developing a faculty
development program?
What opportunities and experiences should be
available to help instructors learn effective
practices in design, delivery, and teaching online
and blended courses?
20. 3. How will you know when faculty are
providing quality online and blended
courses? What tools or services could be
provided for evaluating the effectiveness of
online and blended courses? How will
quality be communicated to others?
21. DISCUSS!
1.) Break into discussion groups using Adobe Connect.
You will be able to use audio or text chat.
2.) Access the Google Docs to document your activities at:
ELIOnlineSeminar.wikispaces.com
3.) Discuss, share, and brainstorm responses to the
following series of questions on the 3 steps to planning
evaluation.
Don’t forget to identify the member of your group and
document your group’s response in the Google Doc.
24. 1. What is a “good” blended or online
course? What pedagogical model
facilitates a “good” blended or online
course?
25. Good course
Higher retention
Better grades on assessment
• Quizzes and exams
• Projects and papers
• Overall
Satisfied students
• Future enrollments
Learning
Engaging
Enrollments
26. Pedagogical model
Content
• Text
• Images
• Audio
• Video
Interactivity
• Discussions
• Groups
• Feedback
Assessment
• Written and oral examination
• Discursive
• Portfolio
31. 2. What elements and format should be
considered in designing and developing a faculty
development program?
What opportunities and experiences should be
available to help instructors learn effective
practices in design, delivery, and teaching online
and blended courses?
41. 3. How will you know when faculty are
providing quality online and blended
courses? What tools or services could be
provided for evaluating the effectiveness of
online and blended courses? How will
quality be communicated to others?
47. Who is your audience?
What variables will be
examined?
How will the data be collected?
analyzed?
How (and where) will the data
be presented?
Who will be involved in the
evaluation process?
When will the evaluation be
completed?
Evaluation plan
49. Input Process Output
Students Interactivity Students/Faculty
Demographics Communication Learning
Age Engagement Satisfaction
Gender Social Presence Performance
Ethnicity Content
Student Status Assessment Course/Institutional
Data
Full time/Part time Retention
Employment Status Drop/Withdrawal Rate
Zip Code Grade
Course/Instructor
Evaluation
Course
Discipline
Course Level
Instructor
Mode of Delivery
Variables
56. Check out some resources:
ELIOnlineSeminar.wikispaces.com
Editor's Notes
As the demand for blended and online learning opportunities increases, so does the need to ensure the quality of online education through faculty development programming. And with the increase in the diffusion of blended and online programming across higher education institutions, stakeholders are looking for ways to ensure the quality of the student experience and better understand the impact on student outcomes. Recently, many of us have been asked to provide evidence of the effectiveness of our faculty development programming: administrators are looking for a return on investment in faculty development to ensure quality in blended and online programming, as we are seeing decreases in state funding and enrollments, which leads to cut budgets. In order to for us to determine the effectiveness of our programming using a backwards design approach, we need to first understand what is a good online or blended course as well as what competencies are required of faculty to teach blended and online courses and how those can be best facilitated in a faculty development program. Then we can consider how to evaluate the impact on student outcomes.This workshop will offer a collaborative and interactive opportunity to connect with colleagues to consider and construct how the effectiveness of faculty development programming can be determined and disseminated. A model of evaluation for a faculty development program will be shared.
ScheduleAgenda12:00-12:30 Welcome to session Intro presentation of UWMLTC, stats of blended and online at UWM, history of blended at UWM (Tanya)12:30-1:15 Break out (Answer 3 questions in groups)1:15-1:45 Question 1 Report out, LTC summary of UWM blended and online course pedagogical model (last 5 minutes) (Nicole)1:45-2:15 Question 2 Report out, LTC summary of UWM faculty development program (last 5 minutes) (Dylan)2:15-3:00 Question 3 Report out, LTC summary of UWM evaluation efforts (last 5 minutes) (Tanya)
The LTC provides faculty development and pedagogical consultation, technology training and support, and evaluation and research of an array of course delivery modes, including tech enhanced, blended, and online.
In addition to facts, data, and numbersPossibly that quality online and blended courses are the depts, schools, and colleges responsibilityWe are not the police
Blended course definition: An Institutional DefinitionTanya can discuss UWM - From an institutional perspective, there are several ways to define blended courses; the example given is that of University of Wisconsin – Milwaukee, though it is not dissimilar to that of other campuses. The definition relies on an assessment of the impact of blended pedagogical strategies. Up to a certain point, about 20% of seat time placed online, the effect of blended learning is roughly equivalent to that of Web enhancement. This isn’t necessarily a bad thing: we often recommend to faculty that they start in the 0-20% range until they feel more comfortable assigning and assessing online work Between 20-50%, a course requires significant redesign work if it is to be taught successfully. The reduction of a third of the face time of a course, for instance, means that the instructor must find new learning activities and new forms of assessment, as well as different ways to deliver course content. Beyond 50%, the online work begins to dominate the pedagogical design of the course, and the instructor must explicitly work with considerable care to resolve the three key issues of blended learning: “closing the loop,”“course-and-a-half,” and “hearing voices.” Finally, when a course reaches 80% or more online, a question becomes increasingly acute: why is this course not being delivered fully online? What is the rationale for having face-to-face meetings at all? Often, there is such a rationale, but it becomes urgent to identify that rationale at this stage of course redesign.
Your faculty development program is
How do we know it is a good online or blended course?What does it look like?How do we help faculty design and develop these types of courses?
Stop comparing F2F to Online (or Blended)
Process is the magic!
Your faculty development program is
What is effective? What is success? What are you trying to accomplish by having a faculty development program or instructional improvement effort? What is your desired result?What makes for an effective online and blended learning model?
Your faculty development program is
Your faculty development program is
Summarize what groups didWhat is effective? What is success? What are you trying to accomplish by having a faculty development program or instructional improvement effort? What is your desired result?What makes for an effective online and blended learning model?
Active learning, student-centerEngaged learningResearch driven effective practicesNICOLEFor each delivery mode, there are pedagogical considerations to be made with regard to content delivery, interactivity, and assessment. The UWMLTC faculty development program and pedagogical consultations with our team guide instructors in making decisions about these considerations. Content Yes - text+images, current, video clips (under 10 minutes) Avoid video lectures, don’t spend to much time on audio lectures eitherInteractivityCreate opportunities for participation, community, collaboration, and connectedness through online class discussions, group discusionss, team projects, team synchronous meetings Assessment Frequent low-stakes feedback, opportunity to fail, make corrections to learning Avoid catastrophic assessment Provide audio feedback, if appropriate Provide varied forms of assessments from cognitive, behaviors, affective domains
Text plus images (Mean study)Rich, current (Web clips, videos, see Ginkgotree study)
Social presence, group activities, thought-provoking discussion boards, interactive Introductions and IcebreakersSupport and Sharing ScholarshipDiscussionsGroup ProjectsFeedback
Frequent and low stakesAvoid Catastrophic Assessment Valid assessmentshttp://assessment.uconn.edu/primer/taxonomies1.html
Summarize what groups did
Summarize what groups did
Who runs the faculty development program? Experienced instructors in online and blended with terminal degrees in our respective fields, represented by the social sciences, natural sciences, humanities, professions, and arts. We hope to continue to bring in more instructors who have been successful. We bring both the practice and the research elements to the program.
Blended approachThe structure focuses around small F2F sessions with online sessions in-between. The work is centered on manageable chunks with concrete deliverables.Taught in a blended format and in multiple formats during the academic yearFace-to-face meetings and online assignmentsModel good blended practicesExperience blended course as a studentEffective teaching modelExperienced blended teachers are program facilitatorsThe faculty development program that we offer is taught itself in a blended mode, since we take seriously the adage that the best way to learn how to teach a blended course is to take one. We offer the faculty development program multiple times during the year, once during each UWM semester (Fall and Spring) and once during the summer.The program comprises both face-to-face and online learning activities, in which our goal is to model good blended teaching practices, to encourage participants to experience blended learning as their students will, and to demonstrate that blended learning is an effective teaching model. To this end, we ensure that all of the presenters in our faculty development program are themselves experienced blended instructors, since that ensures that the presenters speak not only from theory, but also from lengthy experience as practitioners of blended learning.
We use backward design theories of Wiggins and McTighe to focus the redesign process of participants. This also illustrates the importance of studying research in online and blended learning to better inform our practices in the faculty development program. We also use backward design as the model for reconfiguring the program during the lifecycle of evaluation.
We facilitate our program in an active learning classroom so that we can both model effective classroom practices for blended instructors but also to promote active learning in our own program.
Practice-based: Develop Useable MaterialsParticipantsfocus on developing useable course materials so that they have developed chunks of their course when they leave. The deliverables all reinforce this aspect.A practical approach Get startedRedesign courseDevelop course materialAcquire teaching skills The purpose of our faculty development program is to support and encourage faculty who wish to redesign their courses for blended learning. We have adopted a practical rather than a theoretical approach – although our notion of “practice” is supported by current theory – because we know that faculty are busy, and that they are more interested in what works than in abstractions. A practical approach to blended course redesign identifies how to get started, how to redesign a course for blended teaching and learning, how to develop blended course material, and how to acquire the teaching skills needed for a blended course. We assume that faculty are already experienced teachers from a traditional face-to-face perspective, and that their principal task is to learn how to redeploy their skills in a somewhat different context.
Dylan – we should share the evaluation forms and other tools, CATS, etc., that we use. Can you post to the wiki resource page? Thanks!We have 5 main program outcomes. We determine whether these outcomes were met by ask for feedback about our program in several ways. We provide frequent self-reflection on the part of the participants to determine what they still need help with. We also ask for more quantitative feedback through ongoing online surveys through the registration system. As facilitators, we reflect on the progress regularly to make improvements. We also ask for a more thorough, summative evaluation at the end of the program. We use this data to make continuous improvements to our program.Main Program OutcomesStart of a redesigned courseNew teaching skills and knowledgeFaculty know what to expectFaculty get their questions answeredIn blended, re-examine both face-to-face and online component First, faculty start with concrete development of both broad-perspective elements of their new course, such as a course redesign plan and a syllabus, and the development of more specific features of the course such as learning modules. This is especially appropriate in disciplines or areas where “learning modules” are not typically the unit of course redesign, thus requiring faculty to re-examine their conceptual basis for instruction from the bottom up. Second, faculty must acquire new teaching skills and knowledge appropriate to a blended learning frame, such as how to facilitate a peer learning community both face-to-face and online, and how to assess student learning in a manner that is quite distinct from the usual “three exams and a term paper” of a traditional face-to-face course. In the latter context, it is important to note that faculty are often committed to the use of summative assessments such as exams and term papers without ever having questioned previously their efficacy. The prospect of a blended course redesign thus holds out the possibility of a paradigm shift from modes of instruction that once seemed satisfactory to a model of teaching and learning that problematizes everything that has once been taken for granted. This is a source of considerable skepticism and anxiety for faculty who have been teaching for many years in a more traditional manner, often unreflectively.
Evaluating the impact of the program on course, program, and institutional level
Course levelCourse evaluations by instructor ChecklistCourse evaluations by dept or institutionPeer evaluation guideQuality mattersUC Chico RubricStandard student evaluation of course and instructor How does it change when blended or online, if at all? Does it capture an evaluation of the pedagogical model? new skills of teaching? Understand course materials Engaging course activities Collaborative course activities Connected to instructor and classmates Receive frequent low-stakes feedback Opportunities to make corrections in learningStudent survey (engagement, learning, satisfaction, performance)
Nicole—using student evaluation data to ensure course quality. Students have firsthand knowledge of the course that was provided and often have taken other blended courses. Two ways of gathering student perceptions regarding the course is through mid-semester feedback (as pictured) and at the end of the semester. Tanya - I can talk about using CATs as a way to guage course improvements needed
Course levelCourse evaluations by instructor ChecklistCourse evaluations by dept or institutionPeer evaluation guideQuality mattersUC Chico RubricStandard student evaluation of course and instructor How does it change when blended or online, if at all? Does it capture an evaluation of the pedagogical model? new skills of teaching? Understand course materials Engaging course activities Collaborative course activities Connected to instructor and classmates Receive frequent low-stakes feedback Opportunities to make corrections in learningStudent survey (engagement, learning, satisfaction, performance)The certificate programs allows us to transform the culture of our campus to promote online and blended learning, to emphasize the importance of quality teaching and sound pedagogy, and to create a community of online teachers. Components: successful completion of OBTP, delivery of a blended course at UWM, peer course evaluation by LTC staff and/or peer mentor/evaluator, 2-3 page reflection on how the instructor’s pedagogy has changed, and “paying it forward” (e.g., guidance on the listserv, syllabus added to resource repository). Over 70 faculty and instructors have been awarded the certificate represented in the Colleges of Letters and Science, Peck School of the Arts, School of Education, College of Nursing, School of Information Studies, Lubar School of Business, and College of Health SciencesCan discuss how the certificate program came to be and philosophical debates around its development
Incentivizing facultyTime and focusIRBData collection (data warehouses, grade and retention, survey tools)Data analysis (lack of expertise or staff)