A powerpoint summarizing a scholarly paper presented at the Association for the Study of Higher Education (ASHE) on nonresponse to online course evaluations. Uploaded 16 Nov 2010. Indianapolis, IN.
1. WHO DOESN’T RESPOND AND WHY? AN
ANALYSIS OF NONRESPONSE TO ONLINE
STUDENT EVALUATIONS OF TEACHING
Meredith J.D. Adams & Paul D. Umbach
2. CONTEXT
• Participation decline while survey requests are
increasing. (Nonresponse increases the risk for
error.)
• Ongoing change from paper to web-based
course evaluations.
• Results from course evaluations are used for
research and decision-making processes about
personnel.
3. FRAMEWORK
• Survey Theories
– Salience (Dillman et al., 2002; Groves et al., 2004,
2009)
– Survey Fatigue (Groves et al., 2004; Porter &
Whitcomb, 2005)
• Academic Disciplines framework (Holland,
1966; Rosen et al., 1997; Smart et al., 2000)
• Previous survey and course evaluation research
at postsecondary institutions
4. PURPOSE OF STUDY AND RQs
• Purpose of study is to investigate factors of
participation in course evaluations
• Research questions
– To what extent did course salience increase the
likelihood of response? To what extent did between-
level interactions influence these results?
– To what extent did survey fatigue influence
response?
5. METHODS OF ANALYSIS
• Study measured likelihood of response
• Hierarchical modeling techniques
• 30 variables
– Students (demographics, housing, number of surveys
students completed)
– Course (if department of course and student’s major
matched; grade earned)
6. RESULTS
Statistically significant
factors associated with
nonrepsonse:
– Gender (Male)
– Ethnicity (African-American,
Asian)
– Housing (off-
campus/commuters)
– Athletes
– Grades (Ds, Fs,
ungraded
students/courses)
– Age (traditional age)
– Nontransfers
– Class rank
(sophomores & juniors)
– Students with more
than 10 SETs to
complete
8. RESULTS (continued)
• Using Holland’s six major academic types
– Realistic majors were more likely to respond than
social, artistic, conventional, enterprising, and
investigative disciplines.
– Social majors were less likely to respond than all
other students if the course was in the same
department as the student’s major.
• Most variables were no longer statistically
significant when the course was in the same
department as their major.
10. DISCUSSION
• Mostly aligned with previous research and
theories of survey participation*
– Exceptions = Environment of major/course, class
rank
• Introduced new potential influences on
participation
– Campus housing, athletes, transfer status, etc.
* Avery et al., 2006; Cohen, 1981; Clarksberg, et al., 2008; Dey, 1997; Dillman et al., 2002, 2009; Fidelman, 2007; Groves et
al., 2004, 2009; Johnson et al., 2002; Jones, 2009; Kaplowitz et al., 2004; Lepkowski & Couper, 2002; Marsh, 2007; Moore &
Tarnai, 2002; Porter et al., 2004; Porter & Umbach, 2006a; Porter & Whitcomb, 2005; Sax et al., 2003, 2008
12. LIMITATIONS
• Only one institution
• Only undergraduates
• Limited information about the course, the
faculty, class size, etc.
13. IMPLICATIONS
• Evaluate the major’s environment/culture
– How to increase salience?
• Encourage students unlikely to respond
• Examine how we utilize course evaluations
14. FOR MORE INFORMATION
meredith_adams@ncsu.edu
This paper can be found at:
1. https://cedupload.ncsu.edu/dropbox/uploads/Adams%20Um
bach%20Course%20Evaluation%20Nonresponse%20ASHE
%202010.pdf
2. http://bit.ly/9zjMAY
3. http://goo.gl/ByiZo
4. This PowerPoint Presentation is available at
http://www.slideshare.net/meredithNCSU/ashe-2010-course-
eval-nonresponse
Notas do Editor
Realistic Majors =
To what extent did course salience increase the likelihood of response? To what extent did between-level interactions influence these results? To what extent did survey fatigue influence response?
INSTRUMENT: Lab questions then the lab is surveyed too. ADMIN: Cut down on opportunity costs (off-campus students, athletes) ENVIRONMENT: Smart et al., 2000 (academic disciplines book)