This is the handout version of a workshop I gave to the Association for the Assessment

1. “Learning by Doing”
By Patrick B. Barlow
PhD Candidate in Evaluation, Statistics, & Measurement
The University ofTennessee
A Hands onApproach to Multiple ChoiceAssessment

2. A Bit of Background
• Currently a PhD Candidate in Evaluation, Statistics, & Measurement atThe
University ofTennessee
• Been working in higher education assessment since 2008
• Began working on aTeagle Foundation grant teaching an “Assessment 101” course
with Dr. Phil Kramer at Saint John’s University (MN).
• Recently (2011) shifted my focus to assessment in Graduate Medical
Education, specifically in evidence-based medicine, research methods, and
statistics.
• Developing a new biostatistics and clinical epidemiology assessment
instrument for my dissertation using an Item ResponseTheory (IRT)
approach.

3. Objectives ofToday’sWorkshop
After today’s workshop, participants will be able to:
• Translate vague course learning goals to measurable learning objectives
• Develop an effective test blueprint for their multiple-choice assessment
• Create utilization-focused assessments that focus on giving students real-world
application of concepts, and giving instructors useful feedback on student learning.
• Write effective item stems and response options that reflect best practices in item
writing.
• Critique their existing assessment instruments for areas of improvement.
I Hope!

4. Today’sWorkshop at a Glance
• Part I
• LearningObjectives
• Test Blueprints
• Activity I
• Writing learning objectives from vague course
goals
• Part II
• Defining Multiple Choice Questions (MCQs)
• Best practices in MCQ writing
• Challenges to writing effective MCQs
• Activity II
• “Assessment 911!”

5. Fundamentals
What areWeTalking AboutToday?
• What are MCQ Assessments?
• Why
• When
• But…
• Most professors are not trained as assessment experts
• Assessment as a stick vs. carrot
• Questions of Use

6. Back to Basics
Writing High Quality Learning Objectives

7. Integrated Perspective
• Integrated Course Design (Fink, 2003)
• Learning objectives inform assessment
strategies, which inform teaching &
learning activities.
• All components are integrated &
support one another.
Learning
Objectives
Assessment
Teaching &
Learning Activities
Situational Factors

8. Goals vs. Objectives
• For this workshop, we define learning “goals” and learning “objectives” as
(Suskie, 2009):
• Goal: how the students will be different as a result of the course.
• EX: “Students will gain a working knowledge of the fundamental concepts of epidemiologic
research methods and statistics as well as the ability to comfortably read research articles in
their specialty.”
• Objective: a specific, measurable outcome that describes the details of each goal.
• EX: “Students will be able to identify different scales of measurement for variables”

9. Writing a High Quality Objective
• Be concise
• Neither too vague nor too specific—find a
balance
• Focus on the results
• What should the students have after the course
not while in the course?
• Use action words
• Avoid vague or fuzzy terms such as “learn,”
“demonstrate,” or “know”.
• Write in a team
Adapted from Suskie, 2009

10. Moving fromVague Goal to Measurable Objective
Statistics 101 Course Example
• Vague learning goal… Students will
understand basic
statistics concepts
Descriptive statistics
HypothesisTesting
Scales of
measurement
Students will be able to formulate a null
and alternative hypothesis for a given
research scenario
• Concepts related to the
goal
• Measurable objective for
one of those concepts

11. Importance of Being Parsimonious
• Sometimes the list of specific concepts would become too cumbersome if
translated into a long list of learning objectives…
Students will be able to apply a cross-sectional
study design to their own research area
Students will be able to apply a cohort study
design to their own research area
Students will be able to apply a case-control
study design to their own research area
Students will be able to apply an RCT study
design to their own research area
Students will be able to
apply the four key
epidemiologic study
designs to their own
research area.

12. YourTurn…
• In groups of three to four…
• Choose an area of research and/or teaching to
focus on for the activity.
• Feel free to choose among your own interests, or
pick something as a group.
• Use what we have discussed to think of the
following:
• Two learning goals
• At least three specific, measurable learning
objectives that could be used to describe those
goals.

13. Test Blueprints
Using LearningObjectives to PlanAssessment Structure

14. What is aTest Blueprint?
• In general, a test blueprint refers to the allocation of items or sections of a
test according to certain concepts or ideas (Suskie, 2009)

15. Why Use aTest Blueprint?
• Makes tests easier to construct (think of it like
a test “recipe”)
• Assures that you address relevant information
• Assists with generating study materials for
students
• Makes “teaching to the test” more like
“teaching with the test”
• If the test only asks questions on what you see
as important, then you would hope your
teaching reflects these topics.

16. One Approach for Creating aTest Blueprint
• Ingredients
• 1 Course syllabus
• 1 Set of LearningObjectives
• 1 Set of lecture materials or
other classroom activities
• Calculator
Identify Key Learning Objectives
Create test “topics”
Allocate fractions of the test to
each topic
Revise the list of topics & repeat
The Basic Cycle

17. “after this
module
participants
should be
able to…”
1. Start with course learning
objectives
Biostatistics & Clinical Epidemiology Assessment
Example
Apply each
design to their
own area of
research
Identify major
epidemiologic
research designs

18. 1. Start with course learning
objectives
2. Identify test “topics” from
learning objectives
Biostatistics & Clinical Epidemiology Assessment
Example
Apply each
design to their
own area of
research
Identify major
epidemiologic
research designs
Research
Design (20%)

19. 1. Start with course learning
objectives
2. Identify test “topics” from
learning objectives
3. Expand each topic to as many
“concepts” as possible
Biostatistics & Clinical Epidemiology Assessment
Example
Identify major
epidemiologic
research designs
Identify an RCT
Identify a Case-
Control Study
Identify a
Cohort Study
Identify aCross-
Sectional StudyApply an RCT
Apply a Case-
Control Study
Apply a Cohort
Study
Apply a Cross-
Sectional Study
Apply each
design to their
own area of
research

20. 1. Start with course learning
objectives
2. Identify test “topics” from
learning objectives
3. Expand each topic to as many
“concepts” as possible
4. Collapse list of concepts to
remove redundancy
5. Create/modify items
Biostatistics & Clinical Epidemiology Assessment
Example
Identify major
epidemiologic
research designs
“after this
module
participants
should be
able to…”
RCT
Case-Control
Study
Cohort Study
Cross-Sectional
Study
Identify major
epidemiologic research
designs, and apply each
design to their own
research area

21. For Discussion
• What is your approach for designing an objective assessment?
• What has worked out well for you in the past?
• What has not worked out so well for you?
• How do we, as assessment experts, describe the utility of making test
blueprints to our colleagues or clients?

22. Section II: Writing
Effective MCQTest Items
Defining MultipleChoice Questions (MCQs)
Best practices in MCQ writing
Challenges to writing effective MCQs

23. Section II
Writing Effective MCQTest Items
• Defining multiple choice items
• Two main types of MCQs
• Characteristics of multiple choice items that emphasize higher-order thinking
• Writing effective item prompts/stems
• Writing answer choices and distractors
• Challenges to writing multiple choice items
• Item independence
• Avoiding “testwise” students

24. Defining MCQs
• We will be comparing two types of MCQs in
this workshop
• True / False MCQs
• Examinees select all true options such as
“OptionA, option B, both A & B, or Neither A
nor B”
• Simple true / false questions
• One-Best-Answer MCQs
• Examinees select a single best response usually
from 4 or more options
Adapted from Case & Swanson, 1998

25. Examples
True / False MCQ
• Which of the following is the correct formula
for hydrogen peroxide?
a) H2O
b) CO2
c) H4O2
d) 2(HO)
One-Best-Answer MCQ
A lab assistant finds an unmarked beaker containing
a clear, viscous liquid.The assistant further tests the
unknown liquid to find that it is non-flammable, and
slightly acidic (pH = 6.2), and has a boiling point of
150 oC.
What chemical is most likely in the unmarked
beaker?
a) H2O
b) CO2
c) H4O2
d) 2(HO)

26. What’s the Difference?
True / False MCQs can only be judged as absolutely true or absolutely
false.There are no shades of grey.
• Which of the following is the
correct formula for hydrogen
peroxide?
a) H2O
b) CO2
c) H4O2
d) 2(HO)
DC
B
A
False True

27. What’s the Difference?
DCB A
Least
Correct
Most
Correct
A lab assistant finds an unmarked beaker
containing a clear, viscous liquid.The assistant
further tests the unknown liquid to find that it
is non-flammable, and slightly acidic (pH = 6.2),
and has a boiling point of 150 oC.
What chemical is most likely in the unmarked
beaker?
a) H2O
b) CO2
c) H4O2
d) 2(HO)
On the other hand, one-best-answer MCQs can be qualitatively
ranked from “least correct” to “most correct”

28. For Discussion…
• Which MCQ format would do a better job at
measuring higher-order thinking skills (e.g.
application of knowledge)?Why?
• Would the preferred type of MCQ differ by
field of study?Why or why not?

29. Recommendation
• Experts recommend that test writers do not use true / false MCQs
• Choosing the single “correct” answer is problematic when the concept is not actually
an absolutely “true” or “false” dichotomy.
Example:
A student collects information on the socioeconomic status of a group of 1000
incoming first-year college students. She wishes to create a report on the typical family
income for her sample.
What would be the most accurate way to report this data?
a) Report the mean of the income
b) Report the range of the incomes
c) Report the median of the incomes
d) Report the standard deviation of the incomes

30. Writing Effective Item Stems
• A traditional format for a one-best-answer MCQ is to have a stem followed
by the lead in question.
For Example…
• A research team conducts a randomized, prospective study to compare a 1-day, 4-drug regimen with a 7-day, 3-drug
regimen in their efficacy at eradicating a particular infection.They are hypothesizing that the two groups would not differ
in the proportion of patients whose infection was eradicated by a clinically meaningful amount (+/- 15% eradication
percentage)
What statistical analysis would be the most appropriate to way to test this hypothesis?
a) Chi SquareTest of independence
b) Independent samples t-Test
c) Non-inferiority test for proportions
d) Equivalence test for proportions

31. Key Points of Effective Stems
• Be clear, concise, and avoid vague terms
• Define all necessary terms & avoid jargon
• Ask a complete question, that is, the student should not have to read the response
options to understand what they’re being asked.
• Avoid using negatively-worded stems such as “Which of these are not…” or
“All of these except”
• Avoid “Which of the following…”
• Try not to introduce construct irrelevant difficulty
• Items should focus on important concepts only
• Avoid verbal or grammatical associations between the stem and answers

32. So,What DoesThat Mean?
• In a research study, the age of the participants was 26
years +/- 5 years (mean +/- standard deviation).Which
of the following statements is the most correct?
• It is 95% certain that the true mean lies within the
interval of 16-36 years.
• Most of the patients were aged 26 years; the remainder
were aged between 21 and 31 years.
• Approximately 95% of the patients were aged between
16 and 36 years.
• No patients were younger than age 16 or older than age
36.
Vague Stem Example
Adapted from Windish, Hout, & Green 2007; & Ayers, Schulzinger, de laTeja, & GrayWhite, 2009)

33. So,What DoesThat Mean?
• In a research study, the age of the participants was 26
years +/- 5 years (mean +/- standard deviation).Which
of the following statements is the most correct?
• It is 95% certain that the true mean lies within the
interval of 16-36 years.
• Most of the patients were aged 26 years; the remainder
were aged between 21 and 31 years.
• Approximately 95% of the patients were aged between
16 and 36 years.
• No patients were younger than age 16 or older than age
36.
Vague Stem Example
Negatively-Worded
Stem Example
Adapted from Windish, Hout, & Green 2007; & Ayers, Schulzinger, de laTeja, & GrayWhite, 2009)
• All of the statements about women’s rights after the
Revolution are true except…
• Married women could not sign contracts or own
property
• Women could not vote in Massachusetts
• Women were not offered higher education
• Women could not vote in any state

34. Construct Irrelevant Difficulty?
• Several Sources
• Unnecessary background information
• Too little background information (i.e. vague
stem)
• Concepts that do not belong to the construct of
interest (use a blueprint!)
• Grammatical, spelling, or other language
difficulties
• “Speededness”
• Example…

35. The Bottom Line
“Remove all the barriers that will keep
a knowledgeable student from
answering the item correctly...[and]
Remove all clues that will help a less-
than-knowledgeable student answer
the item correctly”
(Suskie 2009, pg. 170)

36. For Discussion
• Time to Share, nobody’s perfect!
• We have all written questions that violate one
or more of these guidelines whether we know
it or not.Thinking back on assessments that
you have created, what are some of the more
common “errors” you find yourself making?
• My own example(s)…

37. Writing Effective
Response Options

38. Dos & Don’ts of Effective Response Options
Dos
• Order responses logically
• Use vertical rather than horizontal responses
• Make all options similar length
• Write distractors that are intrinsically possible
or true statements
• Use homogeneous distractors
Don’ts
• Use “all of the above” or “none of the above”*
• Use verbal clues or absolutes such as “always,”
“never,” “usually,” etc
• Use two-part or double-barreled options*
• Use response options that can be answered by
looking at other questions
• Include grammatical “cues” in the response
options
Case & Swanson (1998); Suskie (2009); & Brunnquell, A., Degirmenci,
U., Kreil, S., Kornhuber, J., & Weih, M. (2011).

39. Dos & Don’ts
A Couple of Examples
Do…
• About 25 of the 92 natural elements are known
to be essential to life.Which four elements
make up approximately 96% of living matter?
a) Carbon, sulfur, phosphorus, hydrogen
b) Carbon, hydrogen, nitrogen, oxygen
c) Carbon, oxygen, hydrogen, calcium
d) Carbon, sodium, chlorine, nitrogen
Don’t…
• Which of the following statements best distinguishes
hypotheses from theories in science?
a) Hypotheses usually are relatively narrow in scope;
theories have broad explanatory power.
b) Theories are hypotheses that have been proved
c) Theories are proved true in all cases; hypotheses are
usually falsified by tests.
d) Hypotheses are guesses; theories are correct answers
make distractors similar length “longest answer = correct answer”
Adapted from multiple exams in a Biology 1001 Course, Spring 2013; North
HennepinCommunityCollege; Brooklyn Park, MN 55445

40. Dos & Don’ts
A Couple of Examples
Write plausible options Use “all (none) of the above”
• Which thinker had a major impact on early-
twentieth-century writers, leading them to
reimagine human identity in radically new ways?
a) Sigmund Freud
b) Sir James Frazer
c) Friedrich Nietzsche
d) All of the above
• Which of the following phrases best characterizes the
late-nineteenth century aesthetic movement which
widened the breach between artists and the reading
public, sowing the seeds of modernism?
a) Art for intellect’s sake
b) Art for God’s sake
c) Art for the masses
d) Art for art’s sake
Adapted from the Norton Anthology of English Literature Multiple Choice
Quizzes (http://www.wwnorton.com/college/english/nael/quiz.htm)
Do… Don’t…

41. Challenges to
Writing MCQs
“Interlocking” Items
AvoidingTestwise Students

42. “Interlocking” or “Interdependent” Items
• Items can become interlocked or dependent on one another when:
• Response options or stems of one item are related to (or give the answers to) another question
• The same passage, calculation, graph, etc. is used for multiple items
• Using common examples, passages, etc. is an attractive option when writing tests for
efficiencies sake.
• However, it disadvantages students who don’t understand the example (even when they may know
the concept)
• Interlocking items are especially a concern when using Item ResponseTheory (IRT)
approaches to test construction, which assumes all items are locally independent of one
another.
(DeAyala, 2009; DeMars, 2010; & Suskie, 2009)

43. An Example of Interlocking Items
1.What is the most likely value for
the slope of this graph?
2. What is the most likelyY-
intercept for the graph above?
3. A mathematics faculty member
has 10 years of experience at the
university. What should be her
approximate income?
• An institutional researcher at a large university is conducting a salary
equity analysis on the university employees. One of these analyses is
to look at the relationship between employee income and years of
experience. Use their results below to answer questions 1-5
Image from http://allpsych.com/researchmethods/images/scatterplot.gif

44. “Cheating”
Avoiding GivingTestwise Students an Easy Answer
• There are several specific errors in response
option writing that lead students to pick up
on the correct answer without actually
knowing the concept in question.This
“testwiseness” introduces a host of issues
with test reliability and validity by
inadvertently lowering the difficulty of the
test.

45. “Cheating”
Avoiding GivingTestwise Students an Easy Answer
• Grammatical cues
• Testwise students will easily eliminate options that do not grammatically connect to
their stem
• Logical cues—not all answers are plausible or can be ranked in “correctness”
• Often seen when test writers need to quickly think of the “D” or “E” option just to have
4-5 response options.
• Absolute terms such as “always” and “never”
• Testwise students are far more likely to eliminate options that deal in absolutes, but
this flaw can be avoided through including the words in the stem rather than the
options.
Key issues related to testwiseness (Case & Swanson, 1998)

46. “Cheating”
Avoiding GivingTestwise Students an Easy Answer
• Long correct answer
• Testwise students can easily see when the correct answer is written more completely, more
detailed, or otherwise longer than the others.
• Repeated words
• If a response option uses the same word as the stem, then it is more likely to be the correct
answer; however, repeating the word in all options is a great strategy.
• Convergence
• Correct answer is the option that has the most in common with other options such as the
middle number in a math MCQ (i.e. it won’t be an extreme number)
Key issues related to testwiseness (Case & Swanson, 1998)

47. For Discussion
How do you avoid testwiseness in your own assessment practices?

48. FinalThoughts
What is theTakeAway Message Here?

49. Feeling Overwhelmed?
• Never fear, no test is perfect…
• As an example, a recent review found at least three formal item writing flaws in all 40
of the continuing medical education items published in the New EnglandJournal of
Medicine (Stagnaro-Green & Downing, 2006).

50. A Couple Parting Recommendations
• Always begin with a clear set of learning objectives and test blueprint
• Write items in teams, and seek a peer reviewer whenever possible
• Work to observe some of the basic dos and don’ts we covered today, but remember to
always balance rigor with practicality!
• Developing assessments, just like conducting assessments, is an iterative process.As
you conduct assessment within a cycle of feedback and revision, so must you make
changes to the instrument itself to generate a better product.

51. Final Activity
MCQ 911!
• You’ve been asked by a colleague to review their latest classroom
assessment instrument. He has not had any formal training in writing test
questions, and is very worried he will not get accurate measurements of his
students’ knowledge.
• Use what you have learned about writing effective MCQ items to give your
colleague’s assessment some “emergency care.”

52. Final Activity
MCQ 911!
• Instructions:
• Review the MCQs in your groups
• What are the writing flaws your colleague has made in their items?Which ones are the most severe?
• What suggestions or changes would you have for you colleague in order to fix some of these flaws?
• Things to look for:
• Vague, confusing, or negatively-worded stems
• Response option length
• “All (none) of the above” options
• Cues for testwise students such as grammatical cues, logical cues, “long-answer=right answer”, and
convergence
• Construct irrelevant difficulty (look at the “TestTopics” for a guide)
• Interlocking items

53. References
• Brunnquell, A., Degirmenci, U., Kreil, S., Kornhuber, J., & Weih, M. (2011). Web-based application to eliminate five contraindicated multiple-
choice question practices. Evaluation & the health professions, 34(2), 226–38. doi:10.1177/0163278710370459
• Case, S., & Swanson, D. (1998). Constructing written test questions for the basic and clinical sciences. Retrieved from http://ibmi3.mf.uni-
lj.si/mf/fakulteta/prenova/stomatologija/mcq.pdf
• Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 309–319.
doi:10.1037/1040-3590.7.3.309
• DeMars, C. (2010). Item response theory. Oxford University Press, USA.
• De Ayala, R.J. (2009). The theory and practice of item response theory. Guilford Press, USA.
• Downing, S., & Baranowski, R. (1995). Item type and cognitive ability measured: the validity evidence for multiple true-false items in medical
specialty certification. Applied measurement …. Retrieved from http://www.tandfonline.com/doi/full/10.1207/s15324818ame0802_5
• Downing, S. M. (2005).The effects of violating standard item writing principles on tests and students: the consequences of using flawed test
items on achievement examinations in medical education. Advances in health sciences education : theory and practice, 10(2), 133–43.
doi:10.1007/s10459-004-4019-5
• Fink, L.D. (2003) Creating Significant Learning Experiences: An Integrated Approach to Designing Courses. John Wiley & Sons, Inc, USA.
• Jozefowicz, R. F., Koeppen, B. M.,Case, S.,Galbraith, R., Swanson, D., & Glew, R. H. (2002).The quality of in-house medical school
examinations. Academic medicine : journal of the Association of American Medical Colleges, 77(2), 156–61. Retrieved from
http://www.ncbi.nlm.nih.gov/pubmed/11841981
• Suskie, L. (2009). Assessing Student Learning: A Common Sense Guide, 2nd Edition. John Wiley & Sons, Inc, USA.
• Windish, D. M., Huot, S. J., & Green, M. L. (2007). Medicine Residents’ Understanding of the Biostatistics and Results in the Medical
Literature. JAMA:The Journal of the American Medical Association, 298(9), 1010–1022. Retrieved from http://jama.ama-
assn.org/content/298/9/1010.abstract