1. Cilacap, February
2017
Compiled by: Farida Fahmalatif

2. HOW TO ARRANGE A GOOD TEST
(Compiled: Farida Fahmalatif)
Every tester writes test cases however many times the test cases are rejected by reviewers
because of bad quality, in order to write good test cases one should know what are the
characteristics of a good test case.
A good test case has certain characteristics which are:
1. Should be accurate and tests what it is intended to test.
2. No unnecessary steps should be included in it.
3. It should be reusable.
4. It should be traceable to requirements.
5. It should be compliant to regulations.
6. It should be independent i.e. You should be able to execute it in any order without any
dependency on other test cases.
7. It should be simple and clear, any tester should be able to understand it by reading once.
8. Now keeping in mind these characteristics you can write good and effective test cases.
A. MULTIPLE CHOICE QUESTIONS
Multiple choice questions are often called fixed choice, selected response or multiple choice
items because they are not always questions, and they require students to select from among
various options that are presented to them. The options are fixed.
These items remain important because they can be scored rapidly, providing quick feedback
to students. Also, they are efficient when assessing large numbers of students over broad
content.
One drawback is that constructing multiple choice items well requires plenty of time for
writing, review, and revision. A time-saving tip is to write a few items each day while
preparing for class or after class, so that the material is fresh in your mind. The items will
then most likely reflect what you emphasized in class, which is fairer for the students. If you
construct the items so that they can be easily shuffled, like on index cards or software with
easy cut and paste, you can simply shuffle items around to build quizzes and tests later.

3. An important consideration in constructing multiple choice items is to make them measure
learning rather than test-taking skills of “test wise” students. The suggestions here are
designed to help you with this, but first some vocabulary needs to be introduced.
1. The Advantages of Multiple Choice
Multiple choice test questions, also known as items, can be an effective and efficient way to
assess learning outcomes. Multiple choice test items have several potential advantages:
Versatility: Multiple choice test items can be written to assess various levels of learning
outcomes, from basic recall to application, analysis, and evaluation. Because students are
choosing from a set of potential answers, however, there are obvious limits on what can be
tested with multiple choice items. For example, they are not an effective way to test students’
ability to organize thoughts or articulate explanations or creative ideas.
Reliability: Reliability is defined as the degree to which a test consistently measures a
learning outcome. Multiple choice test items are less susceptible to guessing than true/false
questions, making them a more reliable means of assessment. The reliability is enhanced
when the number of MC items focused on a single learning objective is increased. In
addition, the objective scoring associated with multiple choice test items frees them from
problems with scorer inconsistency that can plague scoring of essay questions.
Validity: Validity is the degree to which a test measures the learning outcomes it purports to
measure. Because students can typically answer a multiple choice item much more quickly
than an essay question, tests based on multiple choice items can typically focus on a
relatively broad representation of course material, thus increasing the validity of the
assessment.
The key to taking advantage of these strengths, however, is construction of good multiple
choice items.
A multiple choice item consists of a problem, known as the stem, and a list of suggested
solutions, known as alternatives. The alternatives consist of one correct or best alternative,
which is the answer, and incorrect or inferior alternatives, known as distractors.

4. 2. Construct an MCQ test
Constructing effective MCQ tests and items takes considerable time and requires scrupulous
are in the design, review and validation stages. Constructing MCQ tests for high-stakes
summative assessment is a specialist task.
For this reason, rather than constructing a test from scratch, it may be more efficient for you
to see what other validated tests already exist, and incorporate one into any course for which
numerous decisions need to be made.
In some circumstances it may be worth the effort to create a new test. If you can undertake
test development collaboratively within your department or discipline group, or as a larger
project across institutional boundaries, you will increase the test's potential longevity and
sustainability.
By progressively developing a multiple-choice question bank or pool, you can support
benchmarking processes and establish assessment standards that have long-term effects on
assuring course quality.
Use a design framework to see how individual MCQ questions will assess particular topic
areas and types of learning objectives, across a spectrum of cognitive demand, to contribute
to the test's overall balance. As an example, the "design blueprint" in Figure 2 provides a
structural framework for planning.

5. Figure 2: Design blueprint for multiple choice test design (from the Instructional Assessment
Resources at the University of Texas at Austin)
Cognitive
domains
(Bloom's
Taxonomy)
Topic
A
Topic
B
Topic
C
Topic
D
Total
items
Percentage
of total
Knowledge
1 2 1 1 5 12.5
Comprehension
2 1 2 2 7 17.5
Application
4 4 3 4 15 37.5
Analysis
3 2 3 2 10 25.0
Synthesis
1 1 2 5.0
Evaluation
1 1 2.5
TOTAL
10 10 10 10 40 100
Use the most appropriate format for each question posed. Ask yourself, is it best to use:
 a single correct answer
 more than one correct answer
 a true/false choice (with single or multiple correct answers)
 matching (e.g. a term with the appropriate definition, or a cause with the most likely effect)
 sentence completion, or
 questions relating to some given prompt material?
To assess higher order thinking and reasoning, consider basing a cluster of MCQ items on
some prompt material, such as:
 a brief outline of a problem, case or scenario

6.  a visual representation (picture, diagram or table) of the interrelationships among pieces of
information or concepts, or
 an excerpt from published material.
You can present the associated MCQ items in a sequence from basic understanding through
to higher order reasoning, including:
 identifying the effect of changing a parameter
 selecting the solution to a given problem, and
 nominating the optimum application of a principle.
Add some short-answer questions to a substantially MCQ test to minimise the effect of
guessing by requiring students to express in their own words their understanding and
analysis of problems.
Well in advance of an MCQ test, explain to students:
 the purposes of the test (and whether it is formative or summative)
 the topics being covered
 the structure of the test
 whether aids can be taken into the test (for example, calculators, notes, textbooks,
dictionaries)
 how it will be marked, and
 how the mark will contribute to their overall grade.
Compose clear instructions on the test itself, explaining:
 the components of the test
 their relative weighting
 how much time you expect students to spend on each section, so that they can optimise
their time.
3. WRITING EFFECTIVE MULTIPLE CHOICE ITEMS
The following tips can help you create multiple choice items to most effectively measure
student learning.

7.  Write the stem first, then the correct answer, then the distractors to match the correct
answer in terms of length, complexity, phrasing, and style
 Base each item on a learning outcome for the course
 Ask a peer to review items if possible
 Allow time for editing and revising
 Minimize the amount of reading required for each item
 Be sensitive to cultural and gender issues
 Keep vocabulary consistent with student level of understanding
 Avoid convoluted stems and options
 Avoid language in the options and stems that clues the correct answer
a. Writing effective multiple choice item stems:
 Format stems as clearly, concisely phrased questions, problems, or tasks if possible
 If phrasing the stem as a question requires extra words, make the stem into an
incomplete statement
 Include most information in the stem so that the options can be short
 When making the stem an incomplete statement, make sure the options follow the
stem in a grammatically correct manner
 Avoid using negatives in stems when possible
b. Writing effective multiple choice item options:
 Make sure there is only one best or correct answer
 Keep options parallel in format (if all options cannot be constructed in a parallel way,
make 2 options parallel to each other and the rest of the options parallel to each
other…the key is to construct options that do not stand apart from each other purely
because of style)
 Make options mutually exclusive (ex. Avoid 1-4, 2-5, 3-6 etc. as options because they
overlap)
 Make options of similar length and make sure the longest answer is only correct some
of the time
 Avoid “all of the above” or “none of the above”
 Avoid repeating the same words in all of the options by moving the words to the stem
 Arrange options in logical order if possible

8.  Avoid using specific language like “all,” “never,” or “always”
 Keep options plausible for students who do not know the correct option
 Options selected by very few students should be altered if the item is reused
The examples in constructing an effective stem
1. The stem should be meaningful by itself and should present a definite problem. A stem
that presents a definite problem allows a focus on the learning outcome. A stem that does not
present a clear problem, however, may test students’ ability to draw inferences from vague
descriptions rather serving as a more direct test of students’ achievement of the learning
outcome.

9. 2. The stem should not contain irrelevant material, which can decrease the reliability and
the validity of the test scores (Haldyna and Downing 1989).
3. The stem should be negatively stated only when significant learning outcomes require
it. Students often have difficulty understanding items with negative phrasing (Rodriguez
1997). If a significant learning outcome requires negative phrasing, such as identification of
dangerous laboratory or clinical practices, the negative element should be emphasized with
italics or capitalization.

10. 4. The stem should be a question or a partial sentence. A question stem is preferable
because it allows the student to focus on answering the question rather than holding the
partial sentence in working memory and sequentially completing it with each alternative
(Statman 1988). The cognitive load is increased when the stem is constructed with an initial
or interior blank, so this construction should be avoided.

11. 3. Constructing Effective Alternatives
1. All alternatives should be plausible. The function of the incorrect alternatives is to serve
as distractors,which should be selected by students who did not achieve the learning outcome
but ignored by students who did achieve the learning outcome. Alternatives that are
implausible don’t serve as functional distractors and thus should not be used. Common
student errors provide the best source of distractors.
2. Alternatives should be stated clearly and concisely. Items that are excessively wordy
assess students’ reading ability rather than their attainment of the learning objective

12. 3. Alternatives should be mutually exclusive. Alternatives with overlapping content may be
considered “trick” items by test-takers, excessive use of which can erode trust and respect for
the testing process.
4. Alternatives should be homogenous in content. Alternatives that are heterogeneous in
content can provide cues to student about the correct answer.

13. 5. Alternatives should be free from clues about which response is correct. Sophisticated
test-takers are alert to inadvertent clues to the correct answer, such differences in grammar,
length, formatting, and language choice in the alternatives. It’s therefore important that
alternatives
 have grammar consistent with the stem.
 are parallel in form.
 are similar in length.
 use similar language (e.g., all unlike textbook language or all like textbook language).
6. The alternatives “all of the above” and “none of the above” should not be used. When
“all of the above” is used as an answer, test-takers who can identify more than one alternative
as correct can select the correct answer even if unsure about other alternative(s). When “none
of the above” is used as an alternative, test-takers who can eliminate a single option can
thereby eliminate a second option. In either case, students can use partial knowledge to arrive
at a correct answer.
7. The alternatives should be presented in a logical order (e.g., alphabetical or numerical)
to avoid a bias toward certain positions.

14. 8. The number of alternatives can vary among items as long as all alternatives are
plausible. Plausible alternatives serve as functional distractors, which are those chosen by
students that have not achieved the objective but ignored by students that have achieved the
objective. There is little difference in difficulty, discrimination, and test score reliability
among items containing two, three, and four distractors.
Additional Guidelines
1. Avoid complex multiple choice items, in which some or all of the alternatives consist of
different combinations of options. As with “all of the above” answers, a sophisticated test-
taker can use partial knowledge to achieve a correct answer.
2. Keep the specific content of items independent of one another. Savvy test-takers can
use information in one question to answer another question, reducing the validity of the test.
Considerations for Writing Multiple Choice Items that Test Higher-order Thinking
When writing multiple choice items to test higher-order thinking, design questions that focus
on higher levels of cognition as defined by Bloom’s taxonomy. A stem that presents a
problem that requires application of course principles, analysis of a problem, or evaluation of
alternatives is focused on higher-order thinking and thus tests students’ ability to do such
thinking. In constructing multiple choice items to test higher order thinking, it can also be

15. helpful to design problems that require multilogical thinking, where multilogical thinking is
defined as “thinking that requires knowledge of more than one fact to logically and
systematically apply concepts to a …problem” (Morrison and Free, 2001, page 20). Finally,
designing alternatives that require a high level of discrimination can also contribute to
multiple choice items that test higher-order thinking.

16. Additional Resources
 Burton, Steven J., Sudweeks, Richard R., Merrill, Paul F., and Wood, Bud. How to
Prepare Better Multiple Choice Test Items: Guidelines for University Faculty, 1991.
 Cheung, Derek and Bucat, Robert. How can we construct good multiple-choice items?
Presented at the Science and Technology Education Conference, Hong Kong, June 20-
21, 2002.
 Haladyna, Thomas M. Developing and validating multiple-choice test items, 2nd edition.
Lawrence Erlbaum Associates, 1999.
 Haladyna, Thomas M. and Downing, S. M.. Validity of a taxonomy of multiple-choice
item-writing rules. Applied Measurement in Education, 2(1), 51-78, 1989.
 Morrison, Susan and Free, Kathleen. Writing multiple-choice test items that promote and
measure critical thinking. Journal of Nursing Education 40: 17-24, 2001.