1. DESIGNING AND
CONDUCTING
FORMATIVE
EVALUATIONS

2. OBJECTIVES
• Describe the purposes for and various stages of
formative evaluation of instructor-developed
materials, instructor- selected materials, and
instructor- presented instruction. .
• Describe the instruments used in a formative
evaluation. .
• Develop an appropriate formative evaluation plan
and construct instruments for a set of instructional
materials or an instructor presentation. .
• Collect data according to a formative evaluation
plan for a given set of instructional materials or
instructor presentation.

3. FORMATIVE EVALUATION DESIGNS
• Focusing the design only on the goals and
objectives of the instruction would be too limited.
Data on learners’ achievement of goals and objec-
tives would be insufficient, though important,
because these data will only provide information
about where errors occur rather than why they
occur.

4. 5 GENERAL QUESTIONS THAT SHOULD
BE ASKED ABOUT EACH COMPONENT
OF THE MATERIALS?
• 1. Are the materials appropriate for the type of learning
outcome? Specific prescrip-tions for the development of
materials were made based on whether the objectives
were intellectual or motor skills, attitudes, or verbal
information. You should be con-cerned whether the
materials you produced are indeed congruent with
suggestions for learning each type of capability. The
best evaluator of this aspect of the mate-rials would
undoubtedly be an expert in the type of learning
involved.
• 2. Do the materials include adequate instruction on the
subordinate skills, and are these skills sequenced and
clustered logically? The best evaluator for this area of
questions would be an expert in the content area.

5. CONTINUED
• 3. Are the materials clear and readily understood by
representative members of the target group? Obviously, only
members of the target group can answer these ques-tions.
Instructors familiar with target learners may provide you with
preliminary information, but only learners can ultimately judge
the clarity of the materials.
• 4. What is the motivational value of the materials? Do learners
find the materials relevant to their needs and interests? Are
they confident as they work through the materials? Are they
satisfied with what they have learned? Again, the most
appropriate judges of these aspects of the materials are
representative members of the target group.
• 5. Can the materials be managed efficiently in the manner
they are mediated? Both target learners and instructors would
be appropriate to answer these questions.

6. ONE- TO- ONE EVALUATION WITH
LEARNERS
• Criteria
• Selecting Learners
• Data Collection
• Procedures
• Assessments and Questionnaires
• Learning Time
• Data Interpretation
• Outcomes

7. CRITERIA
• During the development of the instructional strategy
and the instruction itself, designers and developers
make a myriad of translations and decisions that link
the content, learners, instructional format, and
instructional setting. The one- to- one trials provide
designers with their first glimpse of the viability of
these links and trans-lations from the learners’
perspective. The three main criteria and the
decisions de-signers will make during the evaluation
are as follows:

8. THE THREE MAIN CRITERIA AND THE
DECISIONS DE-SIGNERS WILL MAKE
DURING THE EVALUATION
• 1. Clarity: Is the message, or what is being
presented, clear to individual target learners?
• 2. Impact: What is the impact of the instruction on
individual learner’s attitudes and achievement of
the objectives and goals?
• 3. Feasibility: How feasible is the instruction given the
available resources ( time/ context)?

9. SELECTING LEARNERS
• One of the most critical decisions by the designer in
the formative evaluation is the selection of learners
to participate in the study. This is not an experiment;
there is no need for random selection of large
numbers of learners. Actually, the designer wants to
select a few learners who represent the range of
ability in the group because prior learning or ability
is usually one of the major determiners of ability to
learn new skills and information.

10. DATA COLLECTION
• The first category, message, relates to how clear the
basic message is to the learner determined by such
factors as vocabulary, sentence complexity, and
message structures. Regardless of whether the
learner reads, hears, or sees the message, he or she
must be able to follow it.

11. DATA COLLECTION
• The second category, links, refers to how the basic
message is tailored for the learner, including
contexts, ex-amples, analogies, illustrations,
demonstrations, and so forth. When these links are
also unfamiliar to the learner, the basic message will
undoubtedly be more complex.

12. DATA COLLECTION
• The third area, procedures, refers to characteristics
of the instruction such as the sequence, the size of
segment presented, the transition between
segments, the pace, and the vari-ation built into the
presentation. The clarity of instruction may change
for the learner when any one of these elements is
inappropriate for her or him.

13. PROCEDURES
• The typical procedure in a one- to- one evaluation
is to explain to the learner that a new set of
instructional materials has been designed and that
you would like his or her reaction to them. You
should say that any mistakes that learners might
make are probably due to deficiencies in the
material and not theirs. Encourage the learners to
be relaxed and to talk about the materials. You
should have the learners not only go through the
instructional materials but also have them take the
test( s) provided with the materials.

14. ASSESSMENTS AND QUESTIONNAIRES
• After the students in the one- to- one trials have
completed the instruction, they should review the
posttest and attitude questionnaire in the same
fashion. After each item or step in the assessment,
ask the learners why they made the particular
responses that they did. This will help you spot not
only mistakes but also the reasons for the mistakes,
which can be quite helpful during the re-vision
process.

15. LEARNING TIME
• One design interest during one- to- one evaluation
is determining the amount of time required for
learners to complete instruction, which is a very
rough estimate, because of the interaction
between the learner and the designer. You can
attempt to subtract a certain percentage of the
time from the total time, but experience has
indicated that such estimates can be quite
inaccurate.

16. DATA INTERPRETATION
• The information on the clarity of instruction, impact
on learner, and feasibility of instruction needs to be
summarized and focused. Particular aspects of the
instruction found to be weak can then be
reconsidered in order to plan revisions likely to
improve the instruction for similar learners.

17. OUTCOMES
• The outcomes of one- to- one trials are instruction
that
• ( 1) contains appropriate vocabulary, language
complexity, examples, and illustrations for the
participating learner;
• ( 2) either yields reasonable learner attitudes and
achievement or is revised with the objective of
improving learner attitudes or performance during
sub-sequent trials; and
• ( 3) appears feasible for use with the available
learners, resources, and setting. The instruction can
be refined further using small group trials.

18. FIELD TRIAL
• In the final stage of formative evaluation the
instructor attempts to use a learning con-text that
closely resembles the intended context for the
ultimate use of the instructional materials. One
purpose of this final stage of formative evaluation is
to determine whether the changes in the instruction
made after the small group stage were effective.

19. CHAPTER SUMMARY
• Formative evaluation of instructional materials is
con-ducted to determine the effectiveness of the
materials and to revise them in areas where they
are ineffective. Formative evaluations should be
conducted on newly developed materials as well
as existing materials that are selected based on the
instructional strategy. Evaluations are necessary for
both mediated and in-structor presented materials.
The evaluations should be designed to produce
data to pinpoint specific areas where the instruction
is faulty and to suggest how it should be revised.

20. CONTINUED
• An iterative process of formative evaluation con-taining
at least three cycles of data collection, analy-sis, and
revision is recommended. Each cycle focuses on
different aspects of quality. The first cycle, one- to-one
evaluation, is conducted to pinpoint gross errors in the
materials. These errors typically relate to both the clarity
of vocabulary, concepts, and examples used, and the
motivational value of all five compo-nents of the
instructional materials. Evaluations can also be
conducted with content experts and individ-uals familiar
with the characteristics of target learn-ers. One- to- one
evaluations must be conducted with representatives of
the target population. An interac-tive interview process
is used so the evaluator can learn what was wrong with
the materials and why it was wrong.