1. Basic A ssessment Literacy
Everything
You W anted to Know
A bout Testing But W as
N ever T hat Interested

2. T est builders in empowered
desktop
Focus just on MC & D BQ s

3. Smartest kid in class

4. Sticky Ideas

5. Q uality data requires a
quality test
Q uality tests are not easy
to create

6. A ssessment literacy means:
5 ld e & n
B Mn a
in
E pat
l hn
e ?

7. A ssessment literacy means:
G t rg eedb in r a n b u
a e dpnal f mt aot
h in e o io
s dn ah vmn
t etciee et
u
U in t a es et rcst po oe
sgh s s n poeso rm t
e s m
mx u ah vmn
aimm ciee et

8. W hat’s a “ good test?”
“ The most basic and obvious
answer to that question is that
good tests measure what you
want to measure, and bad tests
do not.”
Ben C lay
Is This a Trick Question?

9. Q uestions to ask
W hy? W hat?
U sers/U ses H ow? T argets
Method
H ow Much?
Sample
A ccurate?
A void Bias

10. W hy? W hat?
U sers/U ses H ow? T argets
Method
H ow Much?
Sample
A ccurate?
A void Bias

11. W hy?
Is the Purpose for the assessment
clear?
oW hy are these targets being
assessed?
oW ho will use the information?
oW hat will the information be used
for?

12. W hy? W hat?
U sers/U ses H ow? T argets
Method
H ow Much?
Sample
A ccurate?
A void Bias

13. W hat?
A re the Targets clear and
appropriate?
oSpecific & clearly defined
expectations
oC enter on the core of the discipline
oA ligned to program / course / unit
level curriculum

14. Four target types
Knowledge
oG eorge W ashington / 1787 / U S
C onstitution
Reasoning
oEvaluatedifferences between
government types

15. Skills
oA nalyze a primary source document
Products
oC reate a reasoned argument for
allowing immigration reform

16. A n assessment that accurately
reflects its intended target is Valid

17. W hy? W hat?
U sers/U ses H ow? T argets
Method
H ow Much?
Sample
A ccurate?
A void Bias

18. H ow?
D ifferent targets require different
methods
oMatch appropriate method with
intended purpose and target

19. G eneral categories
O bjective
oStudent selects correct response
from several alternatives / supplies a
word or short phrase to answer a
question
oEasy to grade / hard to create

20. Subjective
oStudent organizes / presents an
original answer
oH ard to grade / easy to create

21. O bjective vs. Subjective
o“ Bydefinition, no test can be truly
objective”

22. Specific categories
Selected Response
oMultipleC hoice, T rue/False,
Matching, Fill-in-the-blank
Essays
oShort A nswer, Extended Response
Performance T asks
Personal C ommunication

23. Mark Pett ©2001

24. Let’s try it!
N ame the parts of the human
skeleton
oSelected response
A ppraise a composition on the basis
of its organization
oShort answer essay
D emonstrate safe lab skills

25. C ite four examples of satire in
Huckleberry Finn
oShort answer essay
D esign a logo for a web page
oPerformance task
D escribe the impact of a Bull Market
oExtended response essay

26. D iagnose a physical ailment
oExtended response essay
List important mental attributes
necessary in an athlete
oShort response essay
C ategorize great A merican fiction
writers

27. W hy? W hat?
U sers/U ses H ow? T argets
Method
H ow Much?
Sample
A ccurate?
A void Bias

28. H ow much?
H ow can we Sample appropriately?
o“ A ll assessments are just a small
part of an ‘ideal’ assessment of
infinite length.”

29. “ H ave I gathered enough
information of the right kind so
that I can draw confident
conclusions about student
achievement?”

30. W hy? W hat?
U sers/U ses H ow? T argets
Method
H ow Much?
Sample
A ccurate?
A void Bias

31. Mark Pett ©2001

32. A ccurate?
W hat are possible sources of
Interference or Bias?

33. A ssessment
oPoorly written, poor target-method
match, culturally biased,
inappropriate reading level
A ssessor
oIncorrectly scored

34. Student
oTest anxiety, cheat, guess,
interpretation different than intent
Environment
oC old, hot, poor light, time restraints,
noise, lunch, recess

35. Eliminating errors due to bias
increase the Reliability of our
assessments

36. W hat was new?
W hat was re-new?

37. Steps to Structured Response
test
Prepare a blueprint
Select material to assess
Build test items from propositions

38. Step 1
Prepare a blueprint

39. Cnn
ot t
e Ko &
nw Poes
rcs& Tt
oa
l
Tr t
ae
gs U drad
ne t
sn Ap a n
pl t
ic io
Fr s f
om o 1
0 5 1
5
gvrmn
oe et
n
S ut e f S
t c r oU
r u 7 5 1
2
gvrmn
oe et
n
Rh &
igts 8 5 1
3
r p nibits
e os il
s ie
o cizn
fites
Tt
oa
l 2
5 1
5 4
0

40. Step 2
Select material to assess

41. “ It is always tempting to emphasize
the parts of the course that are
easiest to test, rather than the
parts that are important to test.”
Ben C lay
Is This a Trick Question?

42. So what’s worth
assessing?

43. Review the C ase Study
W hich test items, if any, need to be
changed?
C an you think of a test that included
items you felt weren’t worth
learning or assessing?
H ow do you make decisions about
what’s worth assessing?

44. So . . . ?
A select sub-set of all possible
important elements
Professional judgment of relative
importance

45. Step 3
Build test items from propositions
o“ C learlystated sentences that
reflect elements of content aligned
to C ourse / U nit EU s & EQ s.”
oC reate more than blueprint requires

46. Examples
In a monarchy, the right to govern is
secured through birth.
The executive and legislative
branches differ in that the latter is
elected directly by the people.

47. T rue or False
In a monarchy, the right to govern is
secured through birth.
or
In a monarchy, the right to govern is
secured through the approval of
those governed.

48. Fill in the Blank
H ow is the right to govern secured
in a monarchy?

49. Multiple C hoice
H ow is the right to govern secured
in a monarchy?
A . W ith military power
B. T hrough birth
C . By popular vote
D . Through purchase

50. T rue or False
The executive and legislative
branches differ in that the latter is
elected directly by the people.
or
Members of executive and
legislative branches are both
elected directly by the people.

51. Fill in the Blank
Election of members of the
executive and legislative branches
differ in what way?

52. Multiple C hoice
Election of members of the executive
and legislative branches differ in what
way?
A . Legislators are restricted by term limits;
presidents are not
B. Legislators are elected directly; presidents
are not.
C . O ne must register to vote for legislators;
not for president
D . O ne must register to vote for presidents;

53. Franzipanics Q uiz

54. Specific assessment criteria

55. Multiple choice
A dvantages
oEffectivemeasurement
oReduces guessing
oC an cover wide range of content
D isadvantages
oD ifficult to create

56. Multiple C hoice suggestions
Stem is a clear, explicit & singular
question
U se at least four alternatives
A void grammatical / sentence
structural cues
H ighlight the negative in negatively
stated stems

57. A lternatives are the same format
and length
A ll alternatives should be plausible
A void answer patterns

58. Is the question any good?
Student who knows the content
will know the answer before
reading the alternatives

59. “ Your goal is not to trick students
or require them to make difficult
judgments about two nearly
equal options.”

60. “ Your goal is to write questions that
students who understand the
material will answer correctly and
students who do not understand
will answer incorrectly.”
D r. John Johnson
Penn State U niversity

61. Bill W atterson ©1995

62. T rue or false
A dvantages
oC anmeasure large amounts of basic
knowledge
oStudents can process many questions
quickly
D isadvantages
oG uessing works
o

63. T rue & false suggestions
Statements are absolutely true or
false
Express item as simply & clearly as
possible
Measure only one idea per item
U se possible misconceptions as false
items

64. Matching
A dvantages
oMeasures large amounts of
knowledge level content
oU seful for factual content areas
D isadvantages
oC an be time consuming for students
oD oes not measure higher learning

65. Matching suggestions
U se items that share same
foundation of knowledge
A void grammatical cues
Keep stem list to10-15 items
Response list should be larger than
stem list

66. Fill-in-the-Blank
A dvantages
oG ood for 5 W s and H
oH elps minimize guessing
D isadvantages
oD ifficult to assess high levels of
learning
oMay have more than one correct

67. Fill-in-the-Blank suggestions
O mit only significant items from
stem
Insure only one correct answer
A void grammatical cues
Blank at stem’s end
Should count for more points

68. T hings to discuss
Possible accommodations
Student evaluations of test
Pilot version / full version
W ho takes the test
W ho administers the test