It's a myth: High stakes cause test score inflation

It’s a myth:
High stakes cause test score inflation
Richard P. Phelps
researchED 2017 National Conference
7 October, 2017
Brooklyn, NY

Educational testing in the US: early 1980s
researchED, Brooklyn High stakes & test score inflation 7 October,
2017

Student testing with stakes
reintroduced late 1970s,
early 1980s
Debra P. v. Turlington
“Truth in testing” laws
2017
Educational testing in the US: 1980s

Residency in rural, poor Appalachia, 1980s
Surprised by claims that state and school district scored
“above average” on national tests
Investigated, all US states claimed to be “above average”
John J. Cannell, M.D.
researchED, Brooklyn High stakes & test score inflation 7 October, 2017

“Welcome to Lake Wobegon, where all the women
are strong, all the men are good-looking, and all
the children are above average.”
- Garrison Keillor, A Prairie Home Companion
researchED, October High stakes & test score inflation 7 October, 2017

Cannell’s
suspects
• Lax security
• Outdated or invalid norms
• Deliberate educator manipulation (i.e., cheating)

US Education Establishment Responds
2017

“While supporting Cannell’s
general finding … our
analyses lead us to
conclusions that are
different, and certainly
less sensational, than the
ones he reached.”
— Linn, Graue, Sanders ,
CRESST, 1990
“There are many reasons
for the Lake Wobegon
Effect, most of which are
less sinister than those
emphasized by Cannell.”
— Linn, CRESST, 2000

CRESST’s Lake
Wobegon suspects
Outdated or invalid norms
High stakes, that induce “teaching to the test”
(i.e., test coaching) under pressure
2017

“We know that tests that are
used for accountability tend to
be taught to in ways that
produce inflated scores.”
— Daniel Koretz, CRESST,
1992
“Corruption of indicators is a
continuing problem where tests
are used for accountability or
other high-stakes purposes.”
— Robert Linn, CRESST,
2000
2017

CRESST counters Cannell’s Lake Wobegon
study with their own, 1991
Students took test a few years. Scores rose. Then took
“competing test” district had used before. Scores fell.

CRESST 1991 “Generalization” Study
Unnamed school district
Unnamed tests
Neither replicable nor falsifiable
A conference presentation; not peer-reviewed.

CRESST 1991 “Generalization” Study
3 tests in the study
1.Annual NRT
2.Parallel form
3.A “competing” NRT

1991 CRESST “Generalization” Study

School district test was only “perceived to be high stakes.”

Study’s assumptions
1. Publication of aggregate results = “high stakes”
2. “Competing” NRTs should get same results
3. “Test coaching” improves scores
4. Low-stakes test scores are reliable and can be used
to benchmark unreliable high stakes scores
5. High-stakes cause test-score inflation?

Jim Popham “high stakes” definition 1987
... Such tests include the many statewide achievement
tests whose results are reported by local newspapers on a
school-by-school or district-by-district basis.”
2017
1. Publication of aggregate results = high stakes?

Jim Popham “high stakes” definition 1992
A test “subject to legal scrutiny.”
Tests such as those used “for employment, licensure, or a
high school graduation requirement”
2017

“High-stakes test. A test used to provide results that have
important, direct consequences for examinees,
programs, or institutions involved in the testing.” (p.176)
“Low-stakes test. A test used to provide results that have
only minor or indirect consequences for examinees,
programs, or institutions involved in the testing.” (p.178)
Standards for Educational and
Psychological Testing

“...tests taken to obtain
admission to an educational
program or taken during and
at the conclusion of a program
to obtain a qualification.”
“…high-stakes decisions, such as
whether a student will move on to
the next grade level or receive a
diploma.”

A high-stakes test is a test with important consequences
for the test taker. Passing has important benefits, such as
a high school diploma, a scholarship, or a license to
practice a profession.
Wikipedia
2017

2. Research: Comparability of different tests
Scores Comparable
?
Scores Not Comparable
NRTs
Freeman, Kuhs, Porter, Floden, Schmidt, Schwille
(1983); Debra P. v. Turlington (1984); Cohen,
Spillane (1993); La Marca, Redfield, Winter, Bailey,
and Despriet (2000); Wainer (2011)
Standards
Archbald (1994); Buckendahl, Plake, Impara, Irwin
(2000); Bhola, Impara, Buckendahl (2003); Phelps
(2005)
CRTs
Massell, Kirst, Hoppe (1997); Wiley, Hembry,
Buckendahl, Forte,Towles Nebelsick-Gullett (2015)

3. Research: Effects of test coaching
It works
Significant score
increase from learning
format tricks
Aldeman & Powers
(1980) Samson (1985)
Scruggs (1985)
Roznowski & Bassett
(1992) McMann (1994)
Holmes, Keffer (1995)
Camel & Chung (2002)
Filizola (2008)

4. Research: Low-stakes test reliability
Reliable
“no incentive to manipulate
scores”
Kipliinger, Linn (1992)
O’Neil, Sugre, Baker (1995) *
Hout, Elliot (2011)
* 1 of 2 groups
Not reliable
student effort varies;
scores easy to manipulate
Rothe (1947); Jennings (1953); Uguroglu,
Walberg (1979); Taylor & White (1981);
Arvey, et al. (1990); Schmit, Ryan (1992);
Brown & Walberg (1993); Kim, McLean
(1995), Wolf, Smith (1995), Wolf, Smith,
DiPaulo (1996); Schiel (1996); Sundre
(1999), Sundre, Moore (2002), Sundre, Wise
(2003); DeMars (2000), Wise (2006ª,
2006b), Wise, DeMars (2005, 2005, 2006,
2010), Wise, et al., (2009); Hoyt (2001);
Eklof (2006, 2007, 2010);
….....etc.

2017
“…for consequential exams, the average score on the
motivation scale was quite high with a low standard
deviation. Essentially, most of the students were displaying
uniformly high levels of motivation (i.e., ceiling effect).
However, for the nonconsequential groups, motivation
played an important role in predicting test performance. The
overall motivation scores for the no consequence groups
were lower than the motivation for the consequential groups,
with much greater variability.”
—Cole, Bergin, Whittaker (2008), p. 612
4. Research: Low-stakes test reliability

5. High stakes cause test score inflation?
Then, why no score
inflation with
certification and
licensure tests?

More left-out-
variable bias
CRESST’s Linn (2000) cites higher gains
on a federal anti-poverty program’s pre-
post testing over 9 months than over 12
as evidence of inflation

Cannell found score inflation in
elementary school tests in
dozens of states – none of
those tests had high stakes.
Cannell also found score
inflation in secondary school
tests in dozens of states –
only one had high stakes.
Test Score Inflation Occurs where
Security is Lax

Cannell’s test categorizations confirmed
2017

Confusions from misinformation
1. Tests sample from larger domains
2. Campbell’s Law
3. “Teaching to the test” & “Narrowing the curriculum”
4. Incentives and causes
5. Educators face many incentives; “high stakes” only one
6. Today’s tests have much higher stakes than past tests
1. No one wants to be responsible for test security

1. Tests only sample larger domains
"Tests are about making a measurement, and generally, tests
are trying to measure something huge." — Daniel Koretz
2017
TRUE of many tests, e.g.,
NRTs, aptitude, IQ tests
NOT TRUE of well-done
standards-based tests

2. Campbell’s Law — a truism
2017
"The more any quantitative social indicator is used for social decision-
making, the more subject it will be to corruption pressures and the more apt it
will be to distort and corrupt the social processes it is intended to monitor."
Social indicators can be beneficial:
- for understanding
- monitor progress
- benchmarking
- setting goals
- process improvements

3. Teaching the test; Narrowing the curriculum
2017

4. Incentives and causes
Question:
Do high stakes
present an
incentive to cheat
on tests?
Answer:
Of course they do

5. Educators face many incentives
Incentives of
test “stakes”
is just one

6. Today’s tests have higher stakes
2017
Exactly the
opposite is true.
Koretz: States in
1980s and 1990s were
“chicken feed”
compared to today’s
tests.

7. No one inside education wishes to be
responsible for test security
2017
… including test
development firms.

Large-scale test, tight security
2017

Large-scale test, lax security
2017

Harms of disinformation
1. Acceptance of low standard for research as valid
2. Unfairly discredits useful evaluation tool
3. Test security (in U.S.) remains shoddy
4. Teachers given mixed messages
5. Now spreading worldwide
6. Corruption of Test Standards barely averted

1. Acceptance of very low quality standard
for popular research results
CRESST studies:
- no controls
- secret test
- secret
location
- secret
definitions
Non-replicable,
Non-falsifiable

2. Uniquely useful evalution tool is
discredited
2017
…and, in the US, the
only objective measure
available to the public
(i.e., not under the
control of insiders).

3. Test security (in U.S.) remains shoddy
2017
ACT, SAT, PARCC, SBAC
now administered statewide
by schools, on varying
dates. Tests save money,
hassle, gain customers by
outsourcing (or, ignoring)
test security.

4. Teachers given mixed messages
“Teaching to the test” is
unethical; Don’t do it! Teach
content beyond the
standards.
“Teaching to the test works!
You and your students will
be better off if you do it!

5. Standards corruption barely averted

6. Disinformation spreading worldwide

• Motive alone is not sufficient
if test security is tight.
• Means and opportunity exist
only in the absence of
security measures and form
and item rotation.
Artificial test score gains (score inflation) are
caused by lax security; they require means
and opportunity.

Test Security in
South Carolina:
“Unlike their other two tests,
… teachers are allowed to look at test booklets,
… teachers may obtain test booklets before the day of testing,
… booklets are not sealed, and
… testing is not routinely monitored by state officials.
… Outside test proctors are not used,
… test questions have not been rotated every year, and
… answer sheets have not been scanned for suspicious erasures or
analyzed for cluster variance.
… There are no state regulations that govern test security and test
administration for norm-referenced testing done independently
in the local school districts.”
2017
Cannel’s score-inflated test

Test Security in
South Carolina:
“South Carolina also administers a graduation exam and a criterion
referenced test, both of which have significant security
measures.
… Teachers are not allowed to look at either of these two test
booklets,
… teachers may not obtain booklets before the day of testing,
… the graduation test booklets are sealed,
… testing is routinely monitored by state officials,
… special education students are generally included in all tests,
… outside test proctors administer the graduation exam, and
… most test questions are rotated every year on the criterion
referenced test.”
Tests not in Cannell’s study

Lessons Learned
If terms can be defined arbitrarily, and not specified, any
research result is possible.
Cleverly-disguised falsehoods and obfuscation can be well-
rewarded in US education schools (e.g., with endowed
professorships at Harvard and Stanford).
US education: Research quality
standards extremely low for
popular results; impossibly high
for unpopular results

http://nonpartisaneducation.org/Review/Articles/v6n3.htm
richard@nonpartisaneducation.org

It's a myth: High stakes cause test score inflation

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a It's a myth: High stakes cause test score inflation

Semelhante a It's a myth: High stakes cause test score inflation (20)

Mais de Richard P Phelps

Mais de Richard P Phelps (17)

Último

Último (20)

It's a myth: High stakes cause test score inflation

Notas do Editor