This document discusses how statistics can be misleading if not properly interpreted. It provides principles for interpreting education statistics, such as questioning statistics that don't provide full context, being wary of "mutant statistics" that change over time, and ensuring numbers and rhetoric match. It cautions about comparing different groups and warns that aggregate data can obscure trends within subgroups using an example of Simpson's Paradox.

1. Don't Let
Yourself
Get Snookered
by Education
Statistics!
By Gerald W. Bracey
From Principal
I N seventeen-century France
and England, some people took
to collecting numbers they
thought would reflect the health of
the state—births, deaths, mar-
known as statists. They were usu-
ally collected—or created—for a
purpose, sometimes benign, some-
times not.
Shortly after the creation of sta-
riages, and so on—that came to be tistics, British Prime Minister Ben-
Gerald W. Bracey (gbraceyl@verizdn.net), an independent education re-
searcher based in Alexandria, Virginia, is author of Reading Educational
Research: How to Avoid Getting Statistically Snookered (Heinemann, 2006),
from which this article is adapted. Condensed from Principal, 85 (May/June
2006), 24-27. Published by the National Association of Elementary Schooi
Principals, 1615 Duke St., Alexandria, VA 22314.
44 www.eddlgest.com

2. Grasping Education Statistics
jamin Disraeli is alleged to have ric, it lists 13 indicators of risk. All
declared, "There are three kinds of relate to test scores and many of
lies: lies, damned lies, and statis- them are highly selective.
tics." For example: "There was a steady
I Statistics have always had to de- decline in science achievement
fend their honor, which makes it scores of U.S. 17-year-olds as mea-
curious that so many people are sured by national assessments...."
accepting of statistics despite This was true, but why did the re-
Disraeli's aphorism. That certainly port highlight only science and only
seems to be true about education 17-year-olds?
statistics, and especially true when Well, it seems that there was no
eiducation statistics imply some- "steady decline" in science for 9-
thing negative about American pub- and 13-year-olds, the other ages
lic schools. tested at the time. There also was
no hint of decline for any of the
Lacking the Time three age groups for reading or
Sometimes we accept statistics math. The commissioners who
because we are not in a position to wrote the report had in their pos-
challenge them, although this is far session nine trend lines (three sub-
less true today, thanks to a variety jects by three ages), only one of
of online research tools. Other which could be used to support
times—and I think this is especially crisis rhetoric. That was the only
true for harried principals—we ac- one they used.
cept them because we lack the time
to ferret out the truth. Beware of mutant statistics.
I To interpret statistics in the Perhaps the most dangerous statis-
hope of giving non-researchers a tic is the "mutant statistic." This is
time-saving tool with which to ap- a figure that begins life as an honest
proach statistics with both skepti- number but somehow morphs into
cism and wisdom, I have identified something inaccurate.
what I call my Principles of Data An example is the widely circu-
Interpretation. lated statistic that 150,000 Ameri-
can women die each year of anor-
The first principle is to question exia. But a visit to ihe Statistical
statistics that, whiie accurate Abstract of the United States, an an-
enough, do not teii the whole story. nual compendium from the Census
For example, the 1983 federal gov- Bureau, showed that 8,500 women
ernment report, A Nation at Risk, is aged 15 to 24 die each year of all
a veritable treasury of slanted, spun, causes, as do 47,000 women aged
and distorted statistics. After an 25 to 44.
opening blast of cold-warrior rheto- Given that anorexia is a disorder
September 2006 45

3. THE EDUCATION DIGEST
usually found in young women, and dren because they believe they are
given that 55,500 women aged 15 to better."
44 die of all causes, how likely is it Elxcept that they don't choose
that 150,000 women annually die of private schools. If you look at
anorexia alone? In fact, the initial Doyle's own numbers, you see that
and possibly legitimate statistic was 10.6% of public school teachers and
that 150,000 American women suf- 12.1 % of the general public had chil-
fer from anorexia. dren in private schools in 2000,
Mutant statistics afflict educa- compared with 12.1% of teachers
tion. Here's an example: Columnist and 13.1% of the general public in
George Will, writing in The Wash- 1990.
ington Post, reported that nearly What is remarkable about these
half of the public school teachers in numbers is that, despite all the
Chicago sent their kids to private bashing of public schools and soar-
schools. This was true; the figure at ing rhetoric about vouchers and
the time was 43%—the highest pro- privatization in the 1990s, teachers
portion in the nation. used private schools less than the
But six months later. Will mu- general public, and that a smaller
tated the statistic when he wrote, proportion of both teachers and
"nationally, almost half of all urban the general public were using
public school teachers with private schools in 2000 than in
school-age children send their chil- 1990.
dren to private schools." This was
nof true. But, as with drug-resistant Are group comparisons valid?
bacteria, this mutant has been hard The example above leads to yet
to kill. another Principle of Data Interpre-
tation: When comparing groups,
Do the numhers match the make sure the groups are compa-
rhetoric? This leads to another Prin- rable. In this case, public school
ciple of Data Interpretation: Make teachers and the general public are
certain the rhetoric and the num- not truly comparable. While slightly
bers match. Those with agendas more than a quarter of the general
often use rhetorical flourishes to public has at least a bachelor's de-
distract the reader away from the gree, virtually all public school
actual numbers. teachers do. And while almost 20%
Denis Doyle, from whom Will got of the general public lives in pov-
his original statistic, wrote, "With erty, teachers on average make
teachers choosing private schools, about 2.5 times the poverty thresh-
the truth is self-evident: while they old and are often not the sole wage
work in public schools, they choose earner in a family.
private schools for their own chil- As a variant on the principle of
46 www.ed(ilgest.com

4. Grasping Education Statistics
group comparisons, make sure ever the aggregate data show one
groups that are comparable Initially pattern, and subgroup data, the
stay comparable over time. Con- opposite.
sider this statistic: In 2004, Massa- How could this happen? The
chusetts claimed that 96% of high resolution of the paradox lies in the
school seniors passed its gradua- changing makeup of the two groups.
tion exit test. The figure was true, In 1981, whites made up 85% of SAT
but only for those who were se- test-takers. If we compare the 1981
niors at the start of the 2003-04 scores with those in 2005, when
school year. only 63% of the test-takers were
, To get a more meaningful pic- white, we see Simpson's Paradox in
ture of the condition of education action. The same result will also
as it exists here, we should ask be seen in any examination of scores
what percentage of the class of 2004 from the National Assessment of
as ninth-graders graduated in 2004? Educational Progress (NAEP): Eth-
In Massachusetts, the class of 2004 nic group increases are consistently
numbered about 60,000 seniors, greater than national increases.
while it once contained 78,000 ninth-
graders. Where did those other Do the arithmetic. There are
18,000 students go? many more Principles of Data Inter-
Looking at the number of ninth- pretation than I have space to
graders in the class of 2004 com- present here, but here's one you
pared with the numbers who were can work out for yourself. I call it
graduating, the graduation rates "Do the arithmetic."
range from 54% for Latino students In 1994, an author wrote in a
to 80% for white students, with an respected educational periodical,
overall rate of 74%, considerably "Every year since 1950, the number
lower than the 96% which was of American children gunned
claimed by the state and praised by down has doubled." This statement
test advocates. suffers a failing of many statistical
statements—a lack of specificity.
I Watch out for Simpson's Para- What ages define "children" and
dox. It is particularly important to what does "gunned down" mean?
watch for changes in groups over Now sit down with a piece of
time in an ethnically diversifying paper and a calculator. On the pa-
country like the United States. Look- per, list the years from 1950 to 1994
ing only at changes at the national (the year of the article). Now, as-
level can obscure changes that are sume that one child was gunned
tciking place within ethnic groups. down in 1950 and double that
The phenomenon known as number each successive year. Have
Sitnpson's Paradox occurs when- fun! effl
September 2006 47