Results of high-stakes testing are the message, not the messenger

January 11, 2003

People who think everything in public education is absolutely wonderful and don't want to hear otherwise have latched on to a new study by Arizona State University educators on the supposed negative consequence of "high-stakes" testing. As The Denver Post headlined a New York Times story, "Rigorous testing impedes student progress."

They are delighted to be able to confuse the messenger with the message.

Audrey Amrein and David Berliner were commissioned by the National Education Association to investigate what happens to other educational indicators, such as results on the National Assessment of Educational Progress, when states attach high stakes to their tests. The authors define "high stakes" as essentially any consequences for anyone -- students, teachers, schools -- of doing well, or badly, on state tests.

So their list of high stakes includes an exam required for high-school graduation, state authority over low-performing schools or their staffs, bonuses or scholarships for high-performing students, their teachers or their schools, or the right of children to transfer out of failing schools.

Giving children scholarships for excellent academic performance? How dastardly!

Colorado is one of the states the authors list as having a relatively high number of such consequences, though imposed too recently to show any results in the study. However, the analysis is misleading. Colorado children do have the right to transfer out of failing schools, but only because they have the right to transfer out of any school, period.

It's not the only indication that the study was conducted to highlight a particular result. Berliner is the author of a fatuous book, Manufactured Crisis: Myths, Fraud and the Attack on America's Public Schools, which argues that the schools are perfectly fine and anyone who says otherwise is mean-spirited and politically motivated.

I'm not arguing that because he has a political agenda his conclusions are necessarily wrong. That's the ad hominem fallacy. But he starts from the premise that the schools are fine. If the tests show they're not fine, or even getting worse, then it must be the tests' fault. Logical, but wrong.

Amrein and Berliner themselves describe their results by saying that 67 percent of states that implemented high stakes for their tests showed a decline in fourth-grade math performance on NAEP afterward. That sounds quite alarming, but it greatly exaggerates what their data show.

Their survey includes 28 states, of which 16 showed no change in NAEP scores. Of the 12 that did, eight showed declines and four showed improvement. We know they aren't big on computational skills -- at one point, they calculate two-thirds as 66 percent -- but still, eight is not 67 percent of 28.

Moreover, that's the worst example. On eighth-grade math the balance was almost reversed, with five states improving and three declining, while fourth-grade reading was five versus five.

Overall, that is, no significant change is happening at the national level even though instate scores usually rise as teachers and students become more familiar with the test. That's interesting in its own right, but it falls far short of showing high stakes have negative consequences, or indeed, any consequences at all. How do NAEP results compare in states that don't impose high stakes? They don't say.

Not all states participate in NAEP -- Colorado, for example, sat out the 2000 and 2002 test -- but under the new federal education act participation will be required. NAEP scores are a useful external check on the quality of state testing programs.

The authors offer a hodgepodge of what they call "unintended and negative consequences" of high-stakes testing, for instance that some teachers and school officials cheat. Well, people cheat on their income taxes too, but that is not an argument against having income taxes. If teachers cheat, they should be fired. Indeed, if people have so little faith in themselves and their students that they believe they have to cheat in order to meet expectations, they shouldn't be teaching anyway. There's nothing soft about the bigotry of low expectations.

Teachers are "teaching to the test," Amrein and Berliner complain. It may have escaped their notice that that is not an unintended consequence. But they caricature the process as "requiring students to spend hours memorizing facts." I'm not so down on mere facts as they are, but more to the point, if the state tests are such that memorizing facts is the most useful form of preparation for them, then the tests need to be redesigned.

The authors also note evidence from some states that students are being kept back because they aren't ready to take high-stakes tests the following year. But what's the alternative? Surely these two can't believe that in the absence of the tests, the students would be prepared to pass them. No, they'd have been pushed forward unprepared. That's exactly what the tests are supposed to stop.

The study shows scattered evidence that the high school graduation rate and the tendency for marginal students to drop out and get a GED instead are made worse by the imposition of a high school exit exam. That could well prove true. The likely reason, though, is that these are students who might otherwise stay in school but get a meaningless diploma. High-stakes tests don't cause that problem; they reveal it.

Amrein and Berliner, we gather, would rather not know.

(from Gadfly, the e-mail newsletter of the Thomas Fordham Foundation)

"The Impact of High-Stakes Tests on Student Academic Performance," Great Lakes Center for Education Research and Practice, December 2002, http://www.greatlakescenter.org/research.htm

"High-Stakes Testing, Uncertainty, and Student Learning," Education Policy Analysis Archives, March 28, 2002, http://epaa.asu.edu/epaa/v10n18/

"Make-or-Break Exams Grow, But Big Study Doubts Value," by Greg Winter, The New York Times, December 28, 2002, http://www.nytimes.com/2002/12/28/education/28EXAM.html