REAL PROBLEM WITH SAT POLITICAL, NOT EDUCATIONAL

Saturday, January 24, 2004

If you're interested in how standardized testing affects education - and since we're doing quite a lot of it, you should be interested -- you ought to read Richard Phelps' 2003 book, Kill the Messenger: The War on Standardized Testing.

Phelps has written extensively in defense of testing, but mostly from outside academia, which is probably a good thing (many of the columns on which the book is partly based are at at richardphelps.net on the Web). He sets out to show that the academic establishment, especially the parts of it located within university schools of education, is dominated by people who either believe testing is bad in itself -- they prefer more "authentic" forms of assessment such as portfolios -- or approve of testing only under such stringent conditions that in practice they never approve of it at all.

His secondary thesis is that media coverage of these issues is almost entirely one-sided.

"Most journalists," he writes, "seem to accept all anti-testing research at face value and only rarely is any counter to it sought or offered."

At least as far as the News is concerned, that overstates the case. Testing has been covered extensively all the time I've been here, since 1997, and in a much less one-sided way than the methodology he chose is likely to reveal. Nor do I have any reason to think that the News, as a major regional paper, is atypical in how it covers education.

But that's not to say he has no case. In fact, it's a pretty strong one, which is why you should know about it.

One of his case studies is the campaign to discredit the SAT. He has a devastating analysis of a Frontline documentary, Secrets of the SAT, which among other indicators of the producers' intent includes a count of the lines in the transcript of the show devoted to attacking the SAT -- 101 -- with the number of lines defending it -- 9.

Most of the criticisms aimed at the SAT are specious. It is just about as useful in predicting freshman college grades as high school grades are (and the two together are better yet). College admissions counselors know that, which is why in a survey more said admission test scores were "of considerable importance" than said the same about grades.

The unspoken difficulty with the SAT is not educational, it is political: namely, Asians and whites consistently score on average a couple of hundred points higher than blacks and Hispanics. And it isn't because the tests are biased, any more than scales are biased because they consistently show that men, on average, are heavier than women. Tests predict almost equally well for all races.

Nobody wants testing to reveal these differences, but nobody has any idea how to change them, either, so the only way out is to look for other excuses to lessen the importance of the SAT and other similar tests.

Phelps' effort to quantify the imbalance in news reporting is interesting, even though I think it is not quite so conclusive as he expected. He picked 52 people who could plausibly be consulted by journalists writing stories about standardized tests, half of them opponents and half of them supporters, and then checked each appearance of their names in the Dow Jones Interactive database of about 550 U.S. newspapers and magazines for the years 1998 through 2000. He found 614 articles that cited a testing opponent talking about testing, compared with 127 articles that cited a testing supporter talking about testing.

The method sounded reasonable, but the conclusion didn't. So I checked our internal archives for the top names on his list. The one that turned up most often in our files for those years, although not always in the context of opposition to testing, was Lorrie Shepard, a researcher at the University of Colorado at Boulder, who appeared in 17 of the 614 opponents' articles.

Phelps was gracious enough to search his records for me and found that 10 of Shepard's 17 articles appeared either in the News or The Denver Post. But our education reporters call Shepard not just because she's an expert, but because she's a local expert. For countervailing voices in their stories, they tend to have Colorado Gov. Bill Owens, a strong advocate for state testing, and State Education Commissioner William Moloney, who is responsible for implementing the state's testing program. But their comments don't get picked up by Phelps' method, nor would those of their counterparts in other states.

We exchanged e-mails about this, and Phelps said he thought readers would interpret such stories as exposing non-expert politicians who are supporting "a policy the smart expert knows is stupid."

I think it's much more likely that readers respect people who actually make policy in the real world more than they do professors of education who just write about it, but there's no way to know for sure. If I'm right, it helps account for the hopeful fact that the public appears robustly unpersuaded by the anti-testing media blitz Phelps thinks they're getting.