Bad news outweighs the good in math testing

November 20, 2004

Tom Loveless used to be a sixth-grade math teacher in California, and now he's a researcher at the Brookings Institution. And as author of this year's report on American education from Brookings' Brown Center, he has both good news and bad news about the "nation's report card," the National Assessment of Educational Progress.

"The good news is that NAEP scores have risen dramatically in mathematics over the past decade," he said. The bad news is that the mathematics being tested is way below grade level, and more students than not still can't do it.

The NAEP tests are administered nationally to a large sample of fourth- and eighth-grade students, approximately 150,000 in a given grade and subject. Math questions are classified into five "strands" of mathematics: algebra, data analysis, geometry, measurement and number sense. Arithmetic is not separately classified, though it's obviously a part of the other strands. Items are also characterized along the three dimensions of concepts, procedures and problem-solving.

Over time, more than 500 questions from the math tests have been released, and they are available on the nces .ed.gov Web site. Loveless and his research associates coded each item "by the grade level in which the arithmetic skills and knowledge required by the item are taught."

Grade levels vary, of course, depending on textbooks, but not widely, with addition and subtraction of whole numbers typically taught in first or second grade, multiplication and division introduced in second or third grade, fractions appearing in third or fourth grade and decimals and percentages in fifth or sixth grade.

For problem-solving items that required arithmetic (39 of them for the fourth grade) the arithmetic needed was, typically, taught in the early part of third grade - the mean grade level was 3.1. Even so, no more than half the students could answer, on average, the items from any grade level - including first grade.

The situation is even more dismaying in eighth grade. The items are only slightly more difficult - a mean grade level of 3.4. By this time, a bare majority, 54 percent, get the first-grade questions right. But there's no majority for correct answers at any other grade level, and for the seventh-grade work there are only a bit more than a quarter of correct answers.

Bad enough news for you? Loveless is just getting started. The other striking thing about the questions is that a large majority of them, approximately 70 percent use only whole numbers.

"If students do not possess the tools to solve problems involving fractions, decimals, and percents - if students do not grasp forms of numbers other than whole numbers - then the only problems they will ever be able to solve will be mathematically trivial," Loveless writes.

Actually I know quite a few grownups whose grasp of decimals and percentages is more than a little shaky. Sometimes, unfortunately, the results end up in the paper. I had a colleague once - not here - who called a California school-bond election wrong, in a front-page headline, because she didn't know how to figure whether it had reached the necessary two-thirds vote.

And speaking of calculators, students are usually allowed to use them on any items that contain anything other than whole numbers.

Math scores for fourth-graders on the most recent test were about halfway between the scores of fourth- and eighth-graders in 1990 (reading has stayed almost flat during the same time). Loveless has several recommendations that would help to demonstrate whether that apparent improvement in NAEP scores is real.

First, he says, raise the difficulty of the arithmetic used in problem-solving items to something more nearly appropriate for the grade level being tested, including more fractions, decimals and percentages on the eighth-grade test. Second, track arithmetic separately. A comfortable fluency in arithmetic is essential for success in algebra, which in turn is the foundation of everything else students will do in math. If you're flummoxed by having to divide 1/2 by 2/3, a calculator is not going to help you solve equations with other than whole-number coefficients. Third, eliminate the use of calculators in fourth grade and restrict it in eighth.

Loveless knows there are limitations on his study; he mentions, for instance, that there is no way to know whether the questions that have been released are of typical difficulty. He also points out that when tests are constructed, items that almost everybody gets right (or wrong) are usually discarded so the small number of correct responses may reflect test design rather than students' poor state of knowledge.

Still, he thinks that if the improvements were as large as they seem, it would be reflected in larger numbers of students taking algebra by eighth grade. It is up, but by nothing like the amount to be expected if eighth-graders now are comparable to ninth-graders a decade ago.

It looks as if the bad news is the right news. And if you're still wondering, the answer is 3/4.