I hate it when someone who doesn’t know much about research says, “You can prove anything with statistics.” I hate it because it’s true, you really can lie with statistics. If you want to. Research is not replete with examples of intentionally misleading statistics, however. Researchers sometimes “fool themselves” by not looking hard enough at their data to uncover inconsistencies and mistakes (a problem at least partially addressed by peer review).
However, pundits are under no such obligation. When they use statistics it can be pretty biased (maybe intentionally so). I just came across a fascinating example. Richard Rothstein (formerly a NY Times columnist, now a Research Associate with the Economic Policy Institute) published a piece that I’ll link at the end of this blog). In it, he challenges Bill Gates’s contention that NAEP scores have been relatively flat for 40 years. Rothstein claims that, in fact, kids are doing much better according to NAEP, when you break out the scores by race.
It is an interesting disagreement because if you look at the overall NAEP scores they are flat, and if you look at the scores by race there are gains in reading and math. I went back and examined the data he used for these claims and found they clearly supported his current position… but not his past positions.
For example, previously he had savaged President George W. Bush’s education policies, calling them “incoherent, unworkable, and doomed.” He went on to write, “Even with inordinate attention to math and reading, it is practically and conceptually ludicrous to expect all students to be proficient at challenging levels.” I took a look at the gains that Rothstein touted for African-American’s and found big boosts in reading performance during the years of the “unworkable” and “doomed” policy that Rothstein reviled. So is Rothstein some kind of a racist? Why else would he oppose policies that he now claims helped African-Americans do so much better? Rothstein is clearly NOT a racist as demonstrated by his long history of sensitive and thoughtful writing on racial issues.
Of course, Rothstein has written in the past about the dangers of making decisions on the basis of racial or ethnic slices of data due to their inherent, relative unreliability (the total sample will always provide a more stable accounting than will portions of the total sample). And, yet, here is Rothstein, on the basis of such slices of data, now claiming kids made great learning gains under NCLB.
To me, this has the appearance of someone espousing a policy while grasping to find supportive data, even if those data happen to be out of sync with his past arguments. His latest statistical display fails to account for the demographic shifts in the American populace; the small but steady rise in African-Americans as a percentage of the total, and the much larger decline in the white population percentage during the same period (this decline largely due to the Latino and Asian immigrations). What this means is, that while various subgroups may be improving due to the diligence of our schools (and apparently to the no-good policies that support them), the numbers are such that our schools still need to do better.
I’ve written here many times that the schools are not doing a bad job (in fact, U.S. schools are performing as well as they have at any time in our long history — that’s what flat achievement means). But having 1970 literacy levels in 2011 is still a bad deal for the kids because of the shifting economy (something else that Rothstein disagrees with). He and I are sympatico with the idea that we shouldn’t be kicking teachers in the pants (the statistics don't support that), but we do have to find ways to get them to do an even better job than in the past.
Mr. Rothstein should decide whether improvements in educational outcomes for racial groups are a sufficient national goal, in which case NCLB looks like a winner (and Rothstein ought to admit it), or whether we actually seek better educational outcomes for a larger proportion of our total population (in which case, Bill Gates and Secretary Duncan are onto something). What we shouldn’t do is use the total NAEP scores to attack Bush, and the subgroup scores to attack Gates—that looks like bias, rather than a real attempt to solve a very real problem.
Rothstein is correct that accountability approaches won’t improve achievement, and he has been right to criticize the Bushies and Obamaites for too much reliance on such testing (both ways of looking at NAEP gains would lead you to that conclusion). However, he has not been discerning enough to support the wisdom of the big investments in professional development and curriculum support under NCLB (portions of the law that have now lapsed), nor the current efforts to build a common core curriculum or to make teacher education more data-based (both of which can bear fruit).