Wednesday, January 15, 2014
Monday, July 5, 2010
Recently, the What Works Clearinghouse (WWC) posted a report on multi-sensory programs for teaching children with reading disabilities. The report indicated that Orton-Gillingham, Wilson Reading, and other similar programs lacked convincing empirical evidence of their effectiveness. This has set off a lot of angry e-mailing this weekend from those who “know” these methods work.
Although I serve as a reading content expert for WWC, I had nothing to do with this report. They only involve me in those aspects of their work when they have a substantive question about reading instruction or assessment, and on this report, I guess there were no such questions for me.
In any event, I strongly support WWC, though at times it has made choices that I have disagreed with. In my experience, it is rigorously, carefully, and fairly adjudicated and its reviewers make a serious effort to find the research and to evaluate it in a reliable manner.
I know that it is upsetting to not get good news, but what does it actually mean that WWC says there is no evidence supporting these programs? First, and most important, it does not mean that those programs do not work or cannot work or that you should not use them. “No evidence” means no evidence and nothing more. “No evidence,” one way or the other. You may be able to make such a program work, or you might not, but no one has yet studied it rigorously. (There are many more programs that lack such evidence than there are ones that have it, and let’s face it, plenty of kids learn to read from such programs.) Don’t read too much into a lack of evidence. That is more about the standards we have as a field and how much we are currently willing to invest in programs.
But what about when the programs that WWC says something does work? I think too often teachers and principals think that is the end of the story. But it is not. When WWC says something works, you have some other questions to answer.
What does it mean to “work”?
So, when WWC says that there is evidence that a program worked it means that someone conducted at least one well-designed rigorous study showing that a program worked better than… well, better than something – or nothing. There is not a convention dictating what the control group is to be doing in such studies, so it is well worth taking a look at what they were up to so that you can know what the program was better than. (In some phonics studies, for example, control group children are in a conventional reading program without phonics. In other cases, the control group kids are receiving no reading teaching at all. The effects are generally larger when the comparison gets no treatment than when it is some other version of reading instruction.)
If WWC says the program worked and indicates an effect size, you might figure that the programs with the biggest effect sizes are the ones to buy. That would generally be true if the control conditions were standardized in some way, but that is not the case. Bigger effects are likely to be noted when the comparison is with nothing, than when it is put up against some other effective way of teaching reading.
To know what it means to have worked, it is also important to know what the outcomes were. WWC is very good about laying out what the specific measures of reading were that the students actually improved upon in the studies, and that is something that you want to know. If you are trying to improve kids’ reading comprehension, buying a program proven to improve only decoding may not be a wise choice, despite the WWC imprimatur.
Who did it work for?
Okay, so now, you know this program worked and you know what it means to have worked, so the issue is, with whom did it work? Many programs are sold on the basis of studies that didn’t treat students like the ones who you teach. Were the kids rich or poor, black or white, English learners or native English speakers, preschool or primary, disabled or abled? If a program works with one group, it might not work as well with another.
When did it work?
Finally, under what conditions did it work? Was this instruction given during the school day or afterschool? Was it supplementary to regular classroom instruction or did it take the place of such instruction? How much of it was needed to make a difference? Who delivered the teaching—regular teachers or the program creators? All of these questions get at whether or not you will be able to make the program work in your school, under the circumstances when you will be using the program. I always think of the district that purchased Read 180 on the basis of its evaluation studies; the program, as studied, delivers 90 minutes a day of instruction. The district purchasing it planned to use it for 45 minutes per day instead. I bet it doesn’t work!
By all means look at the research evidence on programs in which you are interested. If there is no evidence, it is definitely a strike against the program, but it should not be a fatal strike given that so few programs have such evidence at this point in history. And if there is positive evidence, then you need to look further to be sure it works at what you are trying to do, with the kinds of kids you are teaching, and under circumstances that could be replicated in your school.
Saturday, June 26, 2010
I remember the first time I was asked to testify before Congress. I was so full of myself. Wow, what a big shot, getting to speak truth to power. But then, once I had done it, it was clear to me that there was nothing impressive about it. The Senator who had invited me asked me to be there because I had done some research that supported a bill he was proposing. I wasn’t really invited to help Congress to understand the issue, but to secure support for his bill (which of course had many provisions with no basis in any research). If my research results were different, he would have gone forward with his bill, but without my testimony.
Over the years, that has gnawed at me every time I get calls from policymakers and media, not asking me for information on a particular topic, but for information with a particular slant. Do you know of any research againstT Reading Recovery? Do you know of any research the supports Reading?
Using evidence in that way might be the way to win an argument (and to get your way), but it is not the approach that we should be taking if we want policies and practices that are really likely to work. Some policy people and reporters do understand that and they are getting wiser in their use of education studies.
But, just this week, on an educational research listserv that I lurk on, there was a question, not from the public, but from a researcher, looking for the research that supported a particular practice. A teacher had come to her because the principal was changing some school literacy policies. The teacher liked things the way they were and so came looking for help. The researcher apparently thought the current policies were great, so she was seeking evidence that she was right (she sent out for calls supporting particular practices in literacy education). That is bizarre. No wonder people get skeptical about how we operate.
We should be looking at the evidence and trying to make a determination of what it means rather than seeking data that support what we already want to do. And, when such evidence does not exist, we should be very honest about our inability to provide data on a particular question (no matter what our point of view). This idea of cherry-picking evidence to support a position is a misuse of evidence and it is ethically shaky. It certainly does not move us forward as a community that is trying to achieve higher literacy levels in our society.
Monday, August 24, 2009
Applying Research to the Teaching of Reading: Here is a Brief Update on the National Reading Panel Findings
I was to talk to them about the reading research as reviewed by the National Reading Panel (NRP). There is a problem with doing that, however. The NRP completed its work in 2000, and there have been two major federal panels since that time, the National Early Literacy Panel (that looked at preschool and kindergarten literacy) and the National Literacy Panel for Language Minority Children and Youth (that looked at second-language literacy). There have also been a plethora of federal research reports and other research, such as the Reading First impact study. We want teachers to follow the research, but not just the research from a decade ago.
What I did is gave a fairly conventional talk in which I laid out the research findings in the five instructional areas where NRP had findings, but for each of them, I have added a what's new section. So, for example, I shared the studies from NRP that show that phonemic awareness instruction matters, but I then noted that NELP had found a payoff for phonological awareness for younger kids (that is, that it is important to start out with larger sound units than phonemes to get the ball rolling). Or, I explained the NRP phonics findings, but supplemented those with the findings showing that English learners sometimes bring adequate phonics to English (e.g., if they can already read Spanish), and that phonics instruction has a smaller effect size with second language kids (meaning that just raising their phonics won't have as big a payoff for these kids). I showed the comprehension findings from NRP, but pointed out that Reading First had little impact on either the teaching of comprehension or comprehension achievement.
I thought it was a useful way to go, and the audience responded positively, so here is a copy of the powerpoint for your use. It is a nifty summary of the NRP, with some useful updates.
Wednesday, July 22, 2009
It’s good to be back from Africa—I guess. Each day (and night), Cyndie and I had fascinating new experiences, sometimes frighteningly so: like the night we were awakened by a lion fight; or the hippo that chased us; or the time we mistakenly found ourselves inconveniently between a bull elephant and what he wanted to eat. Scary stuff.
Not that I’m home, I’ve been going to meetings, reading emails and the latest journals, and maybe I was safer with the cheetahs! Just in the past few days I’ve been hearing over and over the kind of anti-research rhetoric that was popular back when the National Reading Panel report came out.
The claim being made is that now that No Child Left Behind is over we can go back to making decisions based on any kind of evidence—the studies don’t have to be appropriate to the claims and there doesn’t need to be much evidence either. In other words, these folks want to set public policy on the basis of single case studies or determine how teachers should teach based on a single lesson observational study.
The remarkable positive thing during the past decade has been the requirement to say something worked that one has actually tried the something out with kids and shown that the kids actually benefited. Qualitative and correlational research studies are great, and they definitely can be rigorously designed, and they are definitely scientific (when done correctly)… what they can’t do with that kind of research is determine whether something works; whether it confers a benefit on children. If we go back to deciding whether teaching approaches work by looking at indirect or inappropriate research evidence, that really will be scary.
One more interesting development: a couple of weeks ago Education Week asked me to opine on the new literacy bill Congress is considering. I was positive, but said I didn’t think they should have dropped reading comprehension from what will be taught (the law calls instead for instruction in “meaning in context”.) The reporter followed up with a nameless Congressional aide who sniffed, “that is the latest terminology” (or something along those lines). That’s fascinating: I’m in schools all the time, I review a ton of textbooks for teachers and students, and I’m working on various teaching standards and research issues and no one let me in on the secret that reading comprehension was no longer the correct term. It is so frustrating to find that Congress found out about this sea change before me. I do wonder what states are going to require teachers to do to demonstrate that they are teaching “meaning in context.” I wish Ed Week had the well-informed staffer on the record, as I’m sure we could refer the queries to him.