Showing posts with label Research. Show all posts
Showing posts with label Research. Show all posts

Sunday, April 5, 2015

Response to Complaint about What Works Clearinghouse

I have recently encountered some severe criticism leveled at reviews and reviewers from What Works Clearinghous  (see I am concerned about recommending this site to teachers as a resource for program evaluations. I'm wondering if you agree with the criticisms, and if yes, where you would recommend teachers go for evidence-based program reviews. I know that NELP and NRP reports are possibilities but are also static documents that do not get updated frequently with new findings, so some of the information really isn't current. Perhaps the Florida Center for Reading Research is an alternative? Do you have others than you would recommend?

I don’t agree with these criticisms and believe What Works Clearinghouse (WWC) has a valuable role to play in offering guidance to educators. I often recommend it to teachers and will continue to do so. It is the best source for this kind of information.

WWC is operated by the U.S. Department of Education. It reviews research claims about commercial programs and products in education. WWC serves as a kind of Good Housekeeping seal of approval. It is helpful because it takes conflict of interest out of the equation. WWC and its reviewers have no financial interest in whether a research claim is upheld or not.

I am an advisor to the WWC. Basically, that means I’m available, on a case-by-case basis, to help their review teams when questions come up about reading instruction or assessment. Such inquiries arise 2-3 times per year. I don’t think my modest involvement in WWC taints my opinion, but the whole point of WWC is to reduce the commercial influence on the interpretation of research findings, so it would be dishonorable for me not be open about my involvement.  

I wish the “studies” and “reports” you referred me to were as disinterested. 
The DI organization has long been chagrined that the WWC reviews of DI products and programs haven’t been more positive. That the authors of these reports have a rooting interest in the results should be noted.

Different from the disinterested reviews of the Clearinghouse which follow a consistent rule-based set of review procedures developed openly by a team of outstanding scientists, these reports are biased, probably because they are aimed at trying to poke a finger in the eye of the reviewers who were unwilling to endorse their programs. That’s why there is so much non-parallel analysis, questionable assumptions, biased language, etc.

For example, one of the reports indicates how many complaints have been sent to the WWC (62 over approximately 7 years of reviewing). This sounds like a lot, but what is the appropriate denominator… is it 62 complaints out of X reviews? Or 62 complaints about X decisions included in each of the X reviews? Baseball umpires make mistakes, too; but we evaluate them not on the number of mistakes, but the proportion of mistakes to decisions. (I recommend WWC reviews, in part, because they will re-review the studies and revise as necessary when there are complaints).

Or, another example: These reports include a table citing the “reasons for requesting a quality review of WWC findings,” which lists the numbers and percentage of times that complaints have focused on particular kinds of problems (e.g., misinterpretation of study findings, inclusion/exclusion of studies. But there is no comparable table showing the disposition of these complaints. I wonder why not? (Apparently, one learns in another portion of the report, that there were 146 specific complaints, 37 of which led to some kind of revision—often minor changes in a review for the sake of clarity; that doesn’t sound so terrible to me.)

The biggest complaint leveled here is that some studies should not have been included as evidence since they were studies of incomplete or poor implementations of a program.

The problem with that complaint is that issues of implementation quality only arise when a report doesn’t support a program’s effectiveness. There is no standard for determining how well or how completely a program is implemented, so for those with an axe to grind, any time their program works it had to be well implemented and when it doesn’t it wasn’t.

Schoolchildren need to be protected from such scary and self-interested logic.

Wednesday, January 15, 2014

Is There Research on that Reading Intervention?

I am a reading specialist working in an urban school district with struggling readers in K-5.  Do you have any suggestions on intervention programs that you find the most beneficial to students?  Currently, we are using LLI (Fountas and Pinnell), Sonday, Read Naturally and Soar to Success, at the interventionist's discretion. Is there any research supporting or refuting these programs?  Is there another program that you find more effective?  We also use Fast Forward and Lexia as computer-based interventions.  What does the research say about these tools? 

          The best place to get this kind of information is the What Works Clearinghouse (WWC). This is a kind of Consumer Reports for educators that will tell you if commercial products have been studied and how they did. The benefit to you is that all the information is in one place, it is being provided by the U.S. Department of Education so it won’t be biased towards some company, and they vet the research studies to make sure the information is sound.

Some things to be aware of when you seek this information:

          Don’t read too much into the fact that there is no evidence on a program.
           This happens a lot. Instructional programs aren’t like drugs; no one is required to prove that they work before they can be sold. While some companies do commission studies of their products, most do not. The key thing to remember is that a lack of research on a product does not mean that product doesn’t work. In such cases, I usually look to see if a product is as thorough or demanding as those that do have evidence.

Don’t overestimate programs that do have direct research support.
     Programs do not have automatic effects. A positive result tells you that this program can work under some conditions and with some students. It means that in those circumstances this program did better than… whatever the control or comparison group did. It is good to know that someone was able to get a positive result with such a program (that should help teacher confidence), but often a program that works may not work in your circumstances or with your teachers or with your students. Just because something worked, that doesn’t mean that you could make it work.

A basic ethical obligation of a researcher is to report the results of their studies, even when the studies don’t come out the way they wanted. Commercial companies don’t have this same obligation. What that means is that if a company commissions a study and it gets a positive result, they will allow it to be released; but that isn’t ou nenecessarily true when the results don’t show that their product worked. That means available research on a particular program or product may be overestimating the impact. (That’s one of the reasons that I like that WWC is so strong on the evidence: they can’t know about studies that got lost in a file drawer, but they can certainly make sure the available studies meet the highest evaluation standards).

Pay attention to the control group.
     In medicine, there are standards of care. Typically, a new treatment is compared with the standard of care so that you know that if it “worked” it would be better than what you are already doing. In education, we have no shared standards of instruction, so you need to pay attention to what the intervention did better than. It might have done better than what you are already delivering, and that would certainly encourage you to change programs, but it might be doing better than instruction that you, too, are already outperforming.

Monday, July 5, 2010

What Does It Mean that a Reading Program Works?

Recently, the What Works Clearinghouse (WWC) posted a report on multi-sensory programs for teaching children with reading disabilities. The report indicated that Orton-Gillingham, Wilson Reading, and other similar programs lacked convincing empirical evidence of their effectiveness. This has set off a lot of angry e-mailing this weekend from those who “know” these methods work.

Although I serve as a reading content expert for WWC, I had nothing to do with this report. They only involve me in those aspects of their work when they have a substantive question about reading instruction or assessment, and on this report, I guess there were no such questions for me.

In any event, I strongly support WWC, though at times it has made choices that I have disagreed with. In my experience, it is rigorously, carefully, and fairly adjudicated and its reviewers make a serious effort to find the research and to evaluate it in a reliable manner.

I know that it is upsetting to not get good news, but what does it actually mean that WWC says there is no evidence supporting these programs? First, and most important, it does not mean that those programs do not work or cannot work or that you should not use them. “No evidence” means no evidence and nothing more. “No evidence,” one way or the other. You may be able to make such a program work, or you might not, but no one has yet studied it rigorously. (There are many more programs that lack such evidence than there are ones that have it, and let’s face it, plenty of kids learn to read from such programs.) Don’t read too much into a lack of evidence. That is more about the standards we have as a field and how much we are currently willing to invest in programs.

But what about when the programs that WWC says something does work? I think too often teachers and principals think that is the end of the story. But it is not. When WWC says something works, you have some other questions to answer.

What does it mean to “work”?
So, when WWC says that there is evidence that a program worked it means that someone conducted at least one well-designed rigorous study showing that a program worked better than… well, better than something – or nothing. There is not a convention dictating what the control group is to be doing in such studies, so it is well worth taking a look at what they were up to so that you can know what the program was better than. (In some phonics studies, for example, control group children are in a conventional reading program without phonics. In other cases, the control group kids are receiving no reading teaching at all. The effects are generally larger when the comparison gets no treatment than when it is some other version of reading instruction.)

If WWC says the program worked and indicates an effect size, you might figure that the programs with the biggest effect sizes are the ones to buy. That would generally be true if the control conditions were standardized in some way, but that is not the case. Bigger effects are likely to be noted when the comparison is with nothing, than when it is put up against some other effective way of teaching reading.

To know what it means to have worked, it is also important to know what the outcomes were. WWC is very good about laying out what the specific measures of reading were that the students actually improved upon in the studies, and that is something that you want to know. If you are trying to improve kids’ reading comprehension, buying a program proven to improve only decoding may not be a wise choice, despite the WWC imprimatur.

Who did it work for?
Okay, so now, you know this program worked and you know what it means to have worked, so the issue is, with whom did it work? Many programs are sold on the basis of studies that didn’t treat students like the ones who you teach. Were the kids rich or poor, black or white, English learners or native English speakers, preschool or primary, disabled or abled? If a program works with one group, it might not work as well with another.

When did it work?
Finally, under what conditions did it work? Was this instruction given during the school day or afterschool? Was it supplementary to regular classroom instruction or did it take the place of such instruction? How much of it was needed to make a difference? Who delivered the teaching—regular teachers or the program creators? All of these questions get at whether or not you will be able to make the program work in your school, under the circumstances when you will be using the program. I always think of the district that purchased Read 180 on the basis of its evaluation studies; the program, as studied, delivers 90 minutes a day of instruction. The district purchasing it planned to use it for 45 minutes per day instead. I bet it doesn’t work!

By all means look at the research evidence on programs in which you are interested. If there is no evidence, it is definitely a strike against the program, but it should not be a fatal strike given that so few programs have such evidence at this point in history. And if there is positive evidence, then you need to look further to be sure it works at what you are trying to do, with the kinds of kids you are teaching, and under circumstances that could be replicated in your school.

Saturday, June 26, 2010

On the Misuse of Research Evidence in Reading

I remember the first time I was asked to testify before Congress. I was so full of myself. Wow, what a big shot, getting to speak truth to power. But then, once I had done it, it was clear to me that there was nothing impressive about it. The Senator who had invited me asked me to be there because I had done some research that supported a bill he was proposing. I wasn’t really invited to help Congress to understand the issue, but to secure support for his bill (which of course had many provisions with no basis in any research). If my research results were different, he would have gone forward with his bill, but without my testimony.

Over the years, that has gnawed at me every time I get calls from policymakers and media, not asking me for information on a particular topic, but for information with a particular slant. Do you know of any research againstT Reading Recovery? Do you know of any research the supports Reading?

Using evidence in that way might be the way to win an argument (and to get your way), but it is not the approach that we should be taking if we want policies and practices that are really likely to work. Some policy people and reporters do understand that and they are getting wiser in their use of education studies.

But, just this week, on an educational research listserv that I lurk on, there was a question, not from the public, but from a researcher, looking for the research that supported a particular practice. A teacher had come to her because the principal was changing some school literacy policies. The teacher liked things the way they were and so came looking for help. The researcher apparently thought the current policies were great, so she was seeking evidence that she was right (she sent out for calls supporting particular practices in literacy education). That is bizarre. No wonder people get skeptical about how we operate.

We should be looking at the evidence and trying to make a determination of what it means rather than seeking data that support what we already want to do. And, when such evidence does not exist, we should be very honest about our inability to provide data on a particular question (no matter what our point of view). This idea of cherry-picking evidence to support a position is a misuse of evidence and it is ethically shaky. It certainly does not move us forward as a community that is trying to achieve higher literacy levels in our society.

Monday, August 24, 2009

Applying Research to the Teaching of Reading: Here is a Brief Update on the National Reading Panel Findings

Recently, I was asked to give a talk to Expanding the Reach schools in Arizona. Expanding the Reach (ETR) has been a federal effort to help schools to take Reading First style actions without receiving all of the Reading First style support and regulations.

I was to talk to them about the reading research as reviewed by the National Reading Panel (NRP). There is a problem with doing that, however. The NRP completed its work in 2000, and there have been two major federal panels since that time, the National Early Literacy Panel (that looked at preschool and kindergarten literacy) and the National Literacy Panel for Language Minority Children and Youth (that looked at second-language literacy). There have also been a plethora of federal research reports and other research, such as the Reading First impact study. We want teachers to follow the research, but not just the research from a decade ago.

What I did is gave a fairly conventional talk in which I laid out the research findings in the five instructional areas where NRP had findings, but for each of them, I have added a what's new section. So, for example, I shared the studies from NRP that show that phonemic awareness instruction matters, but I then noted that NELP had found a payoff for phonological awareness for younger kids (that is, that it is important to start out with larger sound units than phonemes to get the ball rolling). Or, I explained the NRP phonics findings, but supplemented those with the findings showing that English learners sometimes bring adequate phonics to English (e.g., if they can already read Spanish), and that phonics instruction has a smaller effect size with second language kids (meaning that just raising their phonics won't have as big a payoff for these kids). I showed the comprehension findings from NRP, but pointed out that Reading First had little impact on either the teaching of comprehension or comprehension achievement.

I thought it was a useful way to go, and the audience responded positively, so here is a copy of the powerpoint for your use. It is a nifty summary of the NRP, with some useful updates.

Wednesday, July 22, 2009

Even Scarier than Wild Animals

It’s good to be back from Africa—I guess. Each day (and night), Cyndie and I had fascinating new experiences, sometimes frighteningly so: like the night we were awakened by a lion fight; or the hippo that chased us; or the time we mistakenly found ourselves inconveniently between a bull elephant and what he wanted to eat. Scary stuff.

Not that I’m home, I’ve been going to meetings, reading emails and the latest journals, and maybe I was safer with the cheetahs! Just in the past few days I’ve been hearing over and over the kind of anti-research rhetoric that was popular back when the National Reading Panel report came out.

The claim being made is that now that No Child Left Behind is over we can go back to making decisions based on any kind of evidence—the studies don’t have to be appropriate to the claims and there doesn’t need to be much evidence either. In other words, these folks want to set public policy on the basis of single case studies or determine how teachers should teach based on a single lesson observational study.

The remarkable positive thing during the past decade has been the requirement to say something worked that one has actually tried the something out with kids and shown that the kids actually benefited. Qualitative and correlational research studies are great, and they definitely can be rigorously designed, and they are definitely scientific (when done correctly)… what they can’t do with that kind of research is determine whether something works; whether it confers a benefit on children. If we go back to deciding whether teaching approaches work by looking at indirect or inappropriate research evidence, that really will be scary.

One more interesting development: a couple of weeks ago Education Week asked me to opine on the new literacy bill Congress is considering. I was positive, but said I didn’t think they should have dropped reading comprehension from what will be taught (the law calls instead for instruction in “meaning in context”.) The reporter followed up with a nameless Congressional aide who sniffed, “that is the latest terminology” (or something along those lines). That’s fascinating: I’m in schools all the time, I review a ton of textbooks for teachers and students, and I’m working on various teaching standards and research issues and no one let me in on the secret that reading comprehension was no longer the correct term. It is so frustrating to find that Congress found out about this sea change before me. I do wonder what states are going to require teachers to do to demonstrate that they are teaching “meaning in context.” I wish Ed Week had the well-informed staffer on the record, as I’m sure we could refer the queries to him.