Showing posts with label Research. Show all posts
Showing posts with label Research. Show all posts

Thursday, July 2, 2015

Why Research-Based Reading Programs Alone Are Not Enough


Every teacher has experienced this. While the majority of the class is thriving with your carefully planned, research supported instructional methods, there is often one kid that is significantly less successful. We work with them individually in class, help them after school, sometimes change things up to see what will work, bring them to the attention of the RtI team that is also using the research supported instructional methods. But what if the methods research support for the majority of kids don't work for this kid?

Several months ago I read an article in Discover magazine called "Singled Out" by Maggie Koerth-Baker. Regarding medicine rather than education, the article is about using N of 1 experiments to find out whether an individual patient reacts well to a particular research backed treatment.

"But even the gold standard isn't perfect. The controlled clinical trial is really about averages, and averages don't necessarily tell you what will happen to an individual."

Ever since I read the article, I've been wondering what an N of 1 experiment would look like in the classroom. This would be much easier to implement in the controlled numbers of a special education classroom, but we do so much differentiation in the regular classroom now, I'd like to find a way to objectively tell if what we do for individuals is effective in the short term, rather than waiting for the high stakes testing that the whole class takes. Formative assessment is helpful, but I suspect we need something more finely tuned to tease out what made the difference. We gather tons of data to report at RtI meetings, but at least at my school, it's things like sight word percentages, reading levels, fluency samples, not clear indicators of say, whether a child As a researcher, how would you set up an N of 1 experiment in an elementary classroom?

My response:
This letter points out an important fact about experimental research and its offshoots (e.g., quasi-experiments, regression discontinuity designs): when we say a treatment was effective that doesn’t mean everyone who got the special whiz-bang teaching approach did better than everyone who didn’t. It just means one group, on average, did better than the other group, on average.

For example, Reading First was federal program that invested heavily in trying to use research-based approaches to improve beginning reading achievement in Title I schools. At the end of the study, the RF schools weren't doing much better than the control schools overall. But that doesn't mean there weren’t individual schools that used the extra funding well to improve their students’ achievement, just that there weren’t enough of those schools to make a group difference.

The same happens when we test the effectiveness of phonics instruction or comprehension strategies. A study may find that the average score for the treatment group was significantly higher than that obtained by the control group, but there would be kids in the control group who would outperform those who got the treatment, and students in the successful treatment who weren’t themselves so successful.

That means that even if you were to implement a particular procedure perfectly and with all of the intensity of the original effort (which is rarely the case), you'd still have students who were not very successful with the research-based training.

Awhile back, Atul Gawande, wrote in the New Yorker about the varied results obtained in medicine with research-based practices (“The Bell-Curve”). Dr. Gawande noted that particular hospitals, although they followed the same research-based protocols, were so scrupulous and vigorous in their application of those methods that they obtained better results.

For example, in the treatment of cystic fibrosis, it's a problem when a patient’s breathing capacity falls below level. If the lung capacity reaches that benchmark, standard practice would be to hospitalize the patient to try to regain breathing capacity. However, in the particularly effective hospitals, doctors didn’t wait for the problem to become manifest. As soon as things started going wrong for a patient, breathing capacity started to decline, they intervened.

It is less about formal testing (since our measures usually lack the reliability of those used in medicine) or about studies with Ns of 1, than about thorough and intensive implementation of research-based practices and careful and ongoing monitoring of student performance within instruction.

Many educators and policymakers seem to think that once research-based programs are selected, then we no longer need to worry about learning. That neglects the fact that our studies tell us less about what works, than they do about what may work under some conditions. Our studies tell us about practices that have been used successfully, but people are so complex that you can’t guarantee such programs will always work that way. It is a good idea to use practices that have been successful--for someone--in the past, but such practices do not have automatically positive outcomes. In the original studies, teachers would have worked hard to try to implement successfully; later, teachers may be misled into thinking that if they just take kids through the program the same levels of success will automatically be obtained. 

Similarly, in our efforts to make sure that we don't lose some kids, we may impose testing regimes aimed at monitoring success, such as DIBELing kids several times a year… but such instruments are inadequate for such intensive monitoring and can end up being misleading.

I’d suggest, instead, that teachers use those formal monitors less frequently—a couple or three times a year, but to observe the success of their daily lessons more carefully. For example, a teacher is having students practice hearing differences in the endings of words. Many students are able to implement the skill successfully by the end of the lesson, but some are not. If that’s the case, supplement that lesson with more practice rather than just going onto the next prescribed lesson (or do this simultaneous to the continued progress through the program). If the lesson was supposed to make it possible for kids to hear particular sounds, then do whatever you can to enable them to hear those sounds.

To monitor ongoing success this carefully, the teacher does have to plan lessons that allow students many opportunities to demonstrate whether or not they could implement the skill. The teacher also has to have a sense of what success may look like (e.g., the students don’t know these 6 words well enough if they can’t name them in 10 seconds or less; the students can’t spell these particular sounds well enough if they can’t get 8 out of 10 correct; the student isn’t blending well enough if they… etc.).

If a program of instruction can be successful, and you make sure that students do well with the program—actually learning what is being presented by the program—then you should have fewer kids failing to progress.

Sunday, April 5, 2015

Response to Complaint about What Works Clearinghouse

I have recently encountered some severe criticism leveled at reviews and reviewers from What Works Clearinghous  (see I am concerned about recommending this site to teachers as a resource for program evaluations. I'm wondering if you agree with the criticisms, and if yes, where you would recommend teachers go for evidence-based program reviews. I know that NELP and NRP reports are possibilities but are also static documents that do not get updated frequently with new findings, so some of the information really isn't current. Perhaps the Florida Center for Reading Research is an alternative? Do you have others than you would recommend?

I don’t agree with these criticisms and believe What Works Clearinghouse (WWC) has a valuable role to play in offering guidance to educators. I often recommend it to teachers and will continue to do so. It is the best source for this kind of information.

WWC is operated by the U.S. Department of Education. It reviews research claims about commercial programs and products in education. WWC serves as a kind of Good Housekeeping seal of approval. It is helpful because it takes conflict of interest out of the equation. WWC and its reviewers have no financial interest in whether a research claim is upheld or not.

I am an advisor to the WWC. Basically, that means I’m available, on a case-by-case basis, to help their review teams when questions come up about reading instruction or assessment. Such inquiries arise 2-3 times per year. I don’t think my modest involvement in WWC taints my opinion, but the whole point of WWC is to reduce the commercial influence on the interpretation of research findings, so it would be dishonorable for me not be open about my involvement.  

I wish the “studies” and “reports” you referred me to were as disinterested. 
The DI organization has long been chagrined that the WWC reviews of DI products and programs haven’t been more positive. That the authors of these reports have a rooting interest in the results should be noted.

Different from the disinterested reviews of the Clearinghouse which follow a consistent rule-based set of review procedures developed openly by a team of outstanding scientists, these reports are biased, probably because they are aimed at trying to poke a finger in the eye of the reviewers who were unwilling to endorse their programs. That’s why there is so much non-parallel analysis, questionable assumptions, biased language, etc.

For example, one of the reports indicates how many complaints have been sent to the WWC (62 over approximately 7 years of reviewing). This sounds like a lot, but what is the appropriate denominator… is it 62 complaints out of X reviews? Or 62 complaints about X decisions included in each of the X reviews? Baseball umpires make mistakes, too; but we evaluate them not on the number of mistakes, but the proportion of mistakes to decisions. (I recommend WWC reviews, in part, because they will re-review the studies and revise as necessary when there are complaints).

Or, another example: These reports include a table citing the “reasons for requesting a quality review of WWC findings,” which lists the numbers and percentage of times that complaints have focused on particular kinds of problems (e.g., misinterpretation of study findings, inclusion/exclusion of studies. But there is no comparable table showing the disposition of these complaints. I wonder why not? (Apparently, one learns in another portion of the report, that there were 146 specific complaints, 37 of which led to some kind of revision—often minor changes in a review for the sake of clarity; that doesn’t sound so terrible to me.)

The biggest complaint leveled here is that some studies should not have been included as evidence since they were studies of incomplete or poor implementations of a program.

The problem with that complaint is that issues of implementation quality only arise when a report doesn’t support a program’s effectiveness. There is no standard for determining how well or how completely a program is implemented, so for those with an axe to grind, any time their program works it had to be well implemented and when it doesn’t it wasn’t.

Schoolchildren need to be protected from such scary and self-interested logic.

Wednesday, January 15, 2014

Is There Research on that Reading Intervention?

I am a reading specialist working in an urban school district with struggling readers in K-5.  Do you have any suggestions on intervention programs that you find the most beneficial to students?  Currently, we are using LLI (Fountas and Pinnell), Sonday, Read Naturally and Soar to Success, at the interventionist's discretion. Is there any research supporting or refuting these programs?  Is there another program that you find more effective?  We also use Fast Forward and Lexia as computer-based interventions.  What does the research say about these tools? 

          The best place to get this kind of information is the What Works Clearinghouse (WWC). This is a kind of Consumer Reports for educators that will tell you if commercial products have been studied and how they did. The benefit to you is that all the information is in one place, it is being provided by the U.S. Department of Education so it won’t be biased towards some company, and they vet the research studies to make sure the information is sound.

Some things to be aware of when you seek this information:

          Don’t read too much into the fact that there is no evidence on a program.
           This happens a lot. Instructional programs aren’t like drugs; no one is required to prove that they work before they can be sold. While some companies do commission studies of their products, most do not. The key thing to remember is that a lack of research on a product does not mean that product doesn’t work. In such cases, I usually look to see if a product is as thorough or demanding as those that do have evidence.

Don’t overestimate programs that do have direct research support.
     Programs do not have automatic effects. A positive result tells you that this program can work under some conditions and with some students. It means that in those circumstances this program did better than… whatever the control or comparison group did. It is good to know that someone was able to get a positive result with such a program (that should help teacher confidence), but often a program that works may not work in your circumstances or with your teachers or with your students. Just because something worked, that doesn’t mean that you could make it work.

A basic ethical obligation of a researcher is to report the results of their studies, even when the studies don’t come out the way they wanted. Commercial companies don’t have this same obligation. What that means is that if a company commissions a study and it gets a positive result, they will allow it to be released; but that isn’t ou nenecessarily true when the results don’t show that their product worked. That means available research on a particular program or product may be overestimating the impact. (That’s one of the reasons that I like that WWC is so strong on the evidence: they can’t know about studies that got lost in a file drawer, but they can certainly make sure the available studies meet the highest evaluation standards).

Pay attention to the control group.
     In medicine, there are standards of care. Typically, a new treatment is compared with the standard of care so that you know that if it “worked” it would be better than what you are already doing. In education, we have no shared standards of instruction, so you need to pay attention to what the intervention did better than. It might have done better than what you are already delivering, and that would certainly encourage you to change programs, but it might be doing better than instruction that you, too, are already outperforming.