Showing posts with label classroom observations. Show all posts
Showing posts with label classroom observations. Show all posts

Thursday, July 2, 2015

Why Research-Based Reading Programs Alone Are Not Enough


Every teacher has experienced this. While the majority of the class is thriving with your carefully planned, research supported instructional methods, there is often one kid that is significantly less successful. We work with them individually in class, help them after school, sometimes change things up to see what will work, bring them to the attention of the RtI team that is also using the research supported instructional methods. But what if the methods research support for the majority of kids don't work for this kid?

Several months ago I read an article in Discover magazine called "Singled Out" by Maggie Koerth-Baker. Regarding medicine rather than education, the article is about using N of 1 experiments to find out whether an individual patient reacts well to a particular research backed treatment.

"But even the gold standard isn't perfect. The controlled clinical trial is really about averages, and averages don't necessarily tell you what will happen to an individual."

Ever since I read the article, I've been wondering what an N of 1 experiment would look like in the classroom. This would be much easier to implement in the controlled numbers of a special education classroom, but we do so much differentiation in the regular classroom now, I'd like to find a way to objectively tell if what we do for individuals is effective in the short term, rather than waiting for the high stakes testing that the whole class takes. Formative assessment is helpful, but I suspect we need something more finely tuned to tease out what made the difference. We gather tons of data to report at RtI meetings, but at least at my school, it's things like sight word percentages, reading levels, fluency samples, not clear indicators of say, whether a child As a researcher, how would you set up an N of 1 experiment in an elementary classroom?

My response:
This letter points out an important fact about experimental research and its offshoots (e.g., quasi-experiments, regression discontinuity designs): when we say a treatment was effective that doesn’t mean everyone who got the special whiz-bang teaching approach did better than everyone who didn’t. It just means one group, on average, did better than the other group, on average.

For example, Reading First was federal program that invested heavily in trying to use research-based approaches to improve beginning reading achievement in Title I schools. At the end of the study, the RF schools weren't doing much better than the control schools overall. But that doesn't mean there weren’t individual schools that used the extra funding well to improve their students’ achievement, just that there weren’t enough of those schools to make a group difference.

The same happens when we test the effectiveness of phonics instruction or comprehension strategies. A study may find that the average score for the treatment group was significantly higher than that obtained by the control group, but there would be kids in the control group who would outperform those who got the treatment, and students in the successful treatment who weren’t themselves so successful.

That means that even if you were to implement a particular procedure perfectly and with all of the intensity of the original effort (which is rarely the case), you'd still have students who were not very successful with the research-based training.

Awhile back, Atul Gawande, wrote in the New Yorker about the varied results obtained in medicine with research-based practices (“The Bell-Curve”). Dr. Gawande noted that particular hospitals, although they followed the same research-based protocols, were so scrupulous and vigorous in their application of those methods that they obtained better results.

For example, in the treatment of cystic fibrosis, it's a problem when a patient’s breathing capacity falls below level. If the lung capacity reaches that benchmark, standard practice would be to hospitalize the patient to try to regain breathing capacity. However, in the particularly effective hospitals, doctors didn’t wait for the problem to become manifest. As soon as things started going wrong for a patient, breathing capacity started to decline, they intervened.

It is less about formal testing (since our measures usually lack the reliability of those used in medicine) or about studies with Ns of 1, than about thorough and intensive implementation of research-based practices and careful and ongoing monitoring of student performance within instruction.

Many educators and policymakers seem to think that once research-based programs are selected, then we no longer need to worry about learning. That neglects the fact that our studies tell us less about what works, than they do about what may work under some conditions. Our studies tell us about practices that have been used successfully, but people are so complex that you can’t guarantee such programs will always work that way. It is a good idea to use practices that have been successful--for someone--in the past, but such practices do not have automatically positive outcomes. In the original studies, teachers would have worked hard to try to implement successfully; later, teachers may be misled into thinking that if they just take kids through the program the same levels of success will automatically be obtained. 

Similarly, in our efforts to make sure that we don't lose some kids, we may impose testing regimes aimed at monitoring success, such as DIBELing kids several times a year… but such instruments are inadequate for such intensive monitoring and can end up being misleading.

I’d suggest, instead, that teachers use those formal monitors less frequently—a couple or three times a year, but to observe the success of their daily lessons more carefully. For example, a teacher is having students practice hearing differences in the endings of words. Many students are able to implement the skill successfully by the end of the lesson, but some are not. If that’s the case, supplement that lesson with more practice rather than just going onto the next prescribed lesson (or do this simultaneous to the continued progress through the program). If the lesson was supposed to make it possible for kids to hear particular sounds, then do whatever you can to enable them to hear those sounds.

To monitor ongoing success this carefully, the teacher does have to plan lessons that allow students many opportunities to demonstrate whether or not they could implement the skill. The teacher also has to have a sense of what success may look like (e.g., the students don’t know these 6 words well enough if they can’t name them in 10 seconds or less; the students can’t spell these particular sounds well enough if they can’t get 8 out of 10 correct; the student isn’t blending well enough if they… etc.).

If a program of instruction can be successful, and you make sure that students do well with the program—actually learning what is being presented by the program—then you should have fewer kids failing to progress.

Wednesday, November 11, 2009

Why Reading Coaches Often Emphasize the Wrong Stuff

If you type “reading comprehension observation” into Google, you get 462,000 hits. Not all of those pages will be instruments for observing how teachers and classrooms support reading comprehension development. But a lot of them are.

Some of these instruments are famous, like the CIERA one that Barbara Taylor and P. David Pearson made up awhile back. Others are the brainchildren of small companies or school districts. And t0hey all are supposed to be useful checklists for determining whether a classroom has what it takes.

Studies on such instruments suggests that they work—too some extent. What I mean is that many of the checklists are workable; a coach or other observer can see whether particular actions are taking place, and it may have reliability. So far, so good.

Some of these forms even have a correlation with kids’ learning. But having a relationship and having a strong relationship is the difference between talking to the counter man at the local 7-11 and talking to my wife. The correlations that have emanated from these observation forms tend to be tiny, and experts like Steve Roudenbush now believe there is simply too much variation in the day-to-day activities of classes to allow such observations to reveal much that is worthwhile with regard to the relationship of teaching and learning.

Reliability problems aside, most of the questions in these instruments get at the wrong things. They are simply too superficial to allow anything important to be determined.

A simple example: many forms ask the observer to determine if there is vocabulary instruction. That’s easy enough to figure out, and observers should be able to check the right boxes. However, what do those observations tell us? Well, that almost all teachers deliver some vocabulary instruction, so we’ll check yes to such observation items, but this variable, even when added together with, doesn’t tell us what we want to know.

I guess what I’m saying is that good teachers and poor teachers aren’t that different. In fact, it might be fair to say that many bad teachers look exactly like good teachers, but that the resemblance is superficial. You and I both might get checked off that we are teaching vocabulary, but which of us has selected words worth learning? Which is providing clear explanations of the word meanings? And which is being thorough enough and interactive enough and focused on meaning enough, to help the students to learn the words.

Checklists for observing reading lessons, for the most part, do not require qualitative judgments of how well teachers are keeping kids focused on meaning or how coherent or intellectually demanding the atmosphere is. Two teachers may be asking questions, reading stories, and teaching strategies, but they are probably not doing those things in equally powerful ways. Unfortunately, our observation instruments rarely get at these quality issues (though when they do, it is those items that seem to work best).

That means that most reading coaches are probably looking at the wrong things when they observe, and quite often the prescriptions they develop on the basis of their observations are more aimed at changing the instructional activities, rather than trying to make the current activities more substantial in getting kids to zero in on meaning.