Showing posts with label Reading First. Show all posts
Showing posts with label Reading First. Show all posts

Thursday, June 2, 2011

Comparing the Common Core and Reading First

Sorry about not posting for a while. I've been particularly busy, and even worked in a vacation. But my eye is back on the ball and I hope I have something of interest for you.

Recently, I received a letter: “I am writing to you because members of our state’s Department of Education believe that there is no alignment between the Common Core Standards and Reading First. Do you know of any document that aligns these two approaches for the early grades?” Clearly, the writer is concerned that the gains and improvements made through Reading First may be lost as attention shifts to the new common core, a fair point.

Unfortunately, I am not aware of any document that provides such a comparison, so I wrote my own. And here it is:

There are continuities, contrasts, and clear disagreements across the federally-supported Reading First (RF) effort and the Common Core State Standards (CCSS). I have tried to detail these below.

Continuities. Both CCSS and RF require(d) the teaching of:
1) Phonological awareness in K-1;
2) Phonics in K-3;
3) Oral reading fluency in grades 1-3;
4) Vocabulary (word meaning) K-3.

Anyone who may have anticipated a retreat from the curricular focus of Reading First will likely be disappointed. The key pillars of RF are still intact — now in the form of specific curricular goals that are to be accomplished at particular grade levels.

Discontinuities. But there are several key differences between CCSS and RF as well.
1) RF required the teaching of reading comprehension strategies; CCSS, on the other hand, does not require such teaching and emphasizes greater attention to the meaning of texts (in other words, less strategy teaching and more focus on the text content).
2) RF required that the approaches used to teach reading be proven to work through research; CCSS is, generally, silent about HOW the teaching is to proceed (instruction is treated as a matter of state/local/individual choice).
3) RF required the adoption of a core program (CCSS is silent as to how to teach or what materials to use).
4) RF required the use of screening and monitoring tests, particularly with regard to phonological awareness, phonics, and fluency, while CCSS is silent about the use of such assessments (though they encourage formative testing of reading comprehension with older kids—3rd grade and up).
5) RF required that interventions be available for children who were not making sufficient learning progress; CCSS is silent on that issue.
6) RF required extensive professional development for teachers and principals (through various state and local mechanisms, including reading coaches); CCSS is silent on the value of professional development of any kind.
7) RF was silent about the difficulty of the text that children were to work with; CCSS is quite prescriptive about the readability levels of the texts that teachers must use.
8) RF was silent about the content of the texts that children were to read; CCSS is somewhat prescriptive in encouraging greater attention to canonical literature and the classics, and to the inclusion of informational text.
9) RF was silent about technology, while CCSS requires that students be taught to use technology.
10) RF only focused on reading, but required teachers/schools to follow their state’s English language arts requirements (for writing, spelling, oral language, etc.), while CCSS specifies instructional goals for these other areas of language teaching.

Given these differences, it could mean that, under the common core, a state/district/teacher could simply do their own thing, and stop paying attention to the research, stop monitoring student learning, stop offering professional development for teachers and principals, stop using core materials, and stop their intervention programs… Although these items were required under RF, they clearly are not being emphasized in CCSS.

However, there are some pretty good reasons not to retreat from those RF staples:
a). CCSS may not have many implementation requirements, but that doesn’t mean such requirements are gone. Both IDEA and Title I funding, for example, require several of these items be address including research-based teaching and formative assessment.
b). There is going to be increased pressure to succeed and several of the RF requirements tend to help schools to succeed. States and schools are now going to be compared on the same metrics going forward, which means that there will be even greater accountability pressure on teachers, principals, state superintendents, and now even on the governors themselves (letting everybody do their own thing, no matter how greatly such practices may deviate from research findings may not be something that those being evaluated will be willing to accept.
c). Most state data showed Reading First was supporting better reading achievement for the kids served, so why not build on existing knowledge and tools already available rather than starting all over again?

Sunday, August 31, 2008

Which Reading First Idea Has the Least Research Support?

Reading First is the federal education program that encourages teachers to follow the research on how best to teach reading. The effort requires that teachers teach phonemic awareness (grades K-1), phonics (grades K-2), oral reading fluency (grades 1-3), vocabulary (grades K-3), and reading comprehension strategies (grades K-3). Reading First emphasizes such teaching because so many studies have shown that the teaching of each of these particular things improves reading achievement.

Reading First also requires that kids get 90-minutes of uninterrupted reading instruction each day, because research overwhelmingly shows that the amount of teaching provided makes a big difference in kids’ learning.
It requires that kids who are struggling be given extra help in reading through various of interventions. Again, an idea supported by lots of research. Early interventions get a big thumbs up from the research studies.

It requires that teachers and principals receive lots of professional development in reading, the idea being that if they know how to teach reading effectively, higher reading achievement will result. The research clearly supports this idea, too.

It requires that kids be tested frequently using monitoring tests to identify which kids need extra help and to do this early, before they have a chance to fall far behind. Sounds pretty sensible to me, but where’s the research?

Truth be told, there is a very small amount of research on the learning benefits of “curriculum-based measurement” and “work sampling, but beyond these meager—somewhat off-point—demonstrations, there is little empirical evidence supporting such big expenditures of time and effort.

This isn’t another rant against DIBELS (the tests that have been used most frequently for this kind of monitoring). Replace DIBELS with any monitoring battery you prefer (e.g., PALS, Ames-Webb, ISEL, TPRI) and you have the same problem. What do research studies reveal about the use of these tests to improve achievement? Darned little!

There is research showing that these tests are valid and reliable, that is they tend to measure what they claim to measure and they do this in a stable manner. In other words, the quality of these tests in terms of measurement properties isn’t the problem.

The real issue is how would you use these tests appropriately to help improve kids’ performance? For instance, do we really need to test everyone or are there kids who so clearly are succeeding or failing that we would be better off saving the testing time and simply stipulating that they will or will not get extra help?

Or, are the cut scores really right for these tests? I know when I reviewed DIBELS for Buros I found that the cut scores (the scores used to identify who is at risk) hadn’t been validated satisfactorily. Since then my experiences in Chicago suggest to me that the scores aren’t sufficiently rigorous; that means many kids who need help don’t get it because the tests fail to identify them as being in need.

Perhaps, the monitoring test schemes (and the tests themselves) are adequate, but in practice you can’t make it work. I have personally seen teachers subverting these plans by doing things like having kids memorizing nonsense words, or having kids read as fast as possible (rather than reading for meaning). Test designers can’t be held accountable for such misuse of their tests, but such aberrations cannot be ignored in determining the ultimate value of these testing plans.

There are few aspects of Reading First that make more sense than checking up on the students’ reading progress, and providing extra help to those who are not learning… unfortunately, we don’t have much evidence showing that such schemes—as actually carried out in classrooms—work the way logic says they should. I think it is worth continuing to try to make such approaches pay off for kids, but given the lack of research support, I think real prudence is needed here:

1. Administer these tests EXACTLY in the way the manuals describe.

2. Limit the amount of testing to what is really needed to make a decision (if a teacher is observing everyday and believes that a child is struggling with some aspect of reading, chances are pretty good that extra help is needed).

3. Examine the results of your testing over time. Perhaps if you systematically adjust the cut scores, you can improve student learning. It is usually best to err on the side of giving kids more help than they might need.

4. Don’t neglect aspects of reading instruction that can’t be measured as easily (such as vocabulary or reading comprehension). Monitoring tests do a reasonably good job of helping teachers to sort out performance of “simple skills.” They do not, nor do they purport to, assess higher level processes; these still need to be taught and taught thoroughly and well, however. Special effort may be needed to ensure that these are adequately addressed given the lack of direct testing information.

Saturday, August 2, 2008

Explaining the Reading First Impact Study (New Districts Added)

Recently, I wrote about the Reading First Impact Study in this space. That
struck a nerve and received much attention and generated many questions. Given that, here I will answer some of these inquiries. Feel free to send more along and I’ll see what I can do. Hope this helps readers to better understand this study; various political statements recently have suggested that many politicians, at least, don't get it.

The Impact Study showed that the Reading First schools made no improvements in reading comprehension, so the program did not work, right?
No, that’s not correct. The Reading First Impact Study collected no comparison data from these schools for years prior to their entry into the Reading First program. With no comparable data from previous years, it is impossible to determine if the Reading First schools improved or not.

That makes no sense… the schools had to have test data from previous years to be identified as Reading First eligible. Why not use those test data?
There definitely are past year’s test data from all of the Reading First schools and from all or most of the comparison Title I schools. Unfortunately these data are not comparable or combinable. The problem here is that with so many states and districts participating in the Impact study, it would be very expensive and nearly impossible to do enough equating studies. Of course, many districts and states have been reporting that they are doing better on their local measures. This may seem like the state data are contradicting the national data, but they are not. The Reading First Impact Study did not attempt to measure these kinds of improvements against past years’ achievement levels, though at least some of the states did and Reading First did well in those analyses.

If the Impact Study didn’t look at reading improvement against past years’ performance levels, what did it look at?
The Impact Study compared reading achievement gains for Reading First and non-Reading First schools. The study attempted to tell, from the beginning of the study to the end of the study, whether kids learned to read better as a result of being in Reading First schools.

Is it true that the comparison group schools were too different from the Reading First schools to allow a fair comparison?
No. Although the kids and the schools were not exactly the same at the beginning of the study, the research design that was used provides the closest comparison possible without a randomized control trial. If the schools had been randomly assigned to the Reading First treatment and control group it would have been an even better study. Nevertheless, the regression discontinuity design that was used should have provided a fair and conservative test of the effectiveness of Reading First since it guarantees a comparison with the most similar non-Reading First schools with regard to initial reading achievement, student mobility, free lunch eligibility, and so on. Reading First schools tended to be very slightly worse performing in these various measures at the beginning of the study, but not significantly so.

Why not just do a randomized control trial if that is better?
The Institute of Education Science that commissioned and supervised this study definitely wanted to do it that way, but the study began too late and the Reading First money was already distributed by the time the study was under way. Various Department of Education officials and consultants were very angry about this, but ultimately agreed with the research experts and statisticians that regression discontinuity would provide the best comparison under the circumstances.

If the study compared performance of Reading First schools with the performance of very similar controls, then doesn’t the study show that Reading First doesn’t work?
That was certainly the idea of the study and it may be showing that, but there are reasonable alternative explanations of the data that can’t be ruled out. That’s the problem. If everybody was nearly equal at the beginning of the study in instructional context and student achievement and then you put Reading First programs in half the schools, you should be able to determine whether Reading First kids were advantaged. That was the idea of the study. But it might not have worked the way it was planned.
Why didn’t the study work?

The assumption that the comparison group schools would continue with their initial instructional practices while the Reading First schools were changing theirs seems not to have been met. Researchers refer to this as “contamination.” For a comparison to work, the two groups have to engage in different practices. If they are doing the same things, why would you expect outcome differences to result for one of the groups?

For example, if I were setting up an experiment, I would try to arrange it so that my experimental and control classes were in different schools if possible, because teachers may share ideas in the teacher’s lounge and then my experimental innovations may start appearing in the control classrooms. The more this happens, of course, the less chance I have of finding differences in the end, and when my experiment shows no effects does it mean that my treatment didn’t work or just that it worked in both sets of classrooms.

Is there any reason to believe that kind of contamination affected the Impact Study?
Yes, in fact there are lots of reasons to think this was the case. First, let’s start with the Reading First law itself. The U.S. Department of Education distributed approximately $1 billion per year for Reading First. About 80% of this money was given to the states to pass onto the Reading First schools to be used to purchase materials, hire coaches, provide professional development for teachers and principals, and for interventions for struggling readers. The other $200 million per year was to be spent by the states to try to contaminate the comparison sample.

You made that up. Does NCLB really say that the states were to contaminate these data?
The law reads differently than I said it, but in fact, that is exactly the idea of it. The President and Congress recognize that there are too many failing schools for the feds to bail out all of them (that would just be too expensive). What they do instead is provide money for the establishment of quality programs with the hope that this will leverage state and local dollars towards solving the rest of the problem. In this case, they actually earmarked about $1 billion to be used by the states to try to encourage Reading First reforms through the entire school system, especially with those other failing schools. As I said earlier, the more the other schools adopt Reading First practices, the less meaningful any comparison becomes. (Not only were the states strongly encouraged to spread the program beyond the Reading First schools there were other Department of Education initiatives to encourage this: from presentations of Reading First approaches at Title I conferences to special initiatives like “Expanding the Reach” that set out to incent schools to use their Title I funding to carry out Reading First style initiatives).

Just because lots of money was spent by the feds to get other schools to adopt Reading First strategies does not mean that they actually did it, right?
That’s true. But my own personal experience in visiting various districts suggests to me that there are many examples of districts that adopted Reading First practices district wide. Let’s say, you have 25 schools in your district and four of these schools were Reading First eligible. You take the Reading First funds and carry out the initiative in those four schools, but what about the other schools? You could continue what you have been doing in the past, or you could repurpose your funds to duplicate the Reading First efforts in all of your schools. That means 25 schools would be using the Reading First model, even though only four were funded. That’s a terrific deal for the federal government (they managed to guide a reform in a large number of schools for a relatively small amount of direct expenditure). If your district was part of the Impact Study, however, it would certainly have contaminated the sample to some extent.
This week I sent a few emails to friends around the country. I told them that I personally knew of districts that had done this kind of district-wide Reading First effort and I wondered if they knew of any others. Below I have listed the districts that this very informal (and unscientific) survey uncovered. These are sizeable districts and at least some of them actually did take part in the Impact Study. I wonder what a more formal study would show? Given how many contaminated districts I identified without looking hard, I suspect we’d find that many of the Reading First reforms were intentionally duplicated by comparison schools which wrecks the comparison. (I have not listed those schools and districts that partially adopted the reforms. For example, in many districts, since the Reading First schools were getting a core program, they bought the same program district wide. Or, places like Chicago, adopted DIBELS testing in all primary grade classrooms, not just the Reading First schools. These situations certainly would introduce contamination to the study, but this kind of partial replication is so common and so widespread, I would likely need to list most of the Reading First districts (and a large percentage of states).

In a survey of California Reading First's 121 school districts, 47 of the 52 respondents indicated that they have duplicated the practices of Reading First district wide.

Also, Florida has a state policy requiring that all districts have a core program, a 90-minute literacy block, and screening and monitoring assessments. Of course, some local districts in Florida have made particular efforts to carry over Reading First to their other schools (some of those are listed below). I've been told that the same is true in Alabama, but I haven't been able to verify.

The Bureau of Indian Education has expanded Reading First into 35 non-Reading First schools, too.

Here are some school districts, large and small, that adopted the Reading First reforms district wide. If I hear of more, I’ll add them to the list. You can see the problem.
East Aurora, IL
Syracuse, NY
North Platte, NB
Roseburg, OR
Hillsboro Co., OR
Jefferson Co. , OR
Klamath Co., OR
Butte, MT
Helena, MT
Great Falls, MT
Laramie, WY (Albany #1)
Medford, OR
Ontario, OR
Alamogordo, NM
Moriarty, NM
Espanola, NM
Delta, CO
Fort Morgan, CO
Lamar, CO
Tacoma, WA
Topeka, KS
Ogden City, UT
Richmond Co., GA
North Sanpete School District, UT
San Juan School District, UT
Collier County, FL
Broward County, FL
Wilmington, DE (Christina District)

If a bunch of non-Reading First schools adopted the same reading reforms, would that mean the study was contaminated?
No matter how widely this phenomenon occurred, if it didn’t occur in the districts that participated in the study, then it would not matter. However, as I indicated above at least some of the districts in the study did follow a policy that required the use of Reading First practices in non-Reading First schools. (And my list above only includes districts that were trying to duplicate the entire Reading First effort in their other schools. There were also districts that did this more partially: for example, the Chicago Public Schools adopted DIBELS monitoring district-wide, but didn’t try to spread the entire reform package. Partial imitations are contaminating, too.)
Also, the feds did not just do an Impact Study. They carried out an implementation study that examined the instructional practices in the two sets of schools. Some of these data have already been reported and more will be reported this fall. What the already released data show is that Reading First and non-Reading First schools were quite similar in their instructional practices, and that they became increasingly similar as the study progressed year to year. So, in Year 1 Reading First schools were much more likely to adopt a new, research-based core program than were the comparison schools (a big difference). But by Year 3, most of the non-Reading First schools were using the same kinds of programs (and often the identical program). The same thing seemed to happen with coaches. Coaches were prevalent in Reading First schools from the beginning, but they became increasingly available in Title I comparison schools as the study progressed. This also happened with setting aside an uninterrupted instructional block for reading, as well as for some of the other significant instructional reforms.

Usually in a multi-year experiment of this type, the impact of the treatment grows each year as more innovations are implemented and as the distance between the schools grows with regard to their instructional practices. With the Reading First study, big initial differences declined over time as other schools parroted the Reading First practices.
If lots of schools took on the Reading First reforms we might not be able to see differences among those schools, but should achievement be improving overall since so many schools would be using these practices?
That’s a fair point. And yes, that appears to be the pattern that we are seeing with the National Assessment of Educational Progress. In fact, NAEP scores have been rising during the period in question. NAEP shows small but clear significant improvements for fourth graders (particularly on their trend items) and the various local studies are saying that state test scores have been rising, too. Yes, it is possible that these increases are real and that they have been stimulated by Reading First.

Wouldn’t that mean that Reading First was actually a big success?
It would if it could be proven that the changes in instructional practices that have been taking place are actually due to Reading First. Although there clearly are instances noted above where schools adopted practices because they were used by Reading First, there are also cases where other factors may have actually led to the change. Districts like Los Angeles and Chicago had already hired reading coaches before Reading First money was even available (they were relying on the same research base used by the Reading First creators, but were acting independently). Many districts that use core programs refurbish those programs every 4 or 5 years; their latest adoption may have been a program that could be used in Reading First, but that might not have been why they selected the program. Reading First might have been the pivot point for all of these changes, or it might have been just one of many sources of information used by the districts.

Tuesday, July 29, 2008

Reading First for English Learners

As I write this, I’m in Nashville, TN at the 5th annual National Reading First conference. It appears to be their last meeting, and my first appearance at one of these affairs. Of course, there is a lot of sadness as most state people are resigned to the idea that Reading First funding is not to be renewed. Yesterday, at the opening of the meeting, Laura Bush apparently cheered folks’ spirits by calling for reinstatement of full funding to Reading First (I was on a plane at the time, but morning radio caught me up on what I missed by playing some of her 5-minute speech). As much as I’d like to see Reading First continued, I don’t see any chance of it now and really think we need to turn our attention to a new federal effort. (The one thing that I think to be missing at this great conference is some time for state people to describe what they see as their successes and failures; any smart new program is going to have to be based on both—what they managed to do well and what they didn’t. I think the Department of Education missed out on a great opportunity to get this information. I suspect there won’t be any more meetings with this many state reps, so it is a real loss.)

One of my candidates for improvement in a next program is a more concerted and research-based efforts towards addressing the literacy learning needs of English learners. Most states just carried out the Reading First mandates with these kids despite the fact that the National Reading Panel report (the basis of Reading First) didn’t consider studies of English learners. The panel recognized the importance of this issue, but left it to another panel and that means Reading First directors were stuck trying to adhere to mandates that were at best insufficient for these kids. Fortunately, what Reading First was doing wasn’t that far off so no great harm was likely done, but what a lost opportunity. One suspects with a more tailored approach we could have seen greater success for this significant group. That is what I am doing here in Nashville: talking about the report of the National Literacy Panel for Language Minority Children Youth and what it says we need to do to help second language learners. At least for this group, wouldn’t it have been terrific to have had a sixth instructional element—one focused on the development of oral English?
The information that surprised the Reading First audience the most? They seemed startled when they found out how little research has been carried out with second language learners. I included a chart showing the comparison of the numbers of studies on various topics that the National Reading Panel looked at for first language learners and the numbers available on those topics for second language learners, and that got a visible response from the conference attendees here. This is definitely an area where we could use some more research help, but even if we had more information, we’d need policies that guided the implementation of such research-based efforts.

Below I have included a link to the Power point that I used in my presentation. The front part of the file is a pretty direct summary of the NLP report, and the conclusions are my sense making of the information.

Wednesday, July 9, 2008

Pulling the Plug on Reading First

I wonder if you have seen the various editorials that have been appearing about Reading First recent weeks? These are reactions to the Reading First impact study and Congressional efforts to defund Reading First that I wrote about in this space recently. The Boston Globe came out for reauthorization of Reading First, as did USA Today.

In a deft unsigned editorial, USA Today called for continuation of funding for Reading First, albeit with some needed reforms. Their position: “let’s keep funding a good program even as we try to improve upon it” was their very reasonable position. They paid attention to reports by the GAO, the Center on Education Policy, and a recent U.S. Department of Education review of state reading data and concluded this experiment to improve reading for kids should continue. They still claim this is mainly phonics reform (it is not, or at least, it is not supposed to be that), but their overall view of this is sound and reasonable.

For this kind of editorial, USA Today fairly invites an opposing view to counter their position. Fair enough, but that means if they come out pro-Mother in their Mother’s Day paper, they need to find someone who is anti-mother to state the opposing view. In other words, the opposing position can just be an ill-advised and ridiculous position, as was the case today.

Two things stand out in this editorial: one, the politicians who are leading the charge to stop improving reading instruction for young kids didn’t have the courage to respond. They apparently were ashamed of their political position and figured the fewer people who knew they were voting against Reading First the better. Senator Tom Harkin from Iowa and Congressman Obey from Wisconsin want to hide their dirty work for some unknown reason (I suspect Senator Harkin knows that schools in Iowa have been doing great with Reading First—what an impressive group of teachers and principals—and he probably doesn’t want to make it too public that he is willing to undermine their efforts for some political reason).

USA Today couldn’t get any of these brave political souls to explain why they were going to swipe the kids’ reading money, so they brought in Steve Krashen to do it. Krashen has opposed teaching kids to read for years (he believes that if we just give kids books they will read just fine without all that messy teaching). He manages to pack an amazing amount of misinformation into 300 words, including claims that 99% of American adults can read at a basic level (a finding at variance with all data that have ever been collected on this population), and that the impact study shows that Reading First has made no difference on children’s reading.

I worked on the impact study and think the results are important, but as I have written before, this study can’t possibly prove what Krashen now claims. I appreciate that USA Today seeks opposing views, but Krashen knows nothing about why Obey and Harkin want to swipe the kids’ reading money and, therefore, he couldn’t possibly provide insightful commentary on this issue.

I suggest readers ignore Krashen’s irrelevant opinions, and reply to Harken and Obey directly. Krashen loves libraries, but he continues to show little respect for the work of teachers. (In this piece he says nothing about improving reading instruction for kids, professional development for teachers, or the organization and management of challenged schools—claiming that if we just put more libraries in poor neighborhoods we won’t have a reading problem). Write to Obey and Harkin or your friends and family in their districts. I appreciate that they hope to tweak the administration, but screwing the kids out of this reading program is not a good way to do that.

Tuesday, June 24, 2008

Reading First is Dead

Two very interesting reports came across my desk yesterday--within minutes of each other.

The first one was an Education Week story that said the House Appropriations committee intended to kill off Reading First.
This is no surprise, since Rep. David R. Obey, D-Wis., is the chair of that committee and he has shown a strong penchant for using his power for political reasons with little regard for educational needs. He has been anti-Reading First for a long time (mainly, I suspect, because it was proposed by a Republican), and the unfortunate management problems along with the recent interim report (see my earlier Reading First blog) make it easy for him to play politics with this. Lots of Congressmen will be sad to see Reading First go, since their home districts like it, but Obey will make this pill go down easier by expanding Title I funding (more money to schools that still aren't sure how to spend it in ways that will help kids).

A little later I received a press release from the U.S. Department of Education: NEW READING FIRST DATA FROM STATES SHOWS IMPRESSIVE GAINS IN READING PROFICIENCY.
This report claims that "students from nearly every grade and every subgroup show improvement" and goes on to report on information drawn from the states. This study provides a picture of Reading First very much at odds with the one evaluated in the recent Institute of Education Sciences study that found no reading comprehension improvement. According to these new data, kids improved in reading comprehension, second language kids improved, etc. and these gains were big.

So what does all that mean? In science, when you have conflicting data, you sharpen your pencil and try to figure out how to collect new data that will resolve the differences. In politics, you hold a finger up in the air and try to determine which way the wind might be blowing. I suspect Congressman Obey's finger is going to win the day over any scientific approach. Translation: Reading First is dead. It could have withstood the corruption described in the Inspector General's report or the interim impact study--but not both!

Under the circumstances, "Reading First" is politically toxic, no matter how effective it may be or how popular with the schools. I doubt that any candidate can easily embrace Reading First (Senator Obama seems to have enough money and a large enough margin in the election that he could afford to take a risk on it, but he didn't embrace Reading First when it would have been easy to do so, so I wouldn't look for any support there). His proposals for reforming education have a lot more to do with increasing funding and improving the tests than rethinking curriculum, professional development, or interventions for struggling kids (I documented his education views in an earlier blog).

So, if I am right that Reading First is dead, where are we? I hope that everyone will think of Reading First as only a first salvo in a much needed rethinking of Title I spending. The demise of Reading First will simply mean that we need a second attempt to rethink Title I spending (one that again will provide strong guidance to states and local districts in how to expend certain additional funds in ways more likely to raise reading achievement than what the districts have been doing on their own). This new effort has to be different from Reading First, but clearly based upon lessons learned from it. That means, those aspects of Reading First that were positive (and there were many), need to be preserved--and those that were problematic need to be rethought. I described some key changes that I thought were necessary for a second attempt at using federal money to improve schools and not just fund them.

The idea of this new Reading effort would be three-fold: (1) to try to immediately and powerfully improve a small set of struggling schools (something Reading First might have been doing, but it is impossible to be sure given the conflicting evidence); (2) to serve as an immediate model for all Title I schools to start trying to emulate now (as they so clearly did with Reading First--look at the degree of emulation described in the Reading First implementation study); and (3) to ultimately identify a set of policies that will eventually become requirements for all Title I schools (we cannot continue to spend more than $15 billion of federal money each year in high poverty schools without a sound educational return for that money--Congressman Obey might think that is okay to continue like this, but it has been a meat grinder for poor children!)

Monday, June 16, 2008

More on the Reading First Evaluation

I was talking with Dick Anderson today. For those who do not know, Dick is an outstanding scholar at the University of Illinois at Urbana-Champaign. He was the director of the Center for Reading during the years when the best research on reading was coming from there. His comprehension research is great.

Dick is not anti-Reading First in my opinion, but I think it is fair to say he isn’t exactly a big Reading First fan. He isn’t against phonics, but tends to think Reading First makes too much of phonics. He feels the same about fluency and phonemic awareness (and, frankly, anything in the curriculum that he feels is not intellectually engaging).

Dick was arguing that Reading First was not evaluated properly. He did not feel that the “regression discontinuity” design used in that study provided an adequate or appropriate test of the effectiveness of Reading First. I’m not a statistician and have wondered in this space whether there are problems with that design. Dick thinks that it reduces the variance in outcomes and reduces the chances of finding a difference.

He might be right, as I say, I’m not a statistician. I still think the bigger problems are these: (1) Reading First (and lots of related policies and information) “contaminated” the control group schools by making their reading programs much more like Reading First than they once were—making it an effective policy, but queering the research study; (2) Reading First schools have such high mobility rates that it is impossible to study them longitudinally. My claim in that last point is not that the Reading First schools are incomparable to the non-Reading First schools because they have such high student and teacher movement (the other high poverty schools suffer from that kind of mobility problem). But any reading intervention aimed at poor schools somehow has to work within the confines of such high mobility.

Of course, Lauren Resnick, the head of the Learning Research and Development Center at the University of Pittsburgh, points out that mathematics improvement has taken place over the past 20 years in the U.S., including in schools like those. The big difference being that learning math across languages is not as challenging as learning reading across those differences.

Language differences and high mobility in poverty schools are not excuses for why Reading First didn’t do as well as it needs to in research. But they are issues that must be addressed adequately in any Reading First redesign or that policy will fail too, if the goal is higher reading achievement for poor kids.

Thursday, June 12, 2008

What Every Teacher Should Know about Reading Comprehension

Recently, the impact study on Reading First indicated that the program had shown no effects when it came to improving performance on reading comprehension tests. That is a real problem because the whole idea of Reading First was to improve kids' reading to such an extent that they would comprehend better. I wonder if teachers and principals (and Reading First directors) knew more about reading comprehension if things would have gone better.

I'm asked frequently by schools to come and help with reading comprehension, but as with the Reading First folks, I often sense that there are many things that these folks don't know enough about in order to make real progress in improving reading comprehension. However, I sometimes think they mess up for the opposite reasons that the Reading First people do.

Many teachers, especially in the upper grades, think that they only need to teach comprehension and everything will be fine--neglecting the need for instruction in decoding, fluency, and vocabulary. I think in Reading First teachers too often lost sight of two key points: (1) that reading comprehension can and should be taught explicitly and (2) that the enabling skills (decoding, fluency, vocabulary) need to be taught in ways that aim them at reading comprehension. Decoding needs to be taught, but lessons in decoding should always end with kids reading new text with their new skills. Vocabulary needs to be taught, kids need to read text that uses that vocabulary, and they need to think about what the word meanings have to do with interpreting the text. Fluency instruction should be supported by having students answering questions or reacting to the meaning after each rereading (it is more than a race to read fast).

If you go to this link you can download my new powerpoint, 10 things every teacher should know about reading comprehension.

Wednesday, May 28, 2008

How Would I Fix Reading First?

A recent research report said Reading First failed to improve students’ reading scores. I was disappointed given the hard work of so many teachers, but the study was far from perfect.

The Department of Education was more efficient in getting Reading First underway in the schools than it was in getting the study off the ground, so they couldn’t carry out a nationwide randomized controlled trial. Unfortunately, the study only looked at reading comprehension scores and not at performance in any of the underlying skills that support comprehension (so you can’t tell whether the program impacted those skills or not).

Far more serious was the problem of contamination. While the feds were paying to change the Reading First schools, schools across the nation were adopting the same reforms without this help. It’s good that schools were making these changes, but the more like Reading-First-schools that they became in curriculum, materials, professional development, and assessment, the less chance that Reading First could be shown to be making a difference. If the teaching is the same, you just can’t expect any difference in outcome.

I have been asked by reporters about whether Reading First should be jettisoned. Obviously if it doesn’t work, we don’t want it. But the research so far has not convinced me that it can’t be made to work. As sobering as these data are, I think they should move us to change Reading First rather than kill it.

So what would a new and improved Reading First look like?

1. Amount of instruction would increase.
The feds required Reading First schools to commit to 90 minutes per day of reading instruction. I’ve been critical about that amount because according to surveys, that is less time than the typical primary grade teacher teaches reading. If you want to increase achievement, it is wise to increase the amount of instruction. Reading First contracts should commit schools to 120-180 minutes a day of reading and writing instruction.

2. Greater flexibility in instructional time.
Another Reading First rule is that the 90 minutes of instruction must be uninterrupted. That means that the PA system shouldn’t be going off every two minutes during reading time, or that the “special” teachers shouldn’t be pulling kids out of class during that time. I certainly agree with protecting time as well as you can from these intrusions, but generally it’s the amount of teaching that matters—not the structure of it. Anyone who has taught first grade (I have) knows you can’t teach for 90 minutes uninterrupted by bathroom breaks. I’d be more flexible with the time. The Reading First implementation study found Reading First kids were getting less than an hour a day of teaching. I blame the “reading block” concept. When 10:30AM comes, teachers are done teaching reading. I prefer that teachers provide the full allotment of time, no matter what the clock says. In other words, more flexible scheduling should mean kids get their allotted amount of teaching—no matter how long it takes.

3. Include writing in the curriculum.
The National Reading Panel (NRP) examined five aspects of the curriculum (phonemic awareness, phonics, reading comprehension, vocabulary, and oral reading fluency) and found that instruction in each of these provided students with learning benefits. The panel did not look at writing instruction, but if it had it would have found two important facts: writing can be taught and the teaching of writing can be done in ways that helps reading achievement. Reading First followed the research very closely in some policies, and not so closely in others. If the problem with writing was that it was not reviewed by NRP, then I suggest that this is a profitable area to go beyond the officially-reviewed research.

4. Expansion of phonemic awareness to include phonological awareness.
The National Reading Panel reviewed the research on phonemic awareness and found that teaching it was beneficial in kindergarten and grade one. Unfortunately, a lot of young children struggle unnecessarily with phonemic awareness. At least during the first half of kindergarten (and for those kids who seem to be going at a snail’s pace in developing PA), it is wiser to focus attention on larger sound units (phonological awareness), including word separations, syllables, and onsets-rimes. Expand the curricular focus in this area, especially with younger kids.

5. Greater focus on comprehension, vocabulary and fluency in professional development.
The Reading First implementation study found that Reading First teachers were doing nothing different from other Title I teachers with regard to comprehension and vocabulary instruction (and in most grade levels, oral reading fluency). This suggests to me that the professional development for these teachers (and the materials adoption) did not stress the research findings about reading comprehension, fluency, and vocabulary. Reading First should thoroughly implement all the research findings, not just some of them.

6. Spread the reform over 2-3 years.
Reading First teachers had to learn a new curriculum and new instructional approaches and implement them. They had to adopt a new core curriculum. They had to adopt new assessments. They needed to put in place new interventions for low readers. All of these are good things to do, but in Reading First they were all to be implemented in Year 1. I would suggest that a more effective approach would be to implement these reforms over time. Year 1 upgrade the program and get the new instruction in place, Year 2 start monitoring instruction with classroom assessments and put in place pull-out interventions. Year 3 add classroom interventions to the mix and so on. Give Reading First teachers a chance to get good with these things.

7. Ban use of monitoring assessments as accountability tests.
It’s a great idea to have teachers using assessment data to identify whether kids are learning what they need to. It is a dumb idea to use that data to determine if the teachers are doing a good job or if the program is working. Teachers should have no reason to “cheat” on monitoring assessments. I saw lots of children who were not learning adequately made to look like they were on DIBELS, so that the teachers’ scores would look good. Blah! Using monitoring tests for accountability undermines their success.

8. Focus coaches on coaching and expand professional development.
Reading First schools hired coaches. But these coaches spent an awful lot of time juggling DIBELS data, managing book orders and book rooms, and helping the principals administer the school reading program. The main focus of the coaches should be on improving teaching and that means they should spend most of their time on coaching, observing, critiquing, answering teacher questions, doing workshops, etc. Eighty percent of the coaching time should be devoted to coaching.

9. Involve parents.
The Bush administration was upset that teachers take so little responsibility for kids’ learning that they wanted all of the focus of Reading First on what the schools can do. I am sympathetic to the view that educators need to take greater responsibility, but not to the degree that I’m unwilling to ask for parents’ help. The research says that parents can help their children learn to read. If we are going to spend all this money, we need a “full-tilt boogie” kind of response to kids’ learning—doing everything, and I mean everything, that we can to help them read better. Leaving the parents out of this is hurting.

10. Expand the grades covered.
Reading First focused only on improving the teaching of reading in grades K to 3. It would be wise to aim it at entire elementary schools, no matter what grades they include. The research base of Reading First was not drawn entirely from research on the primary grades, and the implementation of this program needs to be widened. It will be easier to implement some of these changes if they are school wide, instead of just primary (that may also focus teachers and coaches on comprehension and writing more, too).

11. Include classroom management in the professional development.
At least some of the professional development that the coaches could provide should be focused on classroom management and discipline. One of the major reasons Reading First teachers squandered about a third of the required teaching time was due to poor classroom management. Better managed classes provide a greater opportunity for learning to read, so Reading First should expand its menu of required professional development topics to include emphasis on time use and management, not just on reading instruction itself.

12. Help English learners more.
The National Reading Panel did not look at research on second-language learners so Reading First did not adequately address these students’ needs. Now the National Literacy Panel for Language Minority Children and Youth has reviewed this research and Reading First should make some adjustments for the needs of English learners. One of the biggest points that should be made is that it is essential that there be an English oral language period or ESL time included in the instructional model for these kids.

Friday, May 2, 2008

Reading First Impact Study: Trouble in Paradise

This week the U.S. Department of Education released the impact study on the effectiveness of Reading First.

Reading First is the largest reading improvement initiative ever undertaken by the federal government. It is a program that I have been a big supporter of since its inception. Unfortunately, Reading First got mired in controversy from the start. Of course, there were the Reading Wars guys trying to stop any effort to apply research findings that they don't like. You expect that. But the Reading First leadership made some foolish errors that raised ethical questions (like highlighting particular commercial programs). At the time, those claims were refuted successfully, but more recent investigations by the Inspector General showed that the administration of Reading First had been tainted in various ways, and the Department of Education has been trying to fix it ever since.

The argument of Reading First apologists, like me, has been that it doesn't matter how the adults screwed up and what they did wrong as long as the program is still serving needy children effectively. Various state reports have shown benefits to Reading First. The report issued this week is the first nationwide look and it is a pretty nifty piece of research. It found no major outcome benefits for Reading First (some small improvements in some variables, but no big improvements on standardized reading comprehension tests).

So, do I still support Reading First? I do, but serious changes need to be made or this program needs to be replaced by something better. Reading First is very expensive (about $1 billion per year) and it has required a lot of change on the part of teachers. This would be worth it if it helps (since it is aimed at addressing the needs of kids who are not succeeding in school), but it is too expensive of tax dollars, good will, and effort if it doesn’t work. And it is not working sufficiently in my opinion.

Reasons I would not give up on this yet:
The study, as well designed as it was, is contaminated (meaning that the control groups were often doing the same thing that the RF groups were doing). To test the effectiveness of something, you try to make as clear a comparison as possible, and you make efforts to prevent the control group from doing what the experimental groups are doing. But this evaluation was not a true experiment, it was a best faith effort to make sense of a public policy. Public policy in this case both required a research study and then set out intentionally to contaminate the control groups.

The Reading First law encouraged states to spend up to 20% of the money encouraging non-RF schools to adopt the same reforms (money that was used quite aggressively in places like Massachusetts, West Virginia, Georgia, and probably lots of other states). The feds themselves created a program in Washington, Tennessee, and Massachusetts (Expanding the Reach) designed specifically to teach non-Reading First schools how to adopt the Reading First approaches. The textbook companies were part of this, too. All the major core publishers made big changes in their K-6 reading programs to be eligible to sell in RF schools, and according to an earlier RF-implementation study, about 20-25% of the control group schools adopted those same programs EACH YEAR that Reading First was implemented (meaning by the third year, more than 60% of the groups were following the same curriculum). School districts like East Aurora, IL and Syracuse, NY required all of their non-Reading First schools to adopt the same reforms as their own RF-schools, using local money. Districts like Chicago did not go that far, but when Reading First schools adopted DIBELS, so did all of the other schools in the district.

What all that means is that the comparisons were not Reading First with non-Reading First schools, but Reading First with less-Reading First. That means the states who say Reading First is helping improve their test scores could be right, but this effect is coming not simply through the improvement of the small number of schools who get Reading First money, but from altering the whole system. (This isn't just an excuse, it is something I explained to the designers of the study at their very first meeting. In fact, my explanation could even be an argument for ending Reading Firs… since it has already impacted the system overall, there may be no reason for this program to continue.)

Another issue is that the amount of student, teacher, and principal attrition is very high in these Reading First schools (and admittedly it is equally high in the non-Reading First schools with which they were compared). Efforts like Reading First make great sense, perhaps, but measuring their success by comparing beginning of year and end of year cohorts may not be an adequate test of effectiveness given this mobility problem. You could take the approach of only testing kids who are there for the full year, two years, three years, of course, and I have no doubt that this reform would do better with those kids (but the stable part of a school population generally does better). That kind of analysis might suggest that the Reading First teaching is working, but that the policy itself has failed (since because of the high mobility of low-income populations a program configured in this way would not be adequate to meet the student needs). This is a reasonable study of the effectiveness of the policy, it is not such a good test of the effectiveness of these teaching approaches.

Implementation studies show that Reading First has truly been a partial implementation. Reading First was supposed to enhance and increase teaching in 5 areas for 4 grade levels (phonemic awareness was only supposed to increase in K and 1, phonics in K, 1, and 2, and fluency in 1, 2, and 3). The earlier implementation study showed no changes in vocabulary or comprehension instruction at any grade level, and only a change in fluency instruction at a single grade level. There is another implementation study on the way that will provide an even closer look at these kinds of issues. To me these data show inadequate or partial implementation of Reading First (which is a serious issue for those who have managed it), but it is not a clear test of the effectiveness of the overall model itself (though it might say that this model can’t be implemented very effectively). Kids were supposed to receive improved instruction in phonemic awareness, phonics, oral reading fluency, vocabulary, and reading comprehension. If they only received improved instruction in some of the skills at some of the targeted grade levels, it is certainly possible that this lack of thorough implementation has undermined the success.

Finally, as well done as this impact study was, it is based on an assumption that might not be true. The study used a regression-discontinuity design meaning that only the initially-highest-achieving Reading First schools were included in the comparison sample (since they had to be as similar to the other Title I schools as possible). The assumption is that these schools would be appropriately representative of the entire distribution of low-achieving Reading First schools, but that is not necessarily the case. Perhaps more gains would be evident in the schools that were struggling more (and maybe not—this is just a possibility that needs to be considered--are schools below the 15th percentile similar enough in their reformability to allow the highest performing schools in this set to represent all the low schools?).

To me, the meaning of this is that Reading First has not unambiguously been shown to be beneficial. That leaves three choices for policy makers: (1) end Reading First now--either declare victory for helping change the system or failure for not benefiting these schools enough, but end it now; (2) modify the implementation of Reading First--this was really the first big reform efforts done at a federal level and much has been learned. Keep the money flowing, but change the program to make it more effective, and keep evaluating its effectiveness. (3) leave Reading First as is.

The results are too poor to do number 3 in my opinion, but the limitations in these data (and the small gains that are evident) seem sufficient to me to encourage us to keep trying to find ways to improve achievement for these kids and to build on the initial base of Reading First so that it works. Perhaps the critics are right that schools can't succeed with low income kids and that, therefore, poverty issues must take precedence over educational ones. I don't think so, so want to keep trying to make schools work for this generation of kids who are growing up in poverty.

Very disappointing.

Sunday, February 10, 2008

Is Reading First Really Such a Good Idea?

Last week, Garrison Keillor (of Prairie Home Companion fame) wrote a remarkable column. In it he bashed liberals for bashing Reading First. Reading First is the federal program brought on as part of No Child Left Behind. It provides struggling elementary schools with a billion dollars a year with which they are to transform reading instruction in grades K-3.

Specifically, the schools that get Reading First money are to be failing in reading (unlike with Title I money that goes to schools with high percentages of kids from low income families--RF money only goes to low income, low performance schools).

Also, Title I money has few strings. Schools can do almost anything they want with Title I dollars, even if what they do doesn't help kids do better. RF money is not like that: schools have to agree to follow the research, as articulated by the National Reading Panel (of which I was a member). RF schools have to provide substantial professional development to their teachers and principals, and they have to adopt comprehensive instructional programs that include teaching of phonemic awareness, phonics, oral reading fluency, vocabulary, and reading comprehension. And they have to monitor kids learning, and if they are not learning, the school is expected to give the kids even more teaching. In other words, Reading First schools are expected to push hard to help poor kids read.

It has been tough sledding, with lots of bumps along the way. Initially, the U.S. Department of Education held regional workshops to prepare the states to apply for this money. This was important because the money was not going to be given out automatically; the states actually had to demonstrate that they were going to follow the rules (that they would use reliable and valid measures, and have a sound process in place for selecting materials that met the criterion, and that they would work with experts who actually would implement what was required, rather than just taking the money and doing the same old thing that had not worked in the past). Unfortunately, those initial meetings made it look like there were favored commercial programs and the Senate questioned the Secretary of Education about whether some companies were being given an inside track.

Secretary Page gave the right answers and things went on, but the rumors and claims never went away. Part of this was due to the fact that Reading First legislation required such a close adherence to research that it offended lots of "experts" who felt it was limiting their influence over the schools. Another part of it was that various groups wanted the money and they didn't feel that they were getting their fair share. As a result, there were more investigations, including a particularly thorough one by the Inspector General.

That probe found wrong-doing, and the director of Reading First (Chris Daugherty) resigned under a cloud, and lots of changes have been made to how that program is administered. Even that has been controversial, offending Daugherty's supporters, who felt that he was supposed to require schools to purchase programs that were consistent with the law and that he should have been supported by Secretary Spellings.

Congress then made noises about ending the program, but by then various evaluations were starting to indicate that Reading First was being successful. That is, the kids in these schools were actually learning to read better than had been true in the past. The Reading First dollars were making a difference in learning (unlike the much bigger pot of Title I funds that had never really managed to have that effect on scale). It sure makes sense to administer programs legally and ethically, and not to allow anyone to get unfairly enriched with government largesse, but what about the kids?

It looked like Reading First was going to weather the storm. Democrats wanted to trim it a bit to send a message to the President that chicanery was not acceptable, and the program would continue given both its success and the diligence the Department of Education had shown in fixing the problems identified in the investigations.

Unfortunately, the budget battles in Washington, DC undermined those agreements. When the president wouldn't agree to huge increases in the federal education budget (he did agree to large increases), the Democrats slapped him with big cuts to his Reading First program.

The problem with that approach, of course, is that while Congress may have ticked off the Pres, more importantly they haved reduced the chances that America's poorest kids will learn to read. Man, I hope the President doesn't make Congress mad again; I'd hate to see what they will do to us in retaliation.

That is what Keillor, a prominent liberal, is reacting, too. He is no Bush fan, but recognizes that even infuriating, conservative, lame duck presidents can be right on something, and when they are, it is better to make sure kids learn to read than to vent Congressional spleen.

I think Keillor overplays the phonics part of the National Reading Panel report a bit (and I think, looking at the evaluations of Reading First, that it has as well), but ultimately he is right. Why should the effectiveness of phonics (or of any other aspect of an effective reading program) be ceded to any politician or party? Politicians should be working hard to make sure that schools have the money that they need--and that they spend that money in ways that actually helps kids. This time, for purely political motives, the politicians increased the flow of money that hasn't been helping kids to read better, and cut the money that has been helping. That's crazy.