RtI: When Things Don't Work as You Expected

  • Response to Intervention
  • 09 November, 2015

          When I arose today I saw lots of Twitters and Facebook entries about a new U.S. Department of Education study. Then I started getting emails from folks in the schools and in the state departments of education. IES Study on RtI

          “What’s going on here?” was the common trope.
          Basically, the study looked at RtI programs in Grades 1 through 3. The reports say that RtI interventions were lowering reading achievement in Grade 1 and while the RtI interventions weren’t hurting the older kids, they weren’t helping them to read better.
          The idea of RtI is a good one, but the bureaucratization of it was predictable. You can go back and look at the Powerpoint on this topic that I posted years ago.
           I’m not claiming that I predicted the failure of RtI programs. Nevertheless, we should be surprised that research-based interventions aimed at struggling readers, with lots of assessment monitoring harmed rather than helped kids. But I’m not. 
          In fairness, this kind of thing can go either way: on the one hand the idea of giving kids targeted instruction generally should improve achievement… and yet, on the other hand, this assumption is based on the idea that schools will accurately identify the kids and the reading problems, will provide additional instruction aimed at helping these kids to catch up, will offer quality teaching of the needed skills (meaning that usually such teaching will have positive learning outcomes), and that being identified to participate in such an effort won’t cause damage in and of itself (if kids feel marked as poor readers that can become a self-fulfilling prophecy with 6-year-olds just trying to figure things out). 
          When RtI was a hot topic I used to argue, somewhat tongue-in-cheek, for a 9-tier model; the point was that a more flexible and powerful system was going to be needed to make a real learning difference. If the identification of student learning needs is sloppy, or the “Tier 2” reading instruction just replaces an equivalent amount of “Tier 1” teaching, or the quality and intensity of instruction are not there… why would anyone expect RtI to be any better than what it replaced? 
          Unfortunately, in a lot schools that I visit, RtI has just been a new bureaucratic system for getting kids into special education. Instead of giving kids a plethora of IQ and reading tests, seeking a discrepancy, now we find struggling readers, send them down the hall for part of their instructional day, and test the hell out of them with tests that can’t possibly identify whether growth/learning is taking place and moving them lockstep through “research-based” instructional programs. 
          In other words, the programs emphasize compliance rather than encouraging teachers to solve a problem.
          First, there is too much testing in RtI programs. These tests are not fine-grained enough to allow growth to be measured effectively more than 2-4 times per year (in some places I’m seeing the tests administered weekly, biweekly, and monthly, a real time waster. 
          Second, the tests are often not administered according to the standardized instructions (telling kids to read as fast as possible on a fluency test is stupid). 
          Third, skills tests are very useful, but they can only reveal information about skills performance. Teaching only what can be tested easily is a foolish way to attack reading problems. Definitely use these tests to determine whether to offer extra teaching in phonological awareness, phonics, and oral reading fluency. But kids need work on reading comprehension and language as well, and those are not easily monitored. I would argue for a steady dose of teaching in the areas that we cannot test easily, and a variable amount of teaching of those skills that we can monitor.
          Fourth, the Tier 2 instruction should increase the amount of teaching that kids get. If a youngster is low in fluency or decoding, he should get additional fluency or decoding instruction. That means students should get the entire allotment of Tier 1 reading instruction, and then should get an additional dose of teaching on top of that. 
          Fifth, it is a good idea to use programs that have worked elsewhere (“research based”). But that doesn’t mean the program will work for you. Teach that program like crazy with a lot of focus and intensity, just like in the schools/studies where it worked before—in fact, that’s likely why it worked elsewhere. Research-based doesn’t mean that it will work automatically; you have to make such programs work.
          Sixth, don’t put kids in an intervention and assume the problem is solved. The teacher should also beef up Tier 1 teaching, should steal extra instructional moments for these students in class, and should involve parents in their programs as well. What I’m suggesting is a full-court press aimed at making these struggling students successful—rather than a discrete, self-contained, narrow effort to improve things; Tier 2 interventions can be effective, but by themselves they can be a pretty thin strand for struggling readers to hang onto.
          I hope schools don’t drop RtI because of these research findings. But I also hope that they ramp up the quantity and quality of instruction to ensure that these efforts are successful.


See what others have to say about this topic.

EdEd Apr 06, 2017 09:18 PM

Great post Dr. Shanahan - I think we're largely in agreement here. I also hope schools realize that this article largely suggests a failure of RtI as implemented, not theoretically, and not its potential.

I also think you've given some great suggestions for improvement, and schools would be wise to follow them. We somewhat disagree on the testing element - I do see some schools that over-rely on (in my opinion) less effective assessments, such as MAP, but CBM obviously yields results more than 2-3 times per year, is quick & easy to administer, and pretty easy in terms of standardization.

I also disagree somewhat with your comment about skills testing, though not completely. First, skills testing - conceptually - isn't limited to beginning reading skills. Even advanced comprehension tasks are skills, and can be measured as such, though admittedly not as readily with existing, quickly administered assessments. So, the issue isn't so much an over-reliance on skills testing, but - if occurring - on over-reliance on beginning reading skills instruction.

That being said, I don't think our main goal with higher tiers of RtI should be to find "balance," but to find out exactly what students need, then deliver it. If students need beginning skills instruction, that's what they should receive. Where we agree is that we should make sure there is coverage of higher order reading skills in assessments used to plan Tier II+ instruction. To that extent, utilizing newer technology such as CBM comprehension measures, and math reasoning assessments, can be helpful. We also need to NOT just limit ourselves to CBM, but to see other, less frequently administered, skills assessment (e.g., classroom formative assessment) as fair game for informing instructional planning in higher tiers.

Sandie Stone Apr 06, 2017 09:19 PM

Dr. Shanahan thank you for this post. Now, if only those in charge would stop purchasing the latest and greatest, no fail, this is the one, Rti programs and actually let classroom teachers tailor Rti for students. Tell me why I should make my kindergarten student log 90 minutes of time per week on Istation for learning high frequency words. I don't teach using a look and repeat after me approach to reading, instead I use the 6 syllable types and Cued Articulation. My students learn to decode.


Timothy Shanahan Apr 06, 2017 09:20 PM

The issue on over-testing isn't one concerning the ease of administering CBMs, but the usefulness of the information to be provided. You have to look at the standard error of measurement of those tests. You will find they are fairly large when the tests are given exactly as standardized and even larger otherwise. It would be great to have a test that would reveal if kids were learning or not, but no test can do that if the intended amount of learning growth is smaller than the tests standard error. If a child improves 3 wcpm on an oral reading CBM, you can't even know if that is real growth is the SEM is 10, for example. We are testing too often because we are not paying attention to what these tests are capable of.

Comprehension is just too complex to provide a good quick assessment comparable to DIBELS skills tests. In fact, it doesn't operate like a skill, so the notion that it is just a different skill is not universally accepted, nor is it consistent with the best work on comprehension. Instead of piling the instruction up only where one can test easily, I think the convention should be to always provide some instruction attention to reading comprehension as well. You might provide more comprehension instruction than was absolutely needed, but that's fine by me. If you just follow the tests, then you will expend all the instruction on the skills and abilities that are easily tested. I suspect that is what is happening in far to many RtI programs.


Denise Kelly Apr 06, 2017 09:20 PM

Thank you again for a thought-provoking article! Being an impatient society as a whole, we seem to continue to try to find a quick-fix hurry up and make thinkers out of the students. True reflective and effective minds come from lots of modeling, discussing and asking the right questions by the adult. It is validating, exposure to rich texts and feedback at just the right timing and pacing. If we would put ourselves in the place of the child as we "do RTI" to them, rather than "with" them, we would see a powerful change in their ability to read, write and think.


Harriet Apr 06, 2017 09:21 PM

For a great example of why we should proceed with caution when it comes to intervention, read "WHY THE NEW ZEALAND NATIONAL LITERACY STRATEGY HAS FAILED AND WHAT CAN BE DONE ABOUT IT. Evidence from the Progress in International Reading Literacy Study (PIRLS) 2011 and Reading Recovery Monitoring Reports", Massey University Institute of Education, July 2013. It's quite an indictment of Reading Recovery.


EdEd Apr 06, 2017 09:22 PM

Dr. Shanahan - I agree that comprehension, as a whole, is more complex than can be fully captured by CBM, but I do think - like many things - that comprehension does involve a series of more discrete skills, even if there is a "whole" that's more than the sum of its parts. I think we can at least "get at" some of those skills through CBM, even if they aren't 1-minute probes. Still, to you point, I'm not one to suggest that DIBELS/CBM can be everything to everyone, so we should broaden our assessment base.

Second, and this gets back to your first point about using DIBELS/CBM to assess progress: you're right in that the SEM may be higher than weekly ROI scores, but ORF scores were never meant to be interpreted on a weekly basis by looking at one data point and comparing that single data point to baseline scores. ORF scores are meant to be analyzing as most interrupted time series data are - as trends (mean, slope, overlapping points, etc. - referencing Kazdin 1982/2010 here). This is no different from any other form of assessment in which single data points aren't meant to be analyzed in isolation, but are aggregated and analyzed as a sample (e.g., mean of data points).


Rob Ackerman Apr 06, 2017 09:22 PM

Dr Shanahan....it would seem that RTI, in general, has been overblown or teachers simply are not effective reading instructors. The data is pretty clear on the lack of progress for our country on reading achievement.

My question is...how can we help teachers to teach for comprehension? Are we not asking our kids to engage deeply with the texts? Do students not recognize "what you reads defines how you read?"

Decoding Dyslexia RI Apr 06, 2017 09:23 PM

Thank you for your article. I have long had a problem with RTI. There should be A specific amount of time should be allotted for trying A specific program, which after 6 weeks it should be clear if it is working or not. I do feel it is important to test before and after the 6 weeks. As a parent with a dyslexic child I know Reading Recovery does not help at all. I had to argue and provide research to the so-called experts, that this was an inappropriate technique. Further they kept my child, and other children who struggle with literacy for not just one year or two years before they consider the parents assertion that their child appears to have a potential learning disability. It is educational malpractice, & a clear injustice 200, families and society in such a way. RT I should go no longer then 3 months, whereupon they should do a test and if progress is not being made then they should give an educational test to test for suspected learning disability. I would like to look for your PowerPoint and share with others. Thank you for your great assessment


Timothy Shanahan Apr 06, 2017 09:23 PM

Decoding Dyslexia--

Thanks for your letter. I'm sorry to hear about your child's difficulty. On the basis of the research, I disagree with you about Reading Recovery. It has been found to be an effective, but expensive, approach to teaching reading. However, that does not mean that program (or any other research-based program) necessarily works with everybody (in fact, the research on RR shows that as well--that's why so many kids are discontinued). To know if a program is "working" or not takes longer than 6 weeks. Children's normal growth rates are too slow and current tests are too unreliable to determine with any reliability whether a child is learning in that time from a program. Three months is a more reasonable time frame. You are correct that RtI would delay identifying kids as learning disabled--it takes longer to see if a child is responding to teaching than to just test him/her.


Timothy Shanahan Apr 06, 2017 09:24 PM


I'm sorry to say that far too few students are asked to engage deeply with texts. Reading comprehension instruction is often superficial... often kids are asked to read texts not worth reading and what is done with those texts may not focus much on meaning beyond getting the surface meaning. There is a lot of comprehension skills instruction--which means the reading routines are not fit to the texts, but are seen as transferrable to all texts. There is not a big emphasis on developing knowledge from reading and there is not a big emphasis on teaching students to handle the language demands of text. I've been conducting a study of nearly 1000 classrooms (preK-grade 3) nationwide and it is surprising to me how rare these things are.



Harriet Apr 06, 2017 09:24 PM

Here's a link to a letter by over 30 researchers outlining the weaknesses of Reading Recovery.

"In this open letter, more than 30 international reading researchers expressed concerns about the continued use of Reading Recovery. These experts urged policy makers, educational leaders, researchers, and federal research organizations to acknowledge the weaknesses of Reading Recovery.

They concluded, "Reading Recovery leaves too many students behind." - See more at: http://www.wrightslaw.com/info/read.rr.ltr.experts.htm#sthash.cAPhT6C9.dpuf


Timothy Shanahan Apr 06, 2017 09:25 PM


And many of the signers of that document are dear friends... and, yet, I will always believe the research rather than the researchers. The research that I have formally and publicly reviewed (I was the one who first identified many of the problems that document complains about) and that the What Works Clearinghouse has reviewed. Again, Reading Recovery clearly delivers a learning advantage to kids; but it is an expensive approach to accomplishing that and it doesn't meet the needs of all young struggling readers. Like my colleagues who signed onto that document, I believe that more explicit teaching of decoding will move kids along more efficiently and powerfully than the ways RR has handled that (Marie Clay originally created RR to address the needs of kids who were struggling despite receiving daily explicit decoding instruction); it doesn't seem to work as effectively when it isn't addressing a gap in kids' learning experience. In situations where kids aren't getting sufficient amounts of quality decoding instruction, RR should be adjusting to meet that need.


Harriet Apr 06, 2017 09:25 PM

Yes, I think your point about "quality decoding instruction" goes to the heart of the matter. Our district has just purchased Leveled Literacy Intervention (LLI)by Fountas and Pinnell, and I recently went through two days of training. Fountas and Pinnell base much of their work on the work of Marie Clay, and in this training it was clear to me exactly how both LLI and Reading Recovery "leave too many students behind". There is far too much emphasis on guessing and not enough on explicitly teaching students how to decode so that as the years progress they can attack all words, including multi-syllabic ones. We know that our weakest readers are the ones who guess, so this isn't a strategy we should be promoting. I applaud the emphasis that both LLI and Reading Recovery have on reading connected text rather than doing "drill and kill" with isolated phonics worksheets, but I can see why New Zealand has not had the results Reading Recovery promised because the program doesn't address the needs of the seriously struggling reader, the one who simply can't read the words. Not to beat a dead horse, but this is why I only focused on sounds in my kindergarten class--I know how crucial it is that kids learn to decode quickly and efficiently and I don't want to do anything to impede that progress. Thanks for your reply! 11/12/15

Harriet Apr 06, 2017 09:26 PM

Dr. Shanahan, have you had a chance to review the research cited in WHY THE NEW ZEALAND NATIONAL LITERACY STRATEGY HAS FAILED AND WHAT CAN BE DONE ABOUT IT. Evidence from the Progress in International Reading Literacy Study (PIRLS) 2011 and Reading Recovery Monitoring Reports, Massey University Institute of Education, July 2013. If so, I would appreciate your thoughts, which may be similar to your response to the open letter. You can imagine how frustrating it is for those of us in the trenches to receive messages from dueling researchers because we really are trying to use best practices based on the current research, but so often we rely on what researchers tell us because we simply don't have time to examine the research for ourselves. Thank you!


EdEd Apr 06, 2017 09:26 PM

Dr. Shanahan - As always thanks for your willingness to engage. First, I'd love a link or other direction to a good text/source of info for current thoughts on reading comprehension in the field along the lines of what you're discussing, particularly when you say that there is no measurable difference in performance across question types - when you say this, are you saying that research suggests that kids tend to do equally bad or good across all question types - from fact recall ("where did the cat go after leaving the house") to higher level inferential questions? If so, you're suggesting that the power or engine that enables kids to answer ALL forms of comprehension questions comes from a more singular, whole understanding of the text?

My experience with comprehension is that it can be broken down into different skill categories, and can't be summed into a singular whole. My experience, and understanding, is that comprehension involves everything from activation of working memory to language comprehension, background knowledge, text engagement (e.g., note-taking, rereading), etc. The target of comprehension could also vary widely, from "comprehending" basic facts of a text to being able to utilize information from the text in external applications. "Comprehension" as a term is sort of like the term "learning" - it means different things depending on the task demand.

So, my sense is that we may be both right - that comprehension can't be fully captured by an assessment of tasks such as factual sequencing and identification of main idea, but that comprehension does involve definable, observable tasks on at least a basic level. Because of that, we can (and should) attempt to get at the skills we can, while certainly continuing to use evidence-based instruction to teach broader comprehension.

Still, would love links to comprehension research you're referencing - always open to changing my mind.


Mary Apr 06, 2017 09:27 PM

Great Post! I was part of the four original RTI projects where we learned that in urban poverty schools, by 2nd grade more than 50% of students are already Tier 3. Research indicates students that far behind grade level need intense intervention designed for struggling readers. Many RTI educators decided to do kinder "roll ups." Older students were left in the dust. We learned that an RTI paradigm change required the type of explicit language, spelling, vocabulary, & comprehension instruction that Anita Archer writes about. Unless RTI is a dramatic paradigm change, where Tier 3 students are NOT put in humiliating situations reading text at a frustration level, there is no hope of change.
District politics & reading directors worked behind the scenes to sabotage explicit phonics-based RTI projects. Some of the first RTI programs ended as researchers struggled in large districts with reading wars politics. After our RTI kinder students outscored other district students except on the one test Illinois used to test guessing, a district reading director yelled at me in the hall, "This is the saddest thing I've ever seen-how will your kids ever read if they aren't good guessers?" Our kinders, rather than read predictable sentences such as""This is an elephant" tried to sound out the words for the animals. Since it's so difficult to break older students' guessing, we were glad that our kids were using decoding skills. I assured her by the end of 1st grade all of those words would not only be decodable, but would be processed as whole words or word parts.
Originally, we weren't aware that "more isn't always better" but Tier 3 readers didn't go back to classes to receive instruction in Reading Recovery with guessing. However, when the progress stopped of our 2nd grade Tier 3 readers who had been in Tier 3 since kindergarten but were on track to finish the year at a high Tier 2 level, we stayed in that school all day every day to figure out what was happening. Had reading become so difficult they needed more support? On the third day, we learned that a new after school teacher was having all kids read leveled books. These kids still hadn't learned all their phonics skills &resorted to guessing. The guessing bled into reading at other times. We got the after school teacher to read the books & teach vocabulary and comprehension. Then student scores began to increase & they finished the year at a high Tier 2 level. A year later many were Tier 1 readers. Once these students were in 3rd grade, they were ready for the leveled reading that had led to the 2nd grade guessing.
What is critical for a successful RTI program? *An intensive, explicit 90 minutes a day Tier 3 curriculum with proven results for "turn around" for everyone who tests at Tier 3 *A Principal who learns the program, pushes reluctant teachers & honors teachers getting results. *NO FRUSTRATION LEVEL READING where students are humiliated & guess. *Explicit vocabulary, spelling, and comprehension *Mastery, not exposure with core knowledge concepts needed to comprehend higher level text. *A reading coach *NEVER cancelling or reducing reading time. More is related in "Teaching Reading to Students Who Are At Risk or Have Disabilities: A Multi-Tier, RTI Approach." Although our projects were in elem schools, some of the YES PREP charter schools in Houston have successfully used this type of RTI model in 6th-8th grade classrooms demonstrating that it is never too late for struggling readers.


Timothy Shanahan Apr 06, 2017 09:27 PM


Happy to hear things are going well for your students. However, it is important to think about what it means that an RtI program is "working." There are several that are appropriate: (1) since there would be greater attention and emphasis to addressing and preventing reading problems in the classroom--Tier 1--then there should be higher average reading achievement for the schools; (2) since struggling readers would not just be shuttled into Tier 3/Special Education classes, they would receive Tier 2 remedial teaching--students who receive such support would do better than similar kids who are not treated/taught with the Tier 2 intervention, and fewer students should end up in Special Education. Finally, (3) the students who receive Special Education/Tier 3 teaching would do better than such students have in past years (special education kids have usually not done particularly well). To say something worked would require convincing data that 1, 2, or 3 of those goals were actually accomplished beyond chance.


Timothy Shanahan Apr 06, 2017 09:28 PM


I definitely understand that it is confusing to see that a program works--that is confers a learning advantage on children who are taught with it... and that the same program either overstates its effectiveness (as RR has done since the beginning) or that its use has not led to the overall outcomes intended (such as that RR may have good success with the kids who are taught with it, but that it is not impacting overall school literacy rates in any meaningful way). RR works... kids who receive such instruction do better than control group subjects who do not... however, RR does not work as well as its proponents claim, since they have often failed to consider the performance of students who were dropped from the program for not doing well... etc


What Are your thoughts?

Leave me a comment and I would like to have a discussion with you!

Comment *

RtI: When Things Don't Work as You Expected


One of the world’s premier literacy educators.

He studies reading and writing across all ages and abilities. Feel free to contact him.