Interim Report: "Reading First Ineffective" -- But Hold Your Horses

cross-posted at I Speak of Dreams (slightly different version)

On Monday, April 28, 2008, the Institute of Education Sciences (IES) released a study, the Reading First Impact Study: Interim Report. I haven't yet studied in depth the full copy of the study (it will be available for order from Recently Added Publications -- Publication ID: ED004243P. (Reading First is hereafter abbreviated RF). Links to the report can be found at the IES site: http://ies.ed.gov/ncee/pubs/20084016/index.asp

A few preliminary comments: I see the "map territory" problem here.

The headline (in various iterations) is: Study: Bush administration's reading program hasn't helped.

For those new to the story, some background: Reading First:

Reading First provides assistance to States and districts to establish research-based reading programs for students in kindergarten through grade three. Reading First also provides funds to train teachers, including special education teachers, in the essential components of reading (phonemic awareness, phonics, fluency, vocabulary, comprehension) and to select and administer assessments to identify those children who may be at risk of reading failure.

In other words, state educational units (SEAs) applied for grants, and then local education agencies (LEAs) applied for subgrants. The whole kit and caboodle is known as "Reading First". Note that RF was a funding program with limits, not a prescription for actual classroom teaching.

The study referenced above looked at a sample of LEAs executing programs funded by RF.

Mike Petrelli, at the Fordham Institute's , cautions: Read the Report First.

First, none of the states that won the first Reading First grants could participate in the study because their programs got started in advance of the evaluation. These states were the ones most enthusiastic about the program–and most prepared to implement it well. It’s quite likely that Reading First schools in these states are having a major impact.

Second, the schools selected for study were the ones that just barely won grants under the program, which were compared to schools that just barely missed funding. (Schools are ranked according to various criteria, such as poverty, need, etc. Let’s say there was enough money in a given district to fund 10 schools; then the study compared the 10th-ranked school, which got money under the program, to the 11th-ranked school, which did not.) But here’s the rub: the schools where you would expect the greatest impacts from Reading First are the poorest ones, enrolling students who are further behind in reading–schools that would have been ranked at the top of the priority list. Simply put, these schools weren’t included in the study.

The bottom line is that evaluators looked for schools that met their study design conditions, not schools that were nationally representative of the program. So we can’t say anything definitive about the effectiveness of Reading First–all we know is about the effectiveness (or lack thereof) of a handful of Reading First schools.

Jay P. Greene, in the comments:

The thing that strikes me most about this study is that the time spent on phonics increased only marginally when schools adopted Reading First (see p. 44, where the increase is described as an extra 2-5 minutes per day). Perhaps the increase was so small because the implementation was poor, which would be consistent with Mike’s explanation. Or perhaps it was because Reading First is not a well-designed phonics program — so this would be an evaluation of Reading First, not the concept of phonics. Or perhaps the increase in time spent on phonics was small because the emphasis on phonics has already become pretty wide-spread, even in non-Reading First schools. If this last option is the case, then whetever benefit we can get from shifting to more phonics has already largely been achieved.

What I want to know, which the study may or may not answer:

1. Have any schools/districts using RF experienced comprehension gains?
2. If yes, what are the differences between schools/districts using RF successfully and those districts who did not see gains? Non-exhaustive list of differences:
1. in demographics (especially student transiency)
2. in teacher preparation and support (hours of teacher training in phonics and phonics instruction, for example)
   
3. in duration and intensity of instruction)


3. I want to see a fine-grained analysis of "fidelity", for example: : were the teachers, schools and districts really teaching systematic and explicit phonics with adequate follow-on practice with reading materials repeating the graphophonemic elements just taught? Or was it a case of buying the materials and not using them?
   

This District Has Found RF Works

In an article published in Broome County (NY) found that RF had a positive effect:

Locally, the program has provided money for teacher training and additional instructional time and intervention for students below grade level, they said. Early results show some improvement in students' basic skills, although the longer-term impact on reading comprehension remains to be seen.

"My immediate reaction is that we're extremely grateful for the opportunities Reading First has afforded us," said Suzanne McLeod, assistant superintendent for business and elementary education in the Union-Endicott Central School District

USA Today story as illustrative of the "RF doesn't work" in the national media

Reading First has had a complicated history. A good summary of the back story was written by Sol Sternberg at the Thomas F. Fordham Institute: Too Good to Last: The True Story of Reading First

Sample comments from those opposed to Reading First:

Nick Burubles, Education Policy Blog:

I think everyone here knows that the “Reading First” program is just another Bush patronage scam, using NCLB rules to funnel money to campaign supporters and loyalists. Now the Institute of Education Sciences – the unit that says all policy must be based on rigorous scientific evidence – concludes that Reading First is a lousy program. Okay, now there’s scientific evidence: so what’s the response?

Jackie Bennett, Edwize:

So, surprise, surprise: more and more time for reading skills, and no significant results. I don’t know if future studies will bear out these findings, but if they do, isn’t this exactly what many of us have suspected all along? Better reading doesn’t happen in a vacuum. Kids need a diverse curriculum that will give them background knowledge, and any gains from skill drills could be irrelevant after the early grades. As a teacher said to me earlier this week, the teaching of discreet reading skills is “more gimmick than education and more test prep than reading.” Some skill work may have its place in reading instruction, but across the country it has come to replace the instruction kids really need, such as instruction about history and the world around them, and about science and the arts.

Sometimes I get really discouraged. On another discussion list I'm on (sorry, not public) one parent finally succeeded in getting a specific reading remediation program listed in her child's IEP, with a prescription for 90 minutes a week (the recommended minimum, at 3 times a week for 30 minutes). She has since found out that her child is engaged in the reading remediation program once a week for 30 minutes, and that "it isn't working" and "should be discontinued".