1st major study of reform math: epic fail

In the August 2014 issue of Economics of Education Review:

Abstract

We investigate the impact of an ambitious provincial school reform in Canada on students’ mathematical achievements. It is the first paper to exploit a universal school reform of this magnitude to identify the causal effect of a widely supported teaching approach on students’ math scores. Our data set allows us to differentiate impacts according to the number of years of treatment and the timing of treatment. Using the changes-in-changes model, we find that the reform had negative effects on students’ scores at all points on the skills distribution and that the effects were larger the longer the exposure to the reform. [emphasis added]

[snip]

In this paper, we estimate the impact of Quebec’s (the second most populated province in Canada) ambitious and universal school reform implemented in the early 2000’s on children’s mathematical ability throughout primary and secondary school. At the time of the reform, the performance of students in the province of Quebec was comparable to that of students from the top performing countries in international assessments. Nonetheless, the educational system in Quebec was still subject to severe criticism at home due to its alarmingly large high school dropout rate, especially among male students.6 To ensure the success of all students, the province decided to implement an ambitious reform introducing a new program in each and every school across the province which drastically changed the way teaching was delivered to all children in primary and secondary schools. The Quebec education program (MELS, 2001, 2003, 2007) relied on a socio-constructivist teaching approach focused on problem-based and self-directed learning. [emphasis added] This approach mainly moved teaching away from the traditional/academic approaches of memorization, repetitions and activity books, to a much more comprehensive approach focused on learning in a contextual setting in which children are expected to find answers for themselves. [emphasis added]

. . . . More specifically, the teaching approach promoted by the Quebec reform is comparable to the reform-oriented teaching approach in the United States. As of 2006, this approach was widely spread across the United States (although more traditional approaches remained dominant) and it was supported by leading organizations such as the National Council of Teachers of Mathematics, the National Research Council, and the American Association for the Advancement of Science. Yet few studies in economics have addressed the impact of various teaching approaches, let alone the approach promoted by the Quebec reform.

[snip]

[The] approach was designed to enable students to ‘‘find answers to questions arising out of everyday experience, to develop a personal and social value system, and to adopt responsible and increasingly autonomous behaviors’’ (MELS, 2005).

In the classroom, students were expected to be more actively involved in their own learning and take responsibility for it. Critical to this aspect was the need to relate their learning activities to their prior knowledge and transfer their newly acquired knowledge to new situations in their daily lives. ‘‘Instead of passively listening to teachers, students will take in active, hands-on learning. They will spend more time working on projects, doing research and solving problems based on their areas of interest and their concerns. They will more often take part in workshops or team learning to develop a broad range of competencies.’’ (MELS, 1999). This centralized approach in providing the program and training with a school-based execution is in many ways comparable to the current approach taken within the comprehensive school reform (CSR) models at the national level in the United States (Borman et al., 2003). The main differences are that in Quebec, implementation was mandatory in each and every school, funding was not tied to the implementation, and training packages and support are centralized in many ways. These differences are critical: they imply that the reform had to be implemented in all schools, and that the resources and training was not tied to individual school characteristics. Whether private or public, English speaking or French speaking, all schools across the province were mandated to follow the reform according to the implementation schedule. This implies that all children in Quebec were treated according to same timeline, and that parents were not able to self-select their children into or out of the reform, except by moving out of the province which they did not.

The school reform was planned at the highest level by civil servants at the Department of Education (MELS). The MELS imposes the program to be followed in each grade by every school. The 69 School Boards (60 Francophone and 9 Anglophone) responsible for all public schools, their superintendents and the school principals, are the channels and drive belts between the MELS and school teachers and students.

[snip]

Conclusion

We find strong evidence of negative effects of the reform on the development of students’ mathematical abilities. More specifically, using the changes-in-changes estimator, we show that the impact of the reform increases with exposure, and that it impacts negatively students at all points on the skills distribution. . . . Students from the lower end of the distribution do not seem to be in a better position to successfully complete their schooling. Mathematical abilities are strongly related to school attainment and labor market outcomes, and for lower performing students they are at best equivalent post reform, but most likely lower.

The teaching approach dictated by the reform is based on socio-constructivism. According to Pinker (1997), proponents of this method believe that children must construct mathematical knowledge for themselves with the teacher only guiding the discussion on the topics and that repetitions and practice are seen as detrimental to learning. He argues that constructivism is not appropriate for mathematics. For him, ‘‘. . . without the practice that compiles a halting sequence of steps into a mental reflex, a learner will always be building mathematical structures out of the tiniest nuts and bolts’’. Certain skills for mathematics may be very difficult to ‘‘construct’’ at a young age and can possibly be better attained by old-fashioned practice and a more mechanical approach. Pinker suggests that the poor performance of the United States in mathematics could be linked to the teaching approach, which is mainly contextual with no teaching of mathematical concepts. The evidence presented in this paper supports this argument.

The distributional impacts of a universal school reform on mathematical achievements: A natural experiment from Canada by Catherine Haeck, Pierre Lefebvre *, Philip Merrigan

Well, well, well.

Contra Elizabeth Green (again), history does not fold itself meekly into a Bill Gates-approved narrative in which "the traditional approach we take to teaching math — the one that can be mind-numbing, but also comfortingly familiar — does not work."

Using the traditional approach, Quebec schools produced students whose achievement "was comparable to that of students from the top performing countries in international assessments."

Using constructivism, they produced students whose achievement suffered at every grade level, and at every skill level to boot. Good students did worse, bad students did worse, in-between students did worse. Everyone did worse in constructivist math.

Because constructivism doesn't work. 

As to the teachers, whom Green cites as the source of Reform Math failure, the article notes that "Extensive training was provided to support the new program."

Educational malpractice for the sake of Reform Math

A couple of weeks ago, James Milgram, an emeritus Professor of Mathematics at Stanford University, updated me on some recent developments in the controversy over Jo Boaler's "Railside Study." It was only after I reviewed the various critiques, accusations, and rebuttals that I remembered what an enormously consequential case of educational malpractice is afoot here--one that deserves much wider attention than it's gotten so far.

Professor Milgram is known in the education world for his comprehensive critique of a study done by Jo Boaler, an education professor at Stanford, and Megan Staples, then an education professor at Purdue. Boaler and Staples' paper, preprinted in 2005 and published in 2008, is entitled Transforming Students’ Lives through an Equitable Mathematics Approach: The Case of Railside School. Focusing on three California schools, it compares cohorts of students who used either a traditional algebra curriculum, or the Reform Math algebra curriculum The College Preparatory Mathematics (CPM). According to Boaler and Staple's paper, the Reform Math cohort achieved substantially greater mathematical success than the traditional math cohorts.

In early 2005 a high ranking official from the U.S. Department of Education asked Professor Milgram to evaluate Boaler and Staples' study. The reason for her request? She was concerned that, if Boaler and Staples' conclusions were correct, the U.S. department of education would be obliged, in Milgram's words, "to begin to reconsider much if not all of what they were doing in mathematics education." This would entail an even stronger push by the U.S. educational establishment to implement the Constructivist Reform Math curricula throughout K12 education.

Milgram's evaluation of Boaler and Staples' study resulted in a paper, co-authored with mathematician Wayne Bishop and statistician Paul Clopton, entitled A close examination of Jo Boaler's Railside Report. The paper was accepted for publication in peer-reviewed journal Education Next, but statements made to Milgram by some of his math education colleagues caused him to become concerned that the paper's publication would, in Milgram's words, make it "impossible for me to work with the community of math educators in this country"--involved as he then was in a number of other math education-related projects. Milgram instead posted the paper to his Stanford website.

This past October a bullet-point response to Milgram's paper, entitled "When Academic Disagreement Becomes Harassment and Persecution," appeared on Boaler's Stanford website. A month ago, Milgram posted his response and alerted me to it. I have his permission to share parts of it here.

Entitled Private Data - The Real Story: A Huge Problem with Education Research, this second paper reviews Milgram et al's earlier critiques and adds several compelling updates. Together, the two papers make a series of highly significant points, all of them backed up with transparent references to data of the sort that Boaler and Staple's own paper completely lacks.

Indeed, among Milgram et al's points is precisely this lack of transparency. Boaler and Staples refuse to divulge their data, in particular data regarding which schools they studied, claiming that agreements with the schools and FERPA (Family Educational Rights and Privacy Act) rules disallow this. But FERPA only involves protecting the school records of individual students; not those of whole schools. More importantly, refusals to divulge such data violate the federal Freedom of Information Act. Boaler's refusal also violates the policies of Stanford University, specifically its stated "commitment to openness in research" and its prohibitions of secrecy, "including limitations on publishability of results."

Second, Milgram et al's examination of the actual data, once they were able to track it down via California's education records, shows that it was distorted in multiple ways.

1. Boaler and Staple's chosen cohorts aren't comparable:

It appears, from state data, that the cohort at Railside [the pseudonym of the Reform Math school] was comprised of students in the top half of the class in mathematics. For Greendale, it appears that the students were grouped between the 35th and 70th percentiles, and that the students at Hilltop were grouped between the 40th and 80th percentiles. [Excerpted from Milgram; boldface mine]

2. Boaler and Staple's testing instruments are flawed:

Our analysis shows that they contain numerous mathematical errors, even more serious imprecisions, and also that the two most important post-tests were at least 3 years below their expected grade levels.  [Excerpted from Milgram; boldface mine]

3. The data comparing test scores on California's standardized tests (STAR) comes from a comparison of test scores from students not involved in Boaler and Staple's study:

The students in the cohorts Boaler was studying should have been in 11th grade, not ninth in 2003! So [this] is not data for the population studied in [Boaler and Staple's paper]. This 2003 ninth grade algebra data is the only time where the Railside students clearly outperformed the students at the other two schools during this period. There is a possibility that they picked the unique data that might strengthen their assertions, rather than make use of the data relevant to their treatment groups.   [Excerpted from Milgram; boldface mine]

4. The most relevant actual data yields the opposite conclusion about the Reform Math cohort's mathematical success relative that of the traditional math cohorts:

o The most telling data we find is that the mathematics remediation rate for the cohort of Railside students that Boaler was following who entered the California State University system was 61%
o This was much higher than the state average of 37%
o Greendale's remediation rate was 35% o and Hilltop's was 29%.

5. School officials at "Railside" report that the results of the reform math curriculum are even worse than Milgram et al had originally indicated:

A high official in the district where Railside is located called and updated me on the situation there in May, 2010. One of that person's remarks is especially relevant. It was stated that as bad as [Milgram et al's original paper] indicated the situation was at Railside, the school district's internal data actually showed it was even worse. Consequently, they had to step in and change the math curriculum at Railside to a more traditional approach.

Changing the curriculum seems to have had some effect. This year (2012) there was a very large (27 point) increase in Railside's API score and an even larger (28 point) increase for socioeconomically disadvantaged students, where the target had been 7 points in each case.

6. Boaler’s responses to Milgram et al provide no substantiated refutations of any of their key points

In response to comments on an article on Boaler's critique of Milgram, Boaler states:

"I see in some of the comments people criticizing me for not addressing the detailed criticisms from Milgram/Bishop. I am more than happy to this. [...] I will write my detailed response today and post it to my site."

However, as Milgram notes in his December paper:

As I write this, nearly two months have passed since Boaler's rebuttal was promised, but it has not appeared. Nor is it likely to. The basic reason is that there is every reason to believe [Milgram et al's paper] is not only accurate but, in fact, understates the situation at "Railside" from 2000 - 2005.

In a nutshell: under the mantle of purported FERPA protection, we have hidden and distorted data supporting a continued revolution in K12 math education--a revolution that actual data show to be resulting, among other things, in substantially increased mathematics remediation rates among college students. Ever lower mathematical preparedness; ever greater college debt. Just what our country needs.

Nor is Boaler's Reform Math-supporting "research" unique in its lack of transparency, in its lack of independent verification, and in its unwarranted impact on K12 math practices. As Milgram notes,

This seems to be a very common occurrence within education circles.

For example, the results of a number of papers with enormous effects on curriculum and teaching, such as [Diane Briars and Lauren Resnick's paper "Standards, assessments -- and what else? The essential elements of Standards-based school improvement"] and [J. Riordan and P. Noyce's paper, "The impact of two standards-based mathematics curricula on student achievement in Massachusetts"] have never been independently verified.

Yet, [Briars and Resnick's paper] was the only independent research that demonstrated significant positive results for the Everyday Math program for a number of years. During this period district curriculum developers relied on [Briars and Resnick's paper] to justify choosing the program, and, today, EM is used by almost 20% of our students. Likewise [Riordan and Noyce's paper] was the only research accepted by [the U.S. Department of Education's] What Works Clearinghouse in their initial reports that showed positive effects for the elementary school program ``Investigations in Number, Data, and Space,'' which today is used by almost 10% of our students.

As Milgram notes:

Between one quarter and 30% of our elementary school students is a huge data set. Consequently, if these programs were capable of significantly improving our K-12 student outcomes, we would surely have seen evidence by now.

And to pretend that such evidence exists when it doesn't is nothing short of educational malpractice.

The Day of Reckoning, brought to us from India

Together, the rise of Reform Math, the reduction in ability-based grouping and AP classes, the demise of the close reading and the analytical essay (see also this), and the growing rarity of instruction in the finer points of English grammar and sentence construction, have caused current and future American high school graduates to be decreasingly prepared for college. As more and more American college students display skills in math, writing, and reading comprehension that are way below expectations (ending up, even in some of the more selective colleges, in remedial math and writing classes), college admissions committees are increasingly looking abroad.

While much of the news about overseas applicants centers on China, with its thousands of Ivy League-aspiring applicants and their glossy, high-production value applications (and the growing suspicion that a fair amount of cheating is involved), it's India, I predict, that will bring to the American K12 education system the day of reckoning that we so desperately need it to have. First, unlike their Chinese counterparts, college applicants from India face no linguistic barriers; many speak and write a much more eloquent English than American (and even British) students do. Second, there are apparently tons of extremely well-qualified Indian applicants pinning their hopes on America's top colleges.

Indeed, as an October New York Times article inadvertently suggests, the Day of Reckoning may be close at hand:

Moulshri Mohan was an excellent student at one of the top private high schools in New Delhi. When she applied to colleges, she received scholarship offers of $20,000 from Dartmouth and $15,000 from Smith. Her pile of acceptance letters would have made any ambitious teenager smile: Cornell, Bryn Mawr, Duke, Wesleyan, Barnard and the University of Virginia.

But because of her 93.5 percent cumulative score on her final high school examinations, which are the sole criteria for admission to most colleges here, Ms. Mohan was rejected by the top colleges at Delhi University, better known as D.U., her family’s first choice and one of India’s top schools.
...

Ms. Mohan, 18, is now one of a surging number of Indian students attending American colleges and universities, as competition in India has grown formidable, even for the best students. With about half of India’s 1.2 billion people under the age of 25, and with the ranks of the middle class swelling, the country’s handful of highly selective universities are overwhelmed.

True, another reason--indeed, the only reason mentioned in the Times article--why American recruiters are seizing on this opportunity is because so many of the crème de la crème of overseas students are wealthy enough to pay full tuition, unlike many of their American counterparts. But it also helps that the K12 schools they attend aren't using Reform Math, aren't renouncing ability-based grouping, and aren't failing to provide college prep classes that are truly college preparatory. Indeed, if it were primarily her parents' pocket books that make Moulshri Mohan so attractive to Dartmouth and Smith, why are they offering her so many thousands of dollars of scholarship money?

So here are my dire predictions. In the next ten years, as the effects of Reform Math continue to percolate up the American school system, and as the number of highly qualified Indian students continues to outpace the numbers of spots at the best Indian universities, there will be a the growing displacement of American students by Indian students. Only then will a large enough proportion of the Powers that Be start realizing how urgent it is to enact actual education reform--reform, that is, that reverses the century's-long tide that has pushed our K12 schools further and further away from what's happening in the most successful school systems overseas.

(Cross-posted at Out In Left Field)

Investigations Math in action: crashing and burning with large numbers

A third grade girl attempts, unsuccessfully, to add several large numbers using an Investigations Math strategy. She then adds them successfully using traditional "stacking" (disallowed at school) in a fraction of the time the Investigations method took her:




Filmed and edited by a fellow concerned parent who is a specialist in math remediation.