kitchen table math, the sequel: 1st major study of reform math: epic fail

Sunday, August 10, 2014

1st major study of reform math: epic fail

In the August 2014 issue of Economics of Education Review:

We investigate the impact of an ambitious provincial school reform in Canada on students’ mathematical achievements. It is the first paper to exploit a universal school reform of this magnitude to identify the causal effect of a widely supported teaching approach on students’ math scores. Our data set allows us to differentiate impacts according to the number of years of treatment and the timing of treatment. Using the changes-in-changes model, we find that the reform had negative effects on students’ scores at all points on the skills distribution and that the effects were larger the longer the exposure to the reform. [emphasis added]


In this paper, we estimate the impact of Quebec’s (the second most populated province in Canada) ambitious and universal school reform implemented in the early 2000’s on children’s mathematical ability throughout primary and secondary school. At the time of the reform, the performance of students in the province of Quebec was comparable to that of students from the top performing countries in international assessments. Nonetheless, the educational system in Quebec was still subject to severe criticism at home due to its alarmingly large high school dropout rate, especially among male students.6 To ensure the success of all students, the province decided to implement an ambitious reform introducing a new program in each and every school across the province which drastically changed the way teaching was delivered to all children in primary and secondary schools. The Quebec education program (MELS, 2001, 2003, 2007) relied on a socio-constructivist teaching approach focused on problem-based and self-directed learning. [emphasis added] This approach mainly moved teaching away from the traditional/academic approaches of memorization, repetitions and activity books, to a much more comprehensive approach focused on learning in a contextual setting in which children are expected to find answers for themselves. [emphasis added]

. . . . More specifically, the teaching approach promoted by the Quebec reform is comparable to the reform-oriented teaching approach in the United States. As of 2006, this approach was widely spread across the United States (although more traditional approaches remained dominant) and it was supported by leading organizations such as the National Council of Teachers of Mathematics, the National Research Council, and the American Association for the Advancement of Science. Yet few studies in economics have addressed the impact of various teaching approaches, let alone the approach promoted by the Quebec reform.


[The] approach was designed to enable students to ‘‘find answers to questions arising out of everyday experience, to develop a personal and social value system, and to adopt responsible and increasingly autonomous behaviors’’ (MELS, 2005).

In the classroom, students were expected to be more actively involved in their own learning and take responsibility for it. Critical to this aspect was the need to relate their learning activities to their prior knowledge and transfer their newly acquired knowledge to new situations in their daily lives. ‘‘Instead of passively listening to teachers, students will take in active, hands-on learning. They will spend more time working on projects, doing research and solving problems based on their areas of interest and their concerns. They will more often take part in workshops or team learning to develop a broad range of competencies.’’ (MELS, 1999). This centralized approach in providing the program and training with a school-based execution is in many ways comparable to the current approach taken within the comprehensive school reform (CSR) models at the national level in the United States (Borman et al., 2003). The main differences are that in Quebec, implementation was mandatory in each and every school, funding was not tied to the implementation, and training packages and support are centralized in many ways. These differences are critical: they imply that the reform had to be implemented in all schools, and that the resources and training was not tied to individual school characteristics. Whether private or public, English speaking or French speaking, all schools across the province were mandated to follow the reform according to the implementation schedule. This implies that all children in Quebec were treated according to same timeline, and that parents were not able to self-select their children into or out of the reform, except by moving out of the province which they did not.

The school reform was planned at the highest level by civil servants at the Department of Education (MELS). The MELS imposes the program to be followed in each grade by every school. The 69 School Boards (60 Francophone and 9 Anglophone) responsible for all public schools, their superintendents and the school principals, are the channels and drive belts between the MELS and school teachers and students.



We find strong evidence of negative effects of the reform on the development of students’ mathematical abilities. More specifically, using the changes-in-changes estimator, we show that the impact of the reform increases with exposure, and that it impacts negatively students at all points on the skills distribution. . . . Students from the lower end of the distribution do not seem to be in a better position to successfully complete their schooling. Mathematical abilities are strongly related to school attainment and labor market outcomes, and for lower performing students they are at best equivalent post reform, but most likely lower.

The teaching approach dictated by the reform is based on socio-constructivism. According to Pinker (1997), proponents of this method believe that children must construct mathematical knowledge for themselves with the teacher only guiding the discussion on the topics and that repetitions and practice are seen as detrimental to learning. He argues that constructivism is not appropriate for mathematics. For him, ‘‘. . . without the practice that compiles a halting sequence of steps into a mental reflex, a learner will always be building mathematical structures out of the tiniest nuts and bolts’’. Certain skills for mathematics may be very difficult to ‘‘construct’’ at a young age and can possibly be better attained by old-fashioned practice and a more mechanical approach. Pinker suggests that the poor performance of the United States in mathematics could be linked to the teaching approach, which is mainly contextual with no teaching of mathematical concepts. The evidence presented in this paper supports this argument.

The distributional impacts of a universal school reform on mathematical achievements: A natural experiment from Canada by Catherine Haeck, Pierre Lefebvre *, Philip Merrigan
Well, well, well.

Contra Elizabeth Green (again), history does not fold itself meekly into a Bill Gates-approved narrative in which "the traditional approach we take to teaching math — the one that can be mind-numbing, but also comfortingly familiardoes not work."

Using the traditional approach, Quebec schools produced students whose achievement "was comparable to that of students from the top performing countries in international assessments."

Using constructivism, they produced students whose achievement suffered at every grade level, and at every skill level to boot. Good students did worse, bad students did worse, in-between students did worse. Everyone did worse in constructivist math.

Because constructivism doesn't work. 

As to the teachers, whom Green cites as the source of Reform Math failure, the article notes that "Extensive training was provided to support the new program."


Anonymous said...

This randomized controlled trial from Costa Rica came to a similar conclusion:

Anonymous said...

"Good students did worse, bad students did worse, in-between students did worse. Everyone did worse in constructivist math."

So the reform was a complete success.

Anonymous said...

The study once again shows that somebody couldn't do it right.


Barry Garelick said...

Using the Green method of analysis, the Quebec study is invalid because it is judging methods that are not true reform.

Anonymous said...

The various "progressive"/constructivist "reforms" are like socialism; both have failed wherever and whenever they've been tried. The defenders of both use the same arguments; it wasn't the "real" thing and/or it wasn't done properly. Wash, rinse, repeat.

Kai Musing said...

Interesting stuff. I've been reading what you, Barry, and Harry Webb have been writing.

I have to agree with you on the constructivist/traditional stuff. I moved from a charter to a major school district and was criticized by both my administrators for not doing enough "real-world/exploration/struggle" in the math I was trying to teach kids. I even spent an hour in PD listening to a State inspector talk about how much better conceptual understanding was than procedural fluency (she even expressed the opinion that she'd rather have understanding than fluency - not both).

Also, I was reading an interview with Jason Zimba (one of the guys behind CCSS Math) and I wondered what you would think of it. Here's the link:

Most interesting part to me (paragraph 10):

RH: What are one or two things that give you the most pause when it comes to the standards?


ZImba: "But I sometimes worry that talking about the practice standards can be a way to avoid talking about focus and specific math content. Until we see fewer topics and a strong focus on arithmetic in elementary grades, we really aren't seeing the standards being implemented."


Kai Musing said...

Just realized that article was over a year old. You may have talked about something like it already... Sorry.

C T said...

The comparison by Anon at 11:18 to socialism is interesting because there are situations where math reforms appear to work (classes taught by extremely skilled teachers, small studies with students that might not be representative of all students, etc.) and socialism appears to work (Scandinavia). It's unfair (and unhelpful) to backers of either reform or socialism to talk as though their positions are entirely without support. Instead we can show them the evidence against reform math like this study out of Quebec and point out the weaknesses of any evidence used by the reform advocates, just as with socialism we can point to stats showing the low levels of poverty of Scandinavian-Americans, indicating that Scandinavian socialism "works" because Scandinavians work (
Reform advocates have often staked their careers on this issue, and unless they are superhuman, it will take rather gentle, but persistent, treatment to convince them to look at the evidence against their cherished ideas.

Auntie Ann said...

"A socio-constructivist teaching approach"


momof4 said...

A number of "non-representative students" are likely to be from families who provide significant academic opportunities at home and/or outside tutoring (and about which the local schools do not want to hear). Such kids are the most likely to survive/thrive in most school situations because they are not dependent on the schools for all of their academic needs (like low-SES kids usually do).

Auntie Ann said...

The teachers at our kids school have bought into constructivism in a big way, and at the same time talk about how researched-based it is. Yet, when I go looking for articles to support their position, I find little that is convincing. You find short-term studies, poorly-designed studies, studies with a handful of students, studies which don't apply to early grades (one of the few I've seen is that high school and college science classes work well when they use hands-on techniques--considering such classes have long had a lab component, this is neither important nor breathtaking.)

Since I doubt anyone at the school is actually reading journal articles, I assume they are getting this through their PD programs and conferences. In the past we've handed them a stack of journal articles supporting our position (on redshirting in kindergarten) and slowly, they figured out we were right (though it was too late for our kid). I'd love to do the same thing here, but the other side seems overwhelming and dominant across all of ed these days. When everything and everyone in their world is telling them one thing, it's hard to convince them of the opposite.

Anonymous said...

This just ticks me off, especially since the whole thing was such a failure: "Whether private or public, English speaking or French speaking, all schools across the province were mandated to follow the reform according to the implementation schedule. This implies that all children in Quebec were treated according to same timeline, and that parents were not able to self-select their children into or out of the reform, except by moving out of the province which they did not."


Michael Goldenberg said...

Is there a reality in which we stop entangling what comprised reform mathematics teaching in the Math Wars Era (say, 1989 to 2008, or maybe 2001, depending on how one interprets the advent of GWB or BHO as the line of demarcation for the beginning of sky completely falling [sarcasm alert]) with the Common Core State Standards?

Or are we now doomed to be so sloppy as to what we're really talking about that those of us who generally aren't on the anti-NCTM side but are generally critical of Common Core as a political initiative (even though we don't for one second swallow the Charlotte Iserbyt/Beverly Eakman variety of right-wing hysteria that claims that Common Core is an Islamist/Communist plot by Barack Hussein Obama to suck the very brains out of our children; instead, we recognize it for what it is: a non-partisan move by Big Publishing and other business interests (including various holdings of Bill Gates, the Walton Family, and many other billionaire foundations, think tanks, ad nauseam that may or may not have philosophical and/or religious axes to grind, but which are almost entirely about - surprise! - profit) have almost no chance of getting across to the non-insane part of the American public what matters in the world of mathematics education? That would include recognizing the limitations of every major approach and/or textbook series or online miracle (e.g., Khan Academy) to teaching mathematics and being able to engage in relatively calm discussion about teaching and learning math, without the axes folks who frequent this website and others like bring to everything.

What a better country this would be if it were possible to have such conversations. Of course, it is, if those who want to have them ban all the lunatics from the table. But that's a bit undemocratic.