kitchen table math, the sequel: peer review - con and pro

Sunday, May 1, 2011

peer review - con and pro

In the midst of a conversation about peer review (here and here), I happened onto a critique of peer review I though I'd post:
The US Supreme Court has recently been wrestling with the issues of the acceptability and reliability of scientific evidence. In its judgement in the case of Daubert versus Merrell Dow, the Court attempted to set guidelines for US judges to follow when listening to scientific experts. Whether or not findings had been published in a peer-reviewed journal provided one important criterion. But in a key caveat, the Court emphasized that peer review might sometimes be flawed and therefore this criterion was not unequivocal evidence of validity or otherwise. A recent analysis of peer review adds to this controversy by identifying an alarming lack of correlation between reviewers’ recommendations.

Something rotten at the core of science?
by David Horrobin
Peer review has its problems, some of which I became aware of several years ago, when Ed and I learned that autism researchers doing behavioral research were being defunded, apparently in favor of funding researchers studying the brain and biology.* Peer review was involved.

Nevertheless, I do want K-12 curricula to be peer reviewed by specialists in the fields being taught.

Here's a case of a history textbook that was given no peer review:
Another historical malpractice foisted upon American school children came to light in Virginia last week . Once again it comes down to whether the standards of history as a discipline mean anything in the context of elementary and secondary history education.  Few of us would trust our children’s dental care to a historian.  Nor do we assume that anyone who has written a book can write a math textbook, regardless of their educational credentials.  But too often history seems different, subject to lower standards and inadequate review.


The case at hand is straightforward.  Our Virginia: Past and Present  (Five Ponds Press, 2010) was approved by the Virginia Board of Education without a single historian involved in the review process. Fortunately an alert historian reviewing her daughter’s assignments noticed the glaring error: a statement that “thousands of Southern blacks fought in the Confederate ranks, including two black battalions under the command of Stonewall Jackson.”  It’s not true.  The reference to Jackson’s army is a total fabrication, and the broader reference to the Confederate army ignores the fact that slaves were forced into service and that there are no data available in any archive to document the statistic. 

So where did author Joy Masoff (not a historian) get her information?  From the Internet.  More specifically, from the web site of the Sons of the Confederate Veterans.  And even more specifically from a page that claims Frederick Douglass as the source for the statistic, but can’t even get his name spelled right.  The relevant quotation from Douglass is taken out of context, and there are no corroborating sources. 

Historical Malpractice and the Writing of Textbooks
By James R. Grossman Malpractice
October 25, 2010
The disciplines are disciplines: they are fields of study with established bodies of knowledge, rules of evidence, and modes of analysis.

A professional historian possesses knowledge and expertise the rest of us don't possess, and that is the knowledge we send our kids to school to acquire. At least, that is the knowledge I sent my children to school to acquire.

History textbooks should be vetted - or preferably written - by historians, math textbooks by mathematicians, and science textbooks by scientists.

That's not to say K-12 teachers should have no involvement; teachers are the people who can tell us whether a textbook is working with students. If a K-12 teacher writes a textbook that's vetted by disciplinary specialists - then great!

I think parents should have a vote on their children's textbooks and curricular materials, too, though in my dream world we wouldn't need to exercise it.

Rethinking Peer Review
The Editors of The New Atlantis, "Rethinking Peer Review," The New Atlantis, Number 13, Summer 2006, pp. 106-110.

* I no longer remember all the details, and my understanding of what was going on may have been wrong. However, I do recall accurately that behavioral researchers were losing funding at the time. I don't know whether that is true today.


Allison said...

I think your "math textbooks should be vetted by mathematicians" is not hitting the right target on two levels: first, most mathematicians don't understand enough of what happens in a classroom to understand what a textbook needs to do, or what is wrong with it. They haven't spent time thinking about what's appropriate for a child, what background a child has, etc. and they have no idea what a teacher knows and doesn't know. So if they don't, how can they make textbooks level appropriate?

Second, vetting is supposed to do what? Act as a veto? If a mathematician says "this is wrong", then what?

Here is a talk Wu gave a couple days ago:

Here are some of the main points as relate to the issue of mathematicians supporting school math:

"A university mathematician's typical reaction to school mathematics:
-- it is elementary,
-- therefore trivial,
-- therefore if we teach future teachers "real" mathematics the right way", they will understand this elementary stu ff.

This mistake was made in a disastrous way in the New Math era."

"[]the most serious missing component is the contributions of
truly competent mathematicians who want to improve school math education and possess the requisite knowledge of schools.

This knowledge is best illustrated by the recognition that since fractions in SM are taught to ten-year olds, its mathematical development
-- must be sensitive to their knowledge base and mathematical sophistication, and
-- must diff er signifi cantly from that in university algebra
It would be fair to say that we mathematicians have had a dismal
record in educating teachers.

Our eff orts have produced books that range from overly formal and inappropriate for use by teachers, to mathematically oversimplistic in trying too hard to be pedagogically correct."

Catherine Johnson said...

So what do you suggest?

Here's what I don't want.

I don't want kids taught from textbooks so bad that mathematicians are actively opposing them.

I want mathematicians to be part of math.

Allison said...

Read the slides. The suggestions are all long term, over time, about changing the culture. There's no more immediate way out of the hole we've dug. Educating parents and teachers and schools about common core would at least be a sort of "stop digging". So would choking off money.

Allison said...

Read the slides. The suggestions are all long term, over time, about changing the culture. There's no more immediate way out of the hole we've dug. Educating parents and teachers and schools about common core would at least be a sort of "stop digging". So would choking off money.