peer review - con and pro

In the midst of a conversation about peer review (here and here), I happened onto a critique of peer review I though I'd post:

ABSTRACT
The US Supreme Court has recently been wrestling with the issues of the acceptability and reliability of scientific evidence. In its judgement in the case of Daubert versus Merrell Dow, the Court attempted to set guidelines for US judges to follow when listening to scientific experts. Whether or not findings had been published in a peer-reviewed journal provided one important criterion. But in a key caveat, the Court emphasized that peer review might sometimes be flawed and therefore this criterion was not unequivocal evidence of validity or otherwise. A recent analysis of peer review adds to this controversy by identifying an alarming lack of correlation between reviewers’ recommendations.

Something rotten at the core of science?
by David Horrobin

Peer review has its problems, some of which I became aware of several years ago, when Ed and I learned that autism researchers doing behavioral research were being defunded, apparently in favor of funding researchers studying the brain and biology.^* Peer review was involved.

Nevertheless, I do want K-12 curricula to be peer reviewed by specialists in the fields being taught.

Here's a case of a history textbook that was given no peer review:

Another historical malpractice foisted upon American school children came to light in Virginia last week . Once again it comes down to whether the standards of history as a discipline mean anything in the context of elementary and secondary history education.  Few of us would trust our children’s dental care to a historian.  Nor do we assume that anyone who has written a book can write a math textbook, regardless of their educational credentials.  But too often history seems different, subject to lower standards and inadequate review.

[snip]

The case at hand is straightforward.  Our Virginia: Past and Present  (Five Ponds Press, 2010) was approved by the Virginia Board of Education without a single historian involved in the review process. Fortunately an alert historian reviewing her daughter’s assignments noticed the glaring error: a statement that “thousands of Southern blacks fought in the Confederate ranks, including two black battalions under the command of Stonewall Jackson.”  It’s not true.  The reference to Jackson’s army is a total fabrication, and the broader reference to the Confederate army ignores the fact that slaves were forced into service and that there are no data available in any archive to document the statistic. 

So where did author Joy Masoff (not a historian) get her information?  From the Internet.  More specifically, from the web site of the Sons of the Confederate Veterans.  And even more specifically from a page that claims Frederick Douglass as the source for the statistic, but can’t even get his name spelled right.  The relevant quotation from Douglass is taken out of context, and there are no corroborating sources. 

Historical Malpractice and the Writing of Textbooks
By James R. Grossman Malpractice
October 25, 2010

The disciplines are disciplines: they are fields of study with established bodies of knowledge, rules of evidence, and modes of analysis.

A professional historian possesses knowledge and expertise the rest of us don't possess, and that is the knowledge we send our kids to school to acquire. At least, that is the knowledge I sent my children to school to acquire.

History textbooks should be vetted - or preferably written - by historians, math textbooks by mathematicians, and science textbooks by scientists.

That's not to say K-12 teachers should have no involvement; teachers are the people who can tell us whether a textbook is working with students. If a K-12 teacher writes a textbook that's vetted by disciplinary specialists - then great!

I think parents should have a vote on their children's textbooks and curricular materials, too, though in my dream world we wouldn't need to exercise it.



Rethinking Peer Review
The Editors of The New Atlantis, "Rethinking Peer Review," The New Atlantis, Number 13, Summer 2006, pp. 106-110.

^* I no longer remember all the details, and my understanding of what was going on may have been wrong. However, I do recall accurately that behavioral researchers were losing funding at the time. I don't know whether that is true today.