peer review - con and pro

In the midst of a conversation about peer review (here and here), I happened onto a critique of peer review I though I'd post:

ABSTRACT
The US Supreme Court has recently been wrestling with the issues of the acceptability and reliability of scientific evidence. In its judgement in the case of Daubert versus Merrell Dow, the Court attempted to set guidelines for US judges to follow when listening to scientific experts. Whether or not findings had been published in a peer-reviewed journal provided one important criterion. But in a key caveat, the Court emphasized that peer review might sometimes be flawed and therefore this criterion was not unequivocal evidence of validity or otherwise. A recent analysis of peer review adds to this controversy by identifying an alarming lack of correlation between reviewers’ recommendations.

Something rotten at the core of science?
by David Horrobin

Peer review has its problems, some of which I became aware of several years ago, when Ed and I learned that autism researchers doing behavioral research were being defunded, apparently in favor of funding researchers studying the brain and biology.^* Peer review was involved.

Nevertheless, I do want K-12 curricula to be peer reviewed by specialists in the fields being taught.

Here's a case of a history textbook that was given no peer review:

Another historical malpractice foisted upon American school children came to light in Virginia last week . Once again it comes down to whether the standards of history as a discipline mean anything in the context of elementary and secondary history education.  Few of us would trust our children’s dental care to a historian.  Nor do we assume that anyone who has written a book can write a math textbook, regardless of their educational credentials.  But too often history seems different, subject to lower standards and inadequate review.

[snip]

The case at hand is straightforward.  Our Virginia: Past and Present  (Five Ponds Press, 2010) was approved by the Virginia Board of Education without a single historian involved in the review process. Fortunately an alert historian reviewing her daughter’s assignments noticed the glaring error: a statement that “thousands of Southern blacks fought in the Confederate ranks, including two black battalions under the command of Stonewall Jackson.”  It’s not true.  The reference to Jackson’s army is a total fabrication, and the broader reference to the Confederate army ignores the fact that slaves were forced into service and that there are no data available in any archive to document the statistic. 

So where did author Joy Masoff (not a historian) get her information?  From the Internet.  More specifically, from the web site of the Sons of the Confederate Veterans.  And even more specifically from a page that claims Frederick Douglass as the source for the statistic, but can’t even get his name spelled right.  The relevant quotation from Douglass is taken out of context, and there are no corroborating sources. 

Historical Malpractice and the Writing of Textbooks
By James R. Grossman Malpractice
October 25, 2010

The disciplines are disciplines: they are fields of study with established bodies of knowledge, rules of evidence, and modes of analysis.

A professional historian possesses knowledge and expertise the rest of us don't possess, and that is the knowledge we send our kids to school to acquire. At least, that is the knowledge I sent my children to school to acquire.

History textbooks should be vetted - or preferably written - by historians, math textbooks by mathematicians, and science textbooks by scientists.

That's not to say K-12 teachers should have no involvement; teachers are the people who can tell us whether a textbook is working with students. If a K-12 teacher writes a textbook that's vetted by disciplinary specialists - then great!

I think parents should have a vote on their children's textbooks and curricular materials, too, though in my dream world we wouldn't need to exercise it.



Rethinking Peer Review
The Editors of The New Atlantis, "Rethinking Peer Review," The New Atlantis, Number 13, Summer 2006, pp. 106-110.

^* I no longer remember all the details, and my understanding of what was going on may have been wrong. However, I do recall accurately that behavioral researchers were losing funding at the time. I don't know whether that is true today.

Bonnie on peer review and textbooks in computer science

Bonnie writes:

Peer review in computer science is very weird, because unlike other fields, we mainly publish in conferences rather than journals. It is a huge issue at tenure time because tenure committees assume that conference publications are meaningless - and they are in most fields - but not in computer science. Most of our conferences have acceptance rates of around 25%, and the top conferences are below 10%. That is where the peer review is happening in our field.

But that is research peer review, and I thought we were talking about textbooks. In CS, there are textbooks for certain "standard" courses - intro to programming, databases, operating systems, theory of computation, and a few others. The only real churn that I see is in the intro to programming area, because every time a new programming language hits the scene, you get 10 new books doing CS 1 and 2 in that langauge. In databases and operating systems, the same 3 or 4 authors have dominated for 20 years, issuing edition after edition of their book. So in reality, professors adopt the textbook, which isn't purchased or read by most of the class anyway, and then add lots of their own material. For courses that have no reasonable textbook - for example, everyone is adding courses on Android programming right now - we simply have the students buy a book aimed at professional developers, or cobble together notes on our own.

And here is Allison on peer review in mathematics.