Peer Review

An article in Physics World describes an "experiment" in peer review and its effect on the quality of published scientific research.

Just a small number of bad referees can significantly undermine the ability of the peer-review system to select the best scientific papers. That is according to a pair of complex systems researchers in Austria who have modelled an academic publishing system and showed that human foibles can have a dramatic effect on the quality of published science.

....

While the concept of peer review is widely considered the most appropriate system for regulating scientific publications, it is not without its critics. Some feel that the system's reliance on impartiality and the lack of remuneration for referees mean that in practice the process is not as open as it should be. This may be particularly apparent when referees are asked to review more controversial ideas that could damage their own standing within the community if they give their approval. 

The PW article is referenced by a shorter article on the Sante Fe Institute (SFI) website:

Just a small number of impartial, complacent, or incompetent referees can significantly undermine the ability of the scholarly peer-review system to select the best scientific papers.

SFI External Professor Stefan Thurner and collaborator Rudolph Hanel, both of the Medical University of Vienna, created a model of a generic specialist field where referees fall into one of five categories ranging from always correct to impartial.

Note that the first sentence of the SFI article is identical to the first sentence of the PW article (to which the former article links). But, the PW article is about research by an SFI-affiliated scholar. Neither article provides a link to a document describing the original research.

The PW article concludes with these two paragraphs:

When asked by physicsworld.com to offer an alternative to the current peer-review system, Thurner argues that science would benefit from the creation of a "market for scientific work". He envisages a situation where journal editors and their "scouts" search preprint servers for the most innovative papers before approaching authors with an offer of publication. The best papers, he believes, would naturally be picked up by a number of editors leaving it up to authors to choose their journal. "Papers that no-one wants to publish remain on the server and are open to everyone – but without the 'prestigious' quality stamp of a journal," Thurner explains.

This research is described in a paper submitted to the arXiv preprint server. 

arXiv is itself a preprint server. (Count the fractals - even strange loops.)

I tripped across the SFI article from a link on mantleplumes.org (accessed November 17, 2010):

11th Sept. 2010

Dear WM, I would like to bring your attention to an interesting article posted on the website of the Santa Fe Institute "Modeling shows scientific peer review system sensitive to bad refereeing"– Romain Meyer

I found the article apparently referred to: http://arxiv.org/PS_cache/arxiv/pdf/1008/1008.4324v1.pdf

So, if you're looking, too, here it is.


Cross posted at: http://maxfrac.blogspot.com/2010/11/peer-review.html.