The two researchers, Douglas Peters and Stephen Ceci, wanted to test how reliable and unbiased this process actually is. To do this, they selected 12 papers that had been published about two to three years earlier in extremely selective American psychology journals.
The researchers then altered the names and university affiliations on the journal manuscripts and resubmitted the papers to the same journal. In theory, these papers should have been high quality — they’d already made it into these prestigious publications. If the process worked well, the studies that were published the first time would be approved for publication again the second time around.
What Peters and Ceci found was surprising. Nearly 90 percent of the peer reviewers who looked at the resubmitted articles recommended
against publication this time. In many cases, they said the articles had “serious methodological flaws.”
This raised a number of disquieting possibilities. Were these, in fact, seriously flawed papers that got accepted and published? Can bad papers squeak through depending on who reviews them? Did some papers get in because of the prestige of their authors or affiliations? At the very least, the experiment suggested the peer review process was unnervingly inconsistent..............
.............
The
Lancet editor Richard Horton has
called the process “unjust, unaccountable ... often insulting, usually ignorant, occasionally foolish, and frequently wrong.” Not to mention that identifying peer reviewers and getting their comments slows down the progress of science — papers can be held up for months or years — and costs society a lot of money. Scientists and professors, after all, need to take time away from their research to edit, unpaid, the work of others.
Richard Smith, the former editor of the
BMJ, summed up: “We have
little or no evidence that peer review ‘works,’ but we have lots of evidence of its downside.” Another former editor of the
Lancet, Robbie Fox,
used to joke that his journal “had a system of throwing a pile of papers down the stairs and publishing those that reached the bottom.” Not exactly reassuring comments from the editors of the world’s leading medical journals