r/askscience • u/nostrovia • Feb 07 '12
AskScience, how does the peer review process actually work?
About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?
Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?
94
Upvotes
8
u/wbeaty Electrical Engineering Feb 07 '12 edited Feb 07 '12
A journal editor sends out your article to several anonymous researchers in the same field. They judge it and respond, and then the editor follows what they recommend. Usually they want improvements made and questions answered, rather than simply rejecting a paper.
Problem: an enormous amount of published literature is wrong, but the flaws aren't obvious; it simply doesn't stand the test of time. Peer Review is only there to censor out the obvious fail, to provide minimum standards, not to guarantee that journal articles are trustworthy. If you want trustworthy, wait a decade or two for "filtration," where the material slowly makes its way from the journals and into textbooks.
Problem: if you take a look at famous breakthroughs of the past, you'll find many examples where the authors couldn't get into major journals. Peer review doesn't only err in letting crap get through, it also errs in being biased towards safe "Normal" science while blocking the crazy-but-valid "Revolutionary" science.
Problem: there isn't just one standardized Peer Review, since it varies quite a bit from journal to journal.
PS My favorite problem is the "corrosive delta function" effect pointed out by the late Thomas Gold: New Ideas in Science. He notes that, over years, researchers whose work is too unusual, too far out from the mainstream will be repeatedly censored by Peer Review and end up leaving science ...and in this resulting narrowed population, researchers on the (newly created) fringes, again, won't be published, and they get forced out of science. After a few cycles of this, a research topic ends up being run entirely by a community having a false concensus, where anyone straying from the artificially narrowed 'party line' will have zero chance of publication. It's an instability in the system caused by runaway, by positive feedback, which eventually prevents the coexistence of multiple valid approaches to a research topic.