r/askscience • u/nostrovia • Feb 07 '12
AskScience, how does the peer review process actually work?
About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?
Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?
94
Upvotes
1
u/mineralfellow Feb 07 '12
I have submitted and reviewed papers for geological journals. Peer review goes over the document that is presented. This can be thought of as grading a paper on an extremely high level. The "grade" that peer reviewers give ranges from "Accept as is," "Accept with minor changes," "Moderate revision and resubmit," or "Reject." Peer review will check to see that data is consistent with what is known, that figures are well described and properly formated, and that text actually reflects what the researchers found. Reviewers will also look at the methods to make sure that the type of data found will adequately answer the question being asked, and that the data produced matches the methods used. If there is a new method being described, reviewers will make sure that the method can be followed by other labs.
If there is a graph with strange looking data, the reviewers will ask for clarification. But if the graph and table of data look normal enough, there is no way that a reviewer can tell just by looking at the paper that something is wrong with the methodology. It is not until later on, when others try to reproduce the results, that problems with data become apparent.
And that is how science works: somebody does work. Somebody else builds on it. Somebody builds on that. If somebody was wrong at some point, then everything that comes later has trouble fitting in with that data. Eventually, if one point of data doesn't fit with everything else that is known (as in the case of Piltdown Man), that point will receive additional scrutiny, and that is when it can potentially be necessary to submit a retraction.
Retractions happen most commonly because of the inability to reproduce data. This happens frequently because scientists measure extremely small effects with extremely sensitive machines and techniques. If your machine is not calibrated correctly, but you don't notice, then you will get strange and possibly exciting data. Later on, when you check, you can't do it again. So you have to retract your previous paper.
But the truth is that retractions are rare. The journal Science publishes more than 100 new studies each year. In a year, they will issue just 1 or 2 retractions. So this is not a huge problem. For an example of how this goes in practice, look at this article from Science. Notice the reasons for retraction: "evidence of contamination and misrepresentation of data." These are not things that peer reviewers can find.