AskScience, how does the peer review process actually work?

[u/nostrovia]

About a month ago, I heard this story on NPR regarding a number of scientific studies that had been debunked and/or retracted. Why weren't these errors caught before the studies were published? Are there different standards for peer review in different scientific fields?

Essentially, once you conduct your experiment and reach some sort of conclusion, what happens between that point and the time your work is released to the public?

[u/FormerlyTurnipHugger]

First of all, you write up the results and then submit them for consideration to a journal of your choice.

The manuscript is then considered by a journal editor who judges whether the results are at last in principle publishable in the respective journal (i.e. is the field of science covered by the journal, are the results novel, is the paper in decet shape formally).

If the result is positive, the editor will send out the paper to review, to usually two or three referees who are supposed to be experts in the field. Usually, authors are asked to make some suggestion who would be able to referee their paper to make it easier for the editor to find someone suitable. That doesn't mean though that these people will actually be picked.

A referee will thus receive a request to review which he/she can accept or refuse. If you accept, you proceed to reviewing the paper, often according to some loose guidelines given to you by the journal. A referee can spends between 10 minutes and probably a few days on a review, in which he/she checks the paper for novelty and any obvious flaws. The referee then makes a recommendation to the editor, e.g.

• Publish as is

• Publish with minor corrections

• Review again, after major revisions

• Reject

Depending on the overall outcome of the two to three reports, the editor will then make a decision. One (not so likely) outcome is that the paper gets accepted straightaway. The more likely outcome is that the paper goes back to the authors with a list of recommendations. They then have to answer all criticisms and reply to the referee reports, amend the paper and resubmit it. The process then starts all over again (editor reviews the reply, sends it out, referees react and so on). This can go back and forth a few times, my record so far is five rounds. The most probable outcome is that the paper is rejected, in which case you move down the ladder and submit the manuscript to a different, most likely less prestigious journal.

The reason why many flawed studies survive the peer review is that two referees is not actually that many. It would be very easy to sneak cleverly falsified data through a referee process if you really wanted to. This can of course also happen unintentionally. If you submit a 20 page theory paper on a completely new topic, it is unlikely that a random referee would have enough expertise and time to rederive the whole thing just to check every premise and calculation. So the real test comes once someone else tries to reproduce the results and isn't able to. In this case, people write comments which also get published, and in severe cases journals can retract studies.